Multi-Cloud Networking

Published April 5, 2023 | Last modified August 5, 2025

Objective

This guide provides instructions on how to seamlessly connect and secure applications between multiple cloud networks using F5® Distributed Cloud Mesh and F5 Distributed Cloud Console.

The steps to connect and secure applications between multiple cloud networks are:

Figure: Multi-Cloud Networking and Security Setup Steps

The following images show the topology of the example for the use case provided in this document:

Figure: Multi-Cloud Networking and Security Sample Topology

Using the instructions provided in this guide, you can set up Amazon Virtual Private Cloud (Amazon VPC) site, data center cloud gateway, setup secure networking between the 2 clouds, and setup end-to-end monitoring.

Prerequisites

Distributed Cloud Console SaaS account. If you do not have an account, see Getting Started with Console.
Amazon Web Services (AWS) account. This is required to deploy a Site.
Private cloud environment (data center) with networking connectivity to the Internet and TOR from the hardware. The management IP address for your hardware is required.

Configuration

The use case provided in this guide sets up F5 Distributed Cloud Services sites as gateways for the ingress and egress traffic for the two cloud networks. The datacenter gateway site is on a physical hardware in an on-premise datacenter location. This datacenter also has TOR behind which we have VM based hosts sitting on two different subnets.

The following actions outline the activities in setting up secure networking between the AWS VPC and private data center cloud.

Distributed Cloud Services AWS VPC Site is deployed using the Console.
Distributed Cloud Services VMware site is deployed on the ESXi host using the OVA template.
The two cloud environments are connected using the Distributed Cloud Services global network and secured using the network policies.
Local-breakout for hosts on the VMware site is configured. This allows inside network hosts to access the internet using SNAT. This is achieved by configuring a Fleet and related objects - Virtual Networks, Network Interfaces, and Network Connectors on the VMware site.

Note: Ensure that you keep the Amazon Elastic IP VIPs ready for later use in configuration.

Step 1: Deploy Site (Public Cloud)

The following video shows the site deployment workflow:

Perform the following steps to deploy a Site in your VPC:

Step 1.1: Start creating AWS VPC site object.

Select the Multi-Cloud Network Connect service.
Select Manage > Site Management > AWS VPC Sites in the configuration menu. Select Add AWS VPC Site.
Enter a name for your VPC site in the metadata section.

Step 1.1.1: Configure site type selection.

Go to the Site Type Selection section and perform the following:
- Select a region in the AWS Region drop-down field. This example selects us-west-2.
- Select New VPC Parameters for the VPC field. If you want a specific name for the AWS VPC, select Choose VPC Name in the AWS VPC Name field and then enter your preferred name. Otherwise, leave the drop-down selection set to Autogenerate VPC Name. Enter the CIDR in the Primary IPv4 CIDR blocks field. This example sets 192.168.32.0/22 as the CIDR.
- Select Ingress/Egress Gateway (Two Interface) for the Select Ingress Gateway or Ingress/Egress Gateway field.

Figure: AWS VPC Site Configuration of Site Type

Step 1.1.2: Configure ingress/egress gateway nodes.

Select Configure to open the two-interface node configuration wizard.
Select Add Item in the Ingress/Egress Gateway (two Interface) Nodes in AZ section to add the Two Interface Node.
- Select an option for the AWS AZ name field that matches the configured AWS Region. This example selects us-west-2a.
- Select New Subnet for the Workload Subnet field, and then enter the subnet address in the IPv4 Subnet field.
- Similarly configure a subnet address for the Subnet for Outside Interface section.

Figure: Two Interface Node Configuration

Note: This example sets 192.168.32.128/25 as the workload subnet and 192.168.32.0/25 as the outside interface subnet.

Then select Apply to return to the two-interface node configuration wizard.

Figure: Ingress/Egress Gateway Nodes Configuration

Note: The Site Network Firewall configuration will be done as part of step 4 of this quick start guide.

Select Apply to complete the two-interface node configuration and return to the AWS VPC site object creation.

Step 1.1.3: Complete AWS VPC site object creation.

Select the AWS credentials object under the Automatic Deployment field.

Note: Select Add Item in the Cloud Credentials drop-down menu to create the credentials. You will need an AWS access key ID and AWS secret access key. See Step 1 of the guide for more information.

Make a selection in the AWS Instance Type for Node field.
Enter the public key for remote SSH if necessary.
Select Save and Exit to complete creating the AWS VPC object. The AWS VPC site object gets displayed.

Figure: Automatic Deployment and Site Node Parameters

Step 1.2: Deploy AWS VPC site.

Select the Apply button for the created AWS VPC site object. This will create the VPC site.

Figure: Terraform Apply for the VPC Object

After a few minutes, select the Refresh button to verify that the site is created. The Site Admin State should be Online, and the Status should be Applied.

Figure: VPC Object Online

Verify that the site is created and ready to use. Select Sites in the left navigation pane and then select your site from the list of sites at the bottom of the Sites page. Navigate to Site Connectivity > PoP (RE) Connectivity and hover over your newly created site to see basic health and connectivity information.
Select the site to see a slide-out panel with more details.
Select Explore Site in the panel to see the site's full dashboard.

Figure: Site Dashboard and Health Details

Select the Interfaces tab to check the interface status and details such as throughput. You can view inside and outside interfaces using the Inside and Outside options.

Figure: Site Dashboard Interfaces View

Step 2: Deploy Site (Private DC)

Deploying site in your private data center consists of downloading the Distributed Cloud Services site image and installing gateway site on the data center.

Note: Refer to the Prerequisites chapter for data center site deployment prerequisites.

The following video shows the data center site deployment workflow:

Perform the following steps for deploying gateway site on the data center:

Step 2.1: Get/Create site token.

Log into the Console and select the Multi-Cloud Network Connect service.
Navigate to Manage > Site Management > Site Tokens.
Save the site token (the value in the UID column) for use in VMware when creating a Virtual Machine (VM)

Figure: Create site token

Note: If you have not already created a site token, use Add site token to create a token.

Step 2.2: Download and install the VMware site image on your data center.

See VMware Images page for downloading OVA image.

Step 2.3: Install the VM with the OVA template.

Log into VSphere webclient, ESXi console to create a VM from the OVA template.
Select Deploy a virtual machine from an OVF or OVA file and select Next.

Figure: Create New VM from OVA Template

Select OVA file:
- Enter a name for the virtual site.
- Select or drag-and-drop the OVA file.
- Select Next.

Figure: Import OVA Template

Select an appropriate amount of storage for the VM and select Next.

Figure: Select VM Storage

Enter the deployment options:
- Select the OUTSIDE and REGULAR (inside) network mappings.
- Ensure that Power on automatically is checked.
- Press `Next'.

Figure: VM Deployment Options

Enter additional settings:
- Expand the Options section.
- Enter the Hostname. This should match the name of the virtual site.
- Enter the token you saved from step 2.1.
- Enter the Cluster Name, which should also match the name of the virtual site.
- Enter the Name for outside network interface.
- Enter vmware-regular-nic-voltmesh for the Certified Hardware field.
- Enter the Latitude and Longitude for your virtual site.
- Select Next.

Figure: VM Deployment Options

Validate the settings and select Finish. The VM gets booted up.

Step 2.4: Perform site registration in Console.

Log into the Console, select the Multi-Cloud Network Connect service, and navigate to Manage > Site Management > Registrations.
Select Pending Registrations tab. Find the registration request for your site and accept the registration. Validate the information shown and then select the checkbox icon. The Registration acceptance sliding sidebar will show information about your new VM.
Select Save and Exit.
Wait for the registration to complete and the site to come up. You can find the site in the Sites > Site List view. Select your site to open the site dashboard and ensure that its health score is 100 and its interfaces are up in the Interfaces tab.

Step 2.5: Create a fleet.

In the Multi-Cloud Network Connect service, navigate to Manage > Site Management>Fleets.
Select Add fleet.
Enter a name for your fleet and enter a label in the Fleet Label Value field. This label is later used to apply to the site.

Figure: Fleet Name and Label

Step 2.5.1: Configure virtual networks.

Use the Outside (Site Local) Virtual Network pull-down to see and select the Add Item option.
Enter a name for your outside network and select Site Local(Outside) Network for the Select Type of Network field. Select Continue to create the network and add it to the fleet configuration.

Figure: Outside Virtual Network

In the Fleet Configuration screen, create the inside virtual network in the same way. Select Select inside virtual network object and select Add Item. Enter a name for your inside network and select Site Local Inside Network for the Select Type of Network field. Select Continue to create network and add to the fleet configuration.

Step 2.5.2: Configure network interfaces.

Go to the Network Interfaces section and configure the following:

Use the Select Interface Config pulldown to select List of Interfaces.
Use the List of Interfaces pulldown to select the Add Item option.
Enter a name for the interface, select the Ethernet Interface for the Interface Type field, and then select Configure to set up the Ethernet interface.
Select eth1 in the Ethernet Device field.
Go to the IP Configuration section and select DHCP Server for the Select Interface Address Method field. Select Configure under the DHCP Server option to open DHCP server configuration. Perform the configuration using the following guidelines:
- Select Configure in the DHCP Networks section to open DHCP server configuration.
- Enter a prefix in the Network Prefix field.
- Select Add Item in the DHCP Pools section and enter the starting and ending IP addresses.
- Select Apply at the bottom of the form to add the settings to the DHCP server configuration. This sets the DHCP pool, default gateway, and DNS server address.

Figure: DHCP Network Configuration

Select Apply in the DHCP Server configuration to apply the DHCP server to the Ethernet interface configuration.
Select Site Local Network Inside in the Select Virtual Network field in the Ethernet interface configuration.

Figure: Ethernet Interface Configuration

Select Apply to set the ethernet interface to the network interface configuration.

Figure: Network Interface Configuration

Select Continue to create and add the network interface to the fleet.

Step 2.5.3: Configure network connector and complete fleet creation.

This step creates a network connector with one in SNAT mode and the other in the direct mode to the global network.

Select Add Item in the Network Connectors section and then use the Network Connectors drop-down menu to select Add Item.
Enter a name for the network connector and select Continue and then Select Network Connector to add the network connector to the fleet. This sets the network connector to function in the default SNAT mode that connects site local inside network to site local outside network. This is used for the data center private cloud for establishing connectivity from inside subnets to outside network through the Site deployed on the VMware VM.

Figure: Network Connector for Private DC

Scroll down and select Save and Exit in the fleet configuration screen to create the fleet.

At this point, you can verify that the inside subnets can communicate with each other but accessing outside their networks is not possible. You can use ping command to verify the same.

Step 2.5.4: Add VMware site to the fleet.

Select Manage > Site Management > AWS VPC Sites.
Select ...>Manage Configuration for your VMware site and then select Edit Configuration in the upper right corner to open its configuration edit form.
Select in the Labels field and add ves.io/fleet with the value of fleet label you created in previous step.

Figure: Add Fleet Label to Site

Select Save and Exit to apply fleet settings to the site.
Verify that the fleet interfaces are applied to the site. Check the site local UI dashboard for Ethernet interfaces section. The interface Eth1 gets IP address assigned by the DHCP server configured in the fleet.

Figure: Ethernet Interface Details in Local UI Dashboard

At this point, you can verify that the inside subnets can access outside networks via the Site by means of SNAT. You can verify the same with the ping command.

Note: To check connectivity over internet, you can execute ping 8.8.8.8 to Google DNS server.

Step 3: Connect Networks

Connecting networks includes configuring local-breakout for hosts on the VMware site. That is, allowing inside network hosts to access the Internet using SNAT. This is done by configuring a Fleet and related objects - Virtual Networks, Network Interfaces, and Network Connectors. This includes creating network connectors with one in SNAT mode and other in the direct mode to the global network.

After that, connect both the VMware and AWS inside networks using Distributed Cloud Services ADN.

The following video shows the workflow of connecting and securing the two networks:

Perform the following to connect and secure the two cloud networks:

Step 3.1: Create global network.

Log into the Console and select the Multi-Cloud Network Connect service.
Navigate to Manage > Networking > Virtual Networks and select Add virtual network.
Enter a name and select Global Network in the Select Type of Network field.
Select Save and Exit.

Figure: Setup Global Network

Step 3.2: Connect the global network with the VMware site.

Navigate to Manage > Site Management > Fleets to see a list of your fleets.
Select ... > Manage Configuration for the fleet you created in step 2. Then select Edit Configuration in the upper right corner to edit the fleet.
Scroll down to the Network Connectors section and select Add Item and then select Add Item from the newly created Network Connectors drop-down menu.
- Enter a name for the network connector.
- Select Direct, Site Local Inside to a Global Network in the Select Network Connector Type field.
- In the Global Virtual Network field, select the global network you created in the previous step.

Figure: Network Connector Configuration

Select Continue to save the network connector and then Save and Exit to save the updated fleet configuration.

Step 3.3: Connect the global network with the AWS VPC site.

Navigate to Manage > Site Management > AWS VPC Sites.
Select ... > Manage Configuration for the site you created in step 1. Then select Edit Configuration in the upper right corner to edit the fleet.
Scroll down to the Networking Config section and select Edit Configuration under Ingress/Egress Gateway (Two Interface).
Select Connect Global Networks in the Select Global Networks to Connect field and then select Add Item.
- Select the global network you created previously in the Global Virtual Network field.
- Select Apply to save the changes.

Figure: Global Network Connections

Select Manage Static Routes in the Manage Static Routes for Inside Network field.
In the List of Static Routes section, select Add Item.
- Enter the network prefix in the Simple Static Route field.
- Select Add Item.

Figure: Site Static Route Configuration

Select Apply to add the network configuration.
Select Apply to save the Ingress/Egress Gateway configuration.
Select Save and Exit to save updates to the site configuration.

Now you can verify that the connectivity is enabled between the VMware subnets and the AWS cloud EC2 instances. You can use ping to verify the same.

Step 4: Secure Networks

Securing networks includes applying firewall policies to restrict the network accesses for chosen networks. It also includes applying forward proxy policies to allow access to chosen URLs. This is achieved by means of creating a network firewall with the policies and applying to the fleet.

This example creates a firewall policy that allows access only from one subnet of the private DC to the AWS cloud and blocks access for all other subnets. It also creates a forward proxy policy that blocks access to a specific domain and allows everything else.

Perform the following steps to set up secure networks.

Step 4.1 Create and add network firewall to the fleet.

Log into the Console and select the Multi-Cloud Network Connect service.
Select Manage > Site Management > Fleets. Find your fleet from the displayed list and select ... > Manage Configuration to open its configuration form, and then select Edit Configuration in the upper right.
Scroll down to the Network Firewall section and use the Network Firewall pull-down menu to selectAdd Item. Enter a name for the firewall.

Step 4.1.1 Create and add firewall policies to the fleet.

Scroll down to the Firewall Policy section and select Active Firewall Policies. Use the Select Item pull-down menu and select Add Item. This policy will allow all traffic for the server1 prefix.

Figure: Network and Policy for Network Firewall

Enter a name for the policy and add the prefix of the server1 subnet (for which you want to allow access) in the IPv4 Prefix List field.
Select Configure on the Connections From Policy Endpoints section to configure an egress rule. Select Add Item to create the egress rule.
- Set a name for the egress rule and select Allow for the Action field.
- Select Apply to add the egress rule to the list of egress rules.

Figure: Egress Allow Rules

Select Apply to save the egress rules list.
Select Continue to save the server1 firewall policy

Select Add item in the Firewall policy section to add another policy for the server2 prefix. This one will deny access to a subnet.
Select the Select Item field and select Add Item.
Enter a name for the policy and add the prefix of the server2 subnet (for which you want to block access) in the IPv4 Prefix List field.
Select Configure on the Connections From Policy Endpoints section to configure an egress rule. Select Add Item to create the egress rule that denies traffic to a subnet.
- Set a name for the egress rule and leave Deny for the Action field.
- Select List IP Prefix Set in the Select Other Endpoint field.
- Select in the reference field and then select Add Item.
- Enter a name for the prefix set, enter the subnet prefix, and press Continue.
- Select Apply to add the egress rule to the list of egress rules.

Figure: Egress Deny Rule for a Subnet

Select Add Item to create a second egress rule that allows all other traffic.
- Set a name for the egress rule and select Allow for the Action field.
- Select Add item to add the egress rule to the list of egress rules.
Select Apply to save the egress rule list, and select Continue to complete the server2 policy.

Select Add Item to add a third policy slot for the local internet breakout. Then use the Select Item pull-down menu to select Add Item.
Enter a name for the new firewall policy.
Select Any Endpoint for the Endpoint(s) field in the Policy For Endpoints section.
Select Configure on the Connections To Policy Endpoints section to configure an ingress rule that allows all traffic, and then selectAdd Item to create the ingress rule.
- Enter a name for the ingress rule.
- Select Allow for the Action field.
- Select Apply to add the ingress rule to the list of ingress rules.
- Select Apply to add the list to the network policy.
Select Configure on the Connections From Policy Endpoints section to configure an egress rule for this network policy, and then selectAdd Item to create the egress rule.
- Set a name for the egress rule.
- Select Allow for the Action field.
- Select Apply to add the egress rule to the list of egress rules.
Select Apply to save the egress rule list, select Continue to complete the third firewall policy, select Continue to add the network firewall to the fleet configuration, and finally select Save and Exit in the fleet configuration to save changes to fleet.

Step 4.1.2 Verify the policy operation.

Verify that access from only one subnet is allowed to the EC2 instances of AWS. Also, verify that the site local breakout and internet access is still allowed. Enter ping command to an EC2 instance IP address from both subnets and only one is allowed.
You can also verify the policy and rule hits from Console. Navigate to Manage > Firewall > Firewall Policies. Check the Hits field for your policy.

Figure: Policy Hits

Select the value in the Hits column for your policy to view the rule hits.

Step 4.2 Add a forward proxy rule that blocks a specific URL.

To enable URL filtering, update the firewall created in step 4.1.

Go to Manage > Firewall > Network Firewalls.
Select ... > Manage Configuration for the firewall created earlier, and then select Edit Configuration.
Scroll down to Forward Proxy Policy and select Active Forward Proxy Policies.
Use the Select Item pull-down menu to select Create new Forward Proxy Policy.
- Enter a name for the new policy.
- Select All forward Proxies on Site in the Proxy section so that it gets activated everywhere.
- Select Denied connections in the Rules section.
- Select Add Item in the TLS Domains section, enter the URL you wish to block in the Exact Value field, and select Apply to add this URL to your denied TLS domains.

Figure: Enable Forward Proxy for facebook.com

Select Continue and then Save and Exit to save the changes to your firewall.

Step 4.3 Enable forward proxy for network connector of the private DC.

Go to Manage > Networking > Network Connectors. Select ... > Manage Configuration for the VMware site network connector you created, and then select Edit Configuration.

Figure: Enable Forward Proxy for VMware Networks

Select Enable Forward Proxy for the Select Forward Proxy field. Select Save and Exit.

This is required to apply the forward proxy policies.