Create Two Node HA Infrastructure for Load Balancing Using Virtual Sites with Customer Edges

Objective

This document will guide you on how to achieve high availability (HA) of a CE with two single-node CE sites grouped into a single logical entity - a Virtual Site (vSite), and how to use this vSite for application delivery services for your apps. This document will also describe the use cases for application delivery using this type of virtual site deployment.

Introduction

For production deployments, it is recommended to deploy a three-node CE Site for high availability (HA). In this model, the Secure Mesh Site software is deployed on three separate nodes, and they are clustered to operate as a single CE Site. However, this may not be suitable for every customer deployment. For such customer scenarios, the recommended alternative approach is to group two single-node CE sites into a single logical entity – a Virtual Site (vSite). This vSite can be leveraged to provide application delivery services for your apps.

---

Prerequisites

The following general prerequisites apply:

• A Distributed Cloud Services Account. If you do not have an account, see Create an Account.

• Allow traffic from and to the Distributed Cloud public IP addresses to your network and allowlist related domain names. See Firewall and Proxy Server Allowlist Reference guide for the list of IP addresses and domain names.

• The CE sites must follow the Secure Mesh Site v2 workflow using the following procedure: Create Secure Mesh Site v2.

• For public cloud providers, additional manual configuration of cloud provider-specific load balancers may be required.

• Ensure that the two CE sites (that are part of the vSite) are deployed in the same network domain. This implies the following:

  • For public clouds: CE sites are within the same VPC/VNet.
  • For on-premises: Each interface of the CE Sites is within the same Layer 2 broadcast domain.

• Ensure that any CE Site within a vSite can communicate seamlessly with the clients or origin servers in the same vSite.

Important: After you deploy the CE Site, the IP address for the SLO interface cannot be changed. Also, the MAC address cannot be changed.

---

Procedure

This procedure will guide you on how to configure high availability with two single-node CE sites aggregated as a single vSite and how to use this vSite for application delivery services for the apps.

Configure Virtual Site

This section will show you how to create a vSite using two single-node CE sites.

Deploy CE Sites

Create two single-node CE sites in the same VPC/VNet or local network domain. Each site must have two interfaces: a Site Local Outside (SLO) for external connectivity and a Site Local Inside (SLI)/segment for origin servers.

As an example, this procedure creates two CE sites in AWS, which are named pg-aws-vsite-2a and pg-aws-vsite-2b.

Refer to this document for creating CE sites: Create Secure Mesh Site v2.

• For AWS, deploy each site using the procedure here: Deploy Secure Mesh Site v2 in AWS (ClickOps).
• For Azure, deploy each site using the procedure here: Deploy Secure Mesh Site v2 in Azure (ClickOps).
• For KVM, deploy each site using the procedure here: Deploy Secure Mesh Site v2 on KVM (ClickOps).

Create Virtual Site Object

Refer to this document to create a virtual site object: Create Virtual Site.

As an example, this procedure creates a vSite with name pg-aws-vsite-2.

Assign CE Sites to Virtual Site

After the vSite is created, the CE sites created earlier are grouped using labels (with the site selector expressions). As an example, the sites pg-aws-vsite-2a and pg-aws-vsite-2b are grouped together as a single vSite called pg-aws-vsite-2.

Configure Origin Pool Using Virtual Site

To create an origin pool using a vSite, follow the steps provided.

Step 1: Create origin pool.

• In Distributed Cloud Console, click Multi-Cloud App Connect.

• Navigate to Manage > Load Balancers > Origin Pools.

• Click Add Origin Pool.

Step 2: Configure origin pool.

• Enter a name for the origin pool.

• Click Add Item.

• Use the virtual site option to select pg-aws-vsite-2.

Figure: Origin Pool Configuration

• Add the origin servers associated with the sites aggregated under pg-aws-vsite-2.

• Click Apply.

Step 3: Save configuration.

• Ensure you create two origin servers. Both sites are in the same VPC/VNnet or routed networks. Because of this, both CE sites have connectivity to each origin server in two different subnets (10.2.11.0/24 and 10.2.12.0/24).

Figure: Two Subnets

• Configure a health check appropriate for the application origin pool.

Figure: Origin Pool Health Check

• Click Save and Exit.

Use Cases

Public Application Delivery Using Regional Edges

With this particular use case, the application is a public app, and therefore it is delivered to the public Internet using Regional Edges (REs) using F5’s global anycast network. The application itself is hosted in a customer location, like AWS, Azure, or on-premises, and is behind the virtual site (which is a group of two single-node CE sites). The communication between the Regional Edges (REs) and Customer Edges (CEs) is encrypted using either IPsec or SSL tunnels.

Configure Load Balancer

• Configure a vSite. Refer to Configure Virtual Site for details.

• Configure an origin pool using a vSite created in step 1. Refer to Configure Origin Pool Using vSite section using for details.

• To advertise the application to the Regional Edges (REs):

  • Navigate to the Multi-Cloud App Connect service.
  • Select Manage > Load Balancers > HTTP Load Balancers.
  • Click Add HTTP Load Balancer.

• Add the origin pool previously created.

• From the VIP Advertisement menu, select Internet.

Note: As this application is delivered over the Internet, note that the REs manage the VIPs and that the origin servers are managed by sites grouped under vSite2.

Public Application Delivery Using Customer Edges Grouped as a Virtual Site

With this particular use case, the public applications are delivered to the Internet directly using Customer Edges (CEs) sites grouped as a single vSite. The origin pool is also behind the same CEs grouped as a vSite.

In the case of public cloud providers, like AWS and Azure, the cloud provider specific network load balancer (AWS Network Load Balancer or Azure Network Load Balancer) is used for efficient load balancing of incoming traffic from the Internet to the CE sites. The Network Load Balancer ensures high availability and scalability by routing traffic to the appropriate CE sites based on health checks and load distribution policies.

This configuration leverages the flexibility of F5 Distributed Cloud and an AWS/Azure Network Load Balancer to provide robust traffic management, ensuring reliable access to services hosted on the CE sites.

Note: The procedure for AWS and Azure is identical, with the only differences being in the configuration of the cloud provider's Network Load Balancers.

In the diagram below, the App VIP is the AWS or Azure Network Load Balancer, the LB for APP is the F5 Application Load Balancer. App Origin Pool are the applications in the same AWS VPC or Azure VNet as CEs.

Configure Load Balancer

• Configure a vSite. Refer to Configure Virtual Site for details.

• Configure an origin pool using a vSite created in step 1. Refer to Configure Origin Pool Using vSite section using for details.

• Configure the F5 Distributed Cloud HTTP load balancer as described below:

  • Navigate to the Multi-Cloud App Connect service.
  • Select Manage > Load Balancers > HTTP Load Balancers.
  • Click Add HTTP Load Balancer.

• Configure the application domain and application origin pool created previously in Step 2.

• From the VIP Advertisement menu, select the option to advertise the application on the Site Local Outside (SLO) of the vSite created in Step 1.

• Create a cloud provider-specific network load balancer (NLB), in cases of AWS or Azure:

  • For AWS, refer to Configure AWS Network Load Balancer for details
  • For Azure, refer to Configure Azure Network Load Balancer for details.

• To reach the AWS NLB, you can use either the NLB VIP IP address or the AWS-provided domain name.

• To reach the Azure NLB, you can use the NLB VIP IP address.

• Ensure you expose the AWS or Azure VIPs to the Internet by assigning the Public IP address to the VIP.

Configure AWS Network Load Balancer

Create an AWS NLB using the steps in this section.

Step 1:

• In AWS Console, navigate to the EC2 service.

• Click Load balancers.

• Click Create load balancer.

• Choose Network Load Balancer and then click Create to proceed with the configuration.

Figure: Configure AWS NLB

Figure: Choose AWS NLB

Step 2:

• Expose the AWS NLB VIP to the Internet (Internet-facing option).

Figure: AWS NLB VIP Public

• Provide all the necessary information, like the VPC and subnets that contain the CE SLO interfaces.

Figure: AWS NLB Network Mapping

• Define the security group and TCP listener port (TCP 80).

Figure: AWS NLB Security Group

Step 3:

• Define the target group that will point to the F5 Distributed Cloud VIP IP addresses.

Figure: AWS NLB Target Group

• Choose an IP address-based target, as the F5 Distributed Cloud CE has multiple interfaces.

Figure: AWS NLB IP Address Target

• Define the target group port and health checks.

Figure: AWS NLB Health Check

Step 4:

• Locate the Site Local Outside (SLO) interface IP addresses for the CE sites. Follow these steps in F5 Console:
  • In the Multi-Cloud Network Connect service, navigate to Overview > Infrastructure > Sites.
  • For your CE sites, identify the SLO interfaces. Note the OUTSIDE IP addresses for each CE.

Figure: CE Site SLO Interface IP Address

Figure: CE Site SLO Interface IP Address

• Specify the SLO IP addresses as targets.

Figure: Specify SLO Interface IP Addresses

• After you add both IP addresses, click Create target group.

• Verify the IP addresses under Registered targets section.

Figure: Verify SLO Interface IP Addresses

• In the AWS Network Load Balancer window, click Refresh Target groups and choose the previously created target group.

Figure: Refresh Target Groups

Figure: Add Target Group

Figure: Resource Map View

Step 5:

• Enable cross-zone load balancing to distribute traffic across availability zones. Perform the following:
  • In the Network Load Balancer settings, click on Enable cross-zone load balancing.

Figure: Enable CZLB

• As a good practice, enable health checks to validate per CE site VIP reachability by AWS NLB.

Figure: Enable Health Check

Configure Azure Network Load Balancer

Create an Azure NLB using the steps in this section. Note that to advertise apps to the Internet directly using CEs, an Azure Network Load Balancer (NLB) with a public IP address is required in front of the CE.

Step 1:

• In your Resource Group, create a new resource of type NLB.

• Ensure that the Type is set to Public.

Figure: Set NLB Type

Step 2:

• To assign the Public IP to Frontend IP, use one of the existing public IP addresses previously created. If you do not have one, create it using resource Public IP addresses.

Figure: Create Public IP Address

• After the public IP address is ready, add it as the frontend IP address.

Figure: Add Frontend IP Address

• For the backend pool, choose the SLO interfaces as pool members selected by NIC.

Figure: Add SLO Interfaces

Step 3:

• Add load balancing rules.

Figure: Add Load Balancer Rules

Figure: Load Balancer Rule Summary

• Optionally, add health probes to actively monitor the health of Distributed Cloud VIP addresses.

Public Application Delivery Across Customer Edges Grouped into a Virtual Site

With this particular use case, the application is private and is delivered to other Customer Edge (CE) sites grouped into a Virtual Site (vSite). In this example, the application is advertised to Customer Edges (CEs) grouped by vSite-1 and the application origin pool is behind Customer Edge (CE) sites grouped by vSite-2. To balance traffic to CE VIPs on vSite-1, an AWS or Azure Network Load Balancer (NLB) is used. The CE sites in the two vSites can be connected directly using Site Mesh Group, DC Cluster Group, or over Regional Edges (REs).

The procedure for AWS and Azure is identical, with the only difference being in the configuration of the cloud provider's Network Load Balancers.

In the diagram below, the App VIP is the AWS or Azure Network Load Balancer, the LB for APP is the F5 Application Load Balancer. App Origin Pool are the applications in the same AWS VPC or Azure VNet as CEs in vSite-2 origin.

Configure Load Balancer

• Configure Virtual Site (vSite). Refer to Configure Virtual Site for details. In this example, we are creating two vSites: pg-aws-vsite-1 for client and pg-aws-vsite-2 for origin pool. The sites pg-aws-vsite-1a and pg-aws-vsite-1b are grouped into virtual site pg-aws-vsite-1. The sites pg-aws-vsite-2a and pg-aws-vsite-2b are grouped into virtual site pg-aws-vsite-2.

• Configure an origin pool using the virtual site pg-aws-vsite-2 created in Step 1. Refer to Configure Origin Pool Using vSite section using for details.

• To advertise the application to CEs:

  • Navigate to the Multi-Cloud App Connect service.
  • Click Manage > Load Balancers > HTTP Load Balancers.
  • Click Add HTTP Load Balancer.
  • In the Origins section, select the origin pool previously created.

• From the VIP Advertisement menu, select the option to advertise the application on the Site Local Inside (SLI) of the Virtual Site pg-aws-vsite-1 previously created in Step 1.

Note: Each CE Site will have a VIP configured on its SLI interface IP address, which acts as the backend (origin pool) for the external load balancer. In this case, an AWS or Azure Network Load Balancer.

• Create a cloud provider-specific network load balancer (NLB), in cases of AWS or Azure:

  • For AWS, refer to Configure AWS Network Load Balancer for details
  • For Azure, refer to Configure Azure Network Load Balancer for details.

• To reach the AWS NLB, you can use either the NLB VIP IP address or the AWS-provided domain name.

• To reach the Azure NLB, you can use the NLB VIP IP address.

• Ensure you expose the AWS or Azure VIPs to the Internet by assigning the Public IP address to the VIP.

Configure AWS Network Load Balancer

Create an AWS NLB using the steps in this section.

Step 1:

• In AWS Console, navigate to the EC2 service.

• Click Load balancers.

• Click Create load balancer.

• Choose Network Load Balancer and then click Create to proceed with the configuration.

Figure: Configure AWS NLB

Figure: Choose AWS NLB

Step 2:

• Expose the AWS NLB VIP internally (Internal option).

Figure: AWS NLB VIP Internal

• Provide all the necessary information, like the VPC and subnets that contain the CE SLI interfaces.

Figure: AWS NLB Network Subnets

• Define the security group and TCP listener port (TCP 80). Note that the security group must allow listening on a port (in this case, TCP 80).

Figure: AWS NLB Security Group

Step 3:

• Define the target group that will point to the F5 Distributed Cloud VIP IP addresses.

Figure: AWS NLB Target Group

• Choose an IP address-based target, as the F5 Distributed Cloud CE has multiple interfaces.

Figure: AWS NLB IP Address Target

Step 4:

• Locate the Site Local Outside (SLO) interface IP addresses for the CE sites. Follow these steps in F5 Console:
  • In the Multi-Cloud Network Connect service, navigate to Overview > Infrastructure > Sites.
  • For your CE sites, identify the SLI interfaces. Note the INSIDE IP addresses for each CE.

Figure: CE Site SLI Interface IP Address

Figure: CE Site SLI Interface IP Address

• Specify the SLO IP addresses as targets by clicking Include as pending below.

Figure: Specify SLO Interface IP Addresses

• After you add both IP addresses, click Create target group.

• Verify the IP addresses under Registered targets section.

Figure: Verify SLO Interface IP Addresses

• In the AWS Network Load Balancer window, click Refresh Target groups and choose the previously created target group.

• Review the listeners and target group IP addresses, and then click Create load balancer.

Step 5:

• Enable cross-zone load balancing to distribute traffic across availability zones. Perform the following:

  • In the Network Load Balancer settings, click on Enable cross-zone load balancing.

• As a good practice, enable health checks to validate per CE site VIP reachability by AWS NLB.

Configure Azure Network Load Balancer

Create an Azure NLB using the steps in this section.

To expose F5 Distributed Cloud CE VIPs to an Inside or Segment network, an Azure Network Load Balancer (NLB) with a frontend Internal IP is required.

Step 1:

• In your Resource Group, create a new resource of type NLB.

• Ensure that the Type is set to Internal.

• Create the SLI frontend IP address.

Figure: Azure Frontend IP Address

Figure: Azure Frontend IP Address

Step 2:

• Add load balancing rules.

Figure: Azure Load Balancer Rules

Figure: Azure Load Balancer Summary

• Optionally, add health probes to actively monitor the health of Distributed Cloud VIP addresses.

Failover Optimization – Outlier Detection

Origin server failure detection is handled by health checks, which actively monitor the availability of services. These checks are deployed on the sites where the origin servers are defined rather than on the site hosting the load balancer VIP.

If a site, such as Site2a, becomes unavailable, the active health checks will not immediately inform the load balancer on Site1 that the origin servers are inaccessible via Site2a. Instead, the origin servers will only be removed from the load balancer pool once the global controllers confirm that the site is down.

This process generally takes 110-120 seconds. In some scenarios, this delay may be too long, necessitating optimization to minimize downtime. The Outlier Detection feature can assist in addressing this issue effectively.

Configure Outlier Detection

Note that some potential drawbacks for this feature include passive health checks, working only with HTTP/HTTPS, and requiring traffic be sent to production to a broken node, to detect if it is operational.

• In the Multi-Cloud App Connect service, navigate to Manage > Load Balancers and then select Origin Pools.

• For your desired load balancer, click ....

• Click Manage Configuration > Edit Configuration.

• In the Other Settings section, click Configure.

• From the Enable/Disable Outlier Detection menu, enable Outlier Detection.

---

Concepts

• System Overview
• Core Concepts
• Networking
• F5 Distributed Cloud - Customer Edge

---