F5 Distributed Cloud - Customer Edge

What is a Customer Edge (CE)

F5 Distributed Cloud Services Customer Edge (CE) is a Kubernetes-based integrated software stack, which is managed centrally via the SaaS Console and can be instantiated in any of your environments:

• public clouds like AWS, Azure, GCP, etc.
• On-premises Data Centers, Remote Branches, or even at the edge.

It is an extension of F5 Distributed Cloud into customer's environment that is centrally managed via SaaS Console, but fully operated by the customer.

There are two flavors of Customer Edge (CE):

Secure Mesh

This flavor of CE is a Layer 3 - Layer 7 integrated high performance networking stack that provides network connectivity, application connectivity, network security, and application security.

Secure Mesh network connectivity and security services includes following functions:

• Layer 3 Routing
• Network Segmentation
• Connect networks directly or through source NAT (SNAT)
• Network ACLs and Firewall policy.
• BGP routing to advertise VIPs within Site

Secure Mesh application connectivity and security services includes the following functions:

• Application and API security policy
• Forward Proxy
• Load Balancer
• Web Application firewall
• API and Service Discovery

AppStack

This flavor of CE is a managed Kubernetes application hosting stack that allows you to deploy, secure, and operate a fleet of applications across the distributed infrastructure such as public clouds, on-premises data centers, or edge. It can scale to a large number of clusters and locations with centralized orchestration, observability, and operations to reduce the complexity of managing a fleet of distributed clusters.

AppStack Distributed Application Management services includes following functions:

• Distributed Infrastructure Management
• Continuous Deployment and Verification
• Identity and Secrets Management
• Distributed Application Management
• Service Discovery
• Cluster Management

Customer Edge Software Architecture

A CE is made up of a cluster of one or more nodes which can be scaled up or down based on load by adding or deleting nodes. Each node is a Linux-based software appliance (appliance is delivered as ISO or deployment spec on k8s) that is deployed in a virtual machine, in a Kubernetes cluster, commodity hardware, or our edge hardware.

High Availability (HA) with Three Nodes

All software components within CE have microservices-based architecture. These are deployed as Kubernetes pods/services and use ETCD to store all its data such as configuration data, state, and metadata.

ETCD is a distributed key-value store. For production deployments that require high availability, ETCD must be run as cluster of multiple nodes. This ETCD cluster needs a quorum to agree on updates to the cluster state. For a cluster with n members, quorum is (n/2)+1. Therefore, a failure tolerance of 1 requires cluster of 3 nodes. This is reason to recommend a minimum of three-node deployment for high availability.

CE Deployment Model

For any CE, there are two deployment models supported.

Single Node

This deployment model is recommended for only test or POC purposes. As the name suggests, in this model only a single instance of Secure Mesh software is deployed in a virtual machine or bare metal.

Multi Node

This deployment model is recommended for production. A three-node cluster is recommended because it provides high availability. In this model, Secure Mesh software is deployed on three separate nodes and are clustered to operate as a single CE site. In case of Cloud Sites, it is recommended to deploy each node of a CE in a separate availability zone (AZ).

CE Registration

For the CE to come up successfully and provide its intended services, it first needs to register itself with F5 Distributed Cloud Global Controller. For this to be a zero-touch provisioning, the tenant user must generate a token when following site creation workflow. This token must then be inserted into the VM as part of cloud init or given as part of deployment spec on launching on k8s.

When the node's software boots up, it is expected to call home to register. For the call home to succeed, there must be at least one interface up with connectivity to the centralized control and management service - this interface is dynamically identified and configured during bootstrap. During call home, the node sends a registration request with the token to identify the tenant and any additional information about the node e.g. hardware info, etc.

CE-RE Connectivity

When a CE is registered successfully, the F5 Distributed Cloud Global Controller will automatically select two Regional Edges (REs) based on public IP address with which CE registration was received. Optionally, users can also specify which REs they want the CEs to connect to during CE creation. The location of RE is determined based on the geo-proximity using the public address, and the closest RE is connected to.

Once these REs are selected, the list of two selected Regional Edges is passed down to the CE as part of the registration approval. Also, the following additional information is sent as part of this approval:

• CE identity and PKI certificates - these certificates are regularly rotated by the system
• Initial configuration to create and negotiate secure tunnels for the F5 Distributed Cloud.

The CE will try to negotiate both IPSec and SSL tunnels to the selected Regional Edges.

Note: IPSec is the preferred approach and will fallback to SSL if IPSec cannot be established. Once the connectivity is established, a secure network fabric is established and distributed control plane in the Regional Edge takes over control of the CE. These tunnels are used for both management, control, and data traffic. Site to Site traffic and Site to public traffic goes over these tunnels. However, the data traffic is under the user control. Users can leverage the F5 Distributed Cloud global network or route Site-to-Site traffic over the directly established tunnels.

---