Create Kubernetes Site

Objective

This guide provides instructions on how to deploy an F5® Distributed Cloud Services Site as a pod on a Kubernetes (K8s) cluster. For more information on sites, see Site.

Deploying a site as a pod on a K8s cluster is supported for the following:

Note: This feature provides basic functionality of launching a Distributed Cloud Services Site on K8s with limited functionality, tools, and observability.

A site deployed as a pod on a K8s cluster acts as a K8s ingress controller with built-in application security. It also enables the F5® Distributed Cloud Mesh (Mesh) features, such as discovery of services of the K8s cluster, publish of other site's services on this site, publish of this site's discovered services on other sites, etc.

Note: A site deployed with Kubernetes only supports Mesh functionalities and does not support F5® Distributed Cloud App Stack. The supported Kubernetes version is 1.21.

Using the instructions provided in this guide, you can perform the following:

  • Deploy a site of the nodes

  • Decommission the site from the K8s cluster


Prerequisites

The following prerequisites apply:

  • A Distributed Cloud Services Account. If you do not have an account, see Create an Account.

  • A deployed K8s cluster. For documentation on deploying:

  • The following requirements are for the managed K8s environment:

  • Managed K8s (EKS/AKS/GKE): minimum 4 vCPUs and 14 GB of memory per node.

  • Minikube: minimum 14 GB of memory per node.

Note: The driver should support HugePages for managed K8s. For example, for macOS you can use the minikube start --driver=<driver> command to start Minikube with the driver that supports HugePages. For example, use the minikube start --driver=virtualbox --memory 14336 command to start Minikube with VirtualBox driver.

  • Kubernetes StorageClass with enabled Dynamic Persistent Volume Provisioner (PVC) with a minimum of 1 GB space.

Note: Use the kubectl get storageclass command to check if dynamic PVC is enabled for your K8s StorageClass. The output with an entry with standard (default) in the NAME column indicates that K8s storage class is enabled with Dynamic PVC.


Configuration

The following image shows the workflow of deploying a site as a pod on an existing K8s cluster:

Configuration Sequence For Site as K8s Pod
Figure: Configuration Sequence For Site as K8s Pod

The following video shows the deployment workflow:


Configuration Sequence

Deploying a site as a pod on an existing K8s cluster requires you to perform the following sequence of actions:

  1. Create a site token.
  2. Prepare a manifest file with the parameters required for site provisioning.
  3. Deploy the site using the kubeconfig of the K8s cluster and the manifest file.
  4. Perform site registration.
  5. Verify that Distributed Cloud Services are running.

Create a Site Token

Create a site token or use an existing token. If you are configuring a multi-node site, use the same token for all nodes.

Step 1: Log into F5® Distributed Cloud Console (Console) and navigate to site tokens.
  • Click Cloud and Edge Sites.

Figure: Console Homepage
Figure: Console Homepage

  • Select Manage > Site Management > Site Tokens.

Figure: Navigate to Site Tokens
Figure: Navigate to Site Tokens

Step 2: Generate a new site token.
  • Click Add site token to create a new token.

Figure: Site Tokens
Figure: Site Tokens

  • In the Name field, enter the token name.

  • In the Description field, enter a description for the token.

  • Click Add site token.

Figure: Site Token Form
Figure: Site Token Form

Step 3: Note down the new token.
  • Find the token previously created or choose an existing token from the list of tokens displayed.

  • Click > to expand the token details in JSON format and note down the value of the uid field.

Figure: UID Field
Figure: UID Field


Prepare the Manifest File

The manifest file contains a YAML schema used for descriptor information to support deployment of Kubernetes for a site.

Step 1: Create the manifest file.

Download the sample Manifest Template.

Step 2: Edit the configuration.
  • Edit the configuration of the manifest file per the following guidelines:

    • In the ClusterName field, type your cluster name.

    • In the Latitude and Longitude fields, type the latitude and longitude values.

    • In the Token field, type the site token.

    • In the replicas field, enter 1 for a single-node site or 3 for a multi-node site.

  • Save your changes.

This image provides a manifest file example:

Manifest
Figure: Manifest Configuration Example

Note: You can also set the cluster name, latitude, and longitude when you register a site.


Deploy Site

Step 1: Deploy the site using the kubeconfig file for the K8s cluster and manifest file.
  • Type kubectl apply -f <manifest>.yml.
kubectl apply -f <manifest>.yml

This example displays the sample output of the command:

namespace/ves-system created
serviceaccount/volterra-sa created
role.rbac.authorization.k8s.io/volterra-admin-role created
rolebinding.rbac.authorization.k8s.io/volterra-admin-role-binding created
daemonset.apps/volterra-ce-init created
serviceaccount/vpm-sa created
role.rbac.authorization.k8s.io/vpm-role created
rolebinding.rbac.authorization.k8s.io/vpm-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/ver created
configmap/vpm-cfg created
statefulset.apps/vp-manager created
service/vpm created
Step 2: Verify that the K8s pod for the site was created.

Verify that the pod with the vp-manager-0 under the NAME column indicates that the site pod was created.

  • Type kubectl get pods -n ves-system -o=wide.

This example displays the sample output of the command with minikube for a single-node site:

NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE       NOMINATED NODE   READINESS GATES
volterra-ce-init-g8bd7   1/1     Running   0          7m15s   172.16.11.6   minikube   <none>           <none>
vp-manager-0             1/1     Running   0          6m19s   172.17.0.7    minikube   <none>           <none>

This example displays the sample output of the command with minikube for a multi-node site:

NAME                     READY   STATUS    RESTARTS   AGE     IP             NODE       NOMINATED NODE   READINESS GATES
volterra-ce-init-sgklz   1/1     Running   0          4m      172.16.11.12   minikube   <none>           <none>
vp-manager-0             1/1     Running   0          3m59s   172.17.0.3     minikube   <none>           <none>
vp-manager-1             1/1     Running   0          3m24s   172.17.0.4     minikube   <none>           <none>
vp-manager-2             1/1     Running   0          3m16s   172.17.0.5     minikube   <none>           <none>
Step 3: Check if the registration request was created.

You can check the request status from the Console or the kubectl command-line tool.

  • To check with Console:

    • Log into Console and then click Cloud and Edge Sites.

    • Click Manage > Site Management > Registrations > Pending Registrations.

  • To check with kubectl:

    • Type kubectl logs vp-manager-0 -n ves-system.
kubectl logs vp-manager-0 -n ves-system

This example provides the sample output of the command:

Current state: PENDING, registration fe5b5157-7066-4052-95f3-5b9d68e072c6 must be manually APPROVED and then automatically ADMITTED. Object status: &StatusType{ObjectStatus:&ves_io_schema4.StatusType{Status:,Reason:,Code:0,},CurrentState:PENDING,};2022-03-09 21:28:17.07967128 +0000 UTC m=+1318333.138793540,, waiting 15.716434773s before next try

Register the Site

After the Distributed Cloud Services Node is installed, it must be registered as a site in Console.

Note: The USB allowlist is enabled by default. If you change a USB device, such as a keyboard after registration, the device will not function.

Single-node Site Registration

Step 1: Navigate to the site registration page.
  • Log into Console and then click Cloud and Edge Sites.

  • Click Manage > Site Management > Registrations.

Step 2: Complete site registration.
  • Under Pending Registrations, find the Cluster Name and then click the blue checkmark.

  • In the form that appears, fill in all required fields with the asterisk symbol (*).

  • Enter a latitude value and a longitude value, if needed.

  • Enter other configuration information, if needed.

  • Click Save and Exit.

Step 3: Check site status and health.

It may take a few minutes for the site health and connectivity score information to update.

  • Click Sites > Site List.

  • Click on your site name. The Dashboard tab appears, along with many other tabs to inspect your site.

  • Click the Site Status tab to verify the following:

    • The Update Status field has a Successful value for the F5 OS Status section.

    • The Update Status field has a Successful value for the F5 Software Status section.

    • The Tunnel status and Control Plane fields under the RE Connectivity section have up values.

Multi-node Site Registration

Step 1: Navigate to the site registration page.
  • Log into Console and then click Cloud and Edge Sites.

  • Click Manage > Site Management > Registrations.

Step 2: Complete site registration.
  • Under Pending Registrations, find the Cluster Name and then click the blue checkmark. You must perform this step for each of the nodes.

Figure: Multi-Node Site Registration
Figure: Multi-Node Site Registration

  • Enter the same values for the following parameters for all the registration requests for each of the nodes:

    • In the Cluster Name field, enter a name for the cluster. Ensure that all master nodes have the same name.

    • In the Cluster Size field, enter 3. Ensure that all master nodes have the same cluster size.

Figure: Multi-Node Site Cluster Size
Figure: Multi-Node Site Cluster Size

  • Enter information for all mandatory fields marked with the asterisk (*) character.

  • Click Save and Exit.

Step 3: Check site status and health.

It may take a few minutes for the site health and connectivity score information to update.

  • Click Sites > Site List.

  • Click on your site name. The Dashboard tab appears, along with many other tabs to inspect your site.

  • Click the Site Status tab to verify the following:

    • The Update Status field has a Successful value for the F5 OS Status section.

    • The Update Status field has a Successful value for the F5 Software Status section.

    • The Tunnel status and Control Plane fields under the RE Connectivity section have up values.


Verify Services

Verify that services were started after site registration.

Type kubectl get pods -n ves-system -o=wide.

kubectl get pods -n ves-system -o=wide

This example provides the sample output of the command:

NAME                           READY   STATUS      RESTARTS   AGE   IP           NODE                                NOMINATED NODE   READINESS GATES
etcd-5779568655-td4wq          2/2     Running     0          41m   10.244.1.4   aks-nodepool1-29573508-vmss000001   <none>           <none>
etcd-defrag-1587094200-58jjb   0/1     Completed   0          33m   10.244.1.5   aks-nodepool1-29573508-vmss000001   <none>           <none>
ver-0                          13/13   Running     3          41m   10.244.0.8   aks-nodepool1-29573508-vmss000000   <none>           <none>
volterra-ce-init-5wsmb         1/1     Running     0          75m   10.240.0.5   aks-nodepool1-29573508-vmss000001   <none>           <none>
volterra-ce-init-sbjqp         1/1     Running     0          75m   10.240.0.4   aks-nodepool1-29573508-vmss000000   <none>           <none>
vp-manager-0                   1/1     Running     2          72m   10.244.1.3   aks-nodepool1-29573508-vmss000001   <none>           <none>

The site appears in Console, where you can deploy Mesh services.


Decommission a Site

Decommissioning a site requires you to deregister the site from Console, and then delete that site.

Step 1: Navigate to the list of registered sites.
  • Log into Console.

  • Click Cloud and Edge Sites.

Figure: Console Homepage
Figure: Console Homepage

  • Click Manage > Site Management > Registrations.

  • Click the Other Registrations tab.

Step 2: Perform decommissioning.
  • Find your site and then click ....

  • Click Decommission.

Figure: Decommission Site
Figure: Decommission Site

  • Click Decommission in the confirmation window to confirm the operation. After a few minutes, the Current State column will display RETIRED.

Figure: Confirm Decommission Site
Figure: Confirm Decommission Site

Step 3: Delete the resources for the site.

In the terminal, use the kubectl tool to delete resources.

Type kubectl delete -f <vpm-manifest>.yml.

kubectl delete -f <vpm-manifest>.yml
Step 4: Delete the decommissioned site.
  • Click Sites > Site List.

  • Find your site from the list and then click ....

  • Click Manage Configuration to open the site edit form.

  • Click Delete Site.

Figure: Delete Site
Figure: Delete Site

  • Click Delete in the confirmation window to complete the operation.

Figure: Confirm Delete Site
Figure: Confirm Delete Site


Perform Scaling for a Site

You can perform a scale-up or scale-down of site nodes by changing the replicas of the vp-manager pods.

Note: Scale-up for the sites also creates the VER pods, as there can be only one VER pod per node.

Step 1: Update the replicas using the kubeconfig file for the K8s cluster.
  • Type kubectl --kubeconfig=<kubeconfig-of-existing-k8s-cluster> edit statefulset/vp-manager -n ves-system.
kubectl --kubeconfig=<kubeconfig-of-existing-k8s-cluster> edit statefulset/vp-manager -n ves-system
  • In the kubeconfig file replicas field, enter a new number.

  • Save the file. The changes are applied automatically.

Step 2: Verify that the new pods have started.
  • Type kubectl get pods -o=wide -n ves-system.
kubectl get pods -o=wide -n ves-system

This example assumes that the replicas are set to 2:

NAME                           READY   STATUS      RESTARTS   AGE   IP            NODE                                NOMINATED NODE   READINESS GATES
etcd-5bcbfc8689-8fx4g          2/2     Running     0          19h   10.244.1.23   aks-nodepool1-29573508-vmss000001   <none>           <none>
etcd-defrag-1587184200-dg765   0/1     Completed   0          42m   10.244.1.52   aks-nodepool1-29573508-vmss000001   <none>           <none>
ver-0                          13/13   Running     2          16h   10.244.0.12   aks-nodepool1-29573508-vmss000000   <none>           <none>
ver-1                          13/13   Running     0          17m   10.244.1.53   aks-nodepool1-29573508-vmss000001   <none>           <none>
volterra-ce-init-5wsmb         1/1     Running     0          26h   10.240.0.5    aks-nodepool1-29573508-vmss000001   <none>           <none>
volterra-ce-init-sbjqp         1/1     Running     0          26h   10.240.0.4    aks-nodepool1-29573508-vmss000000   <none>           <none>
vp-manager-0                   1/1     Running     0          12m   10.244.1.54   aks-nodepool1-29573508-vmss000001   <none>           <none>
vp-manager-1                   1/1     Running     2          15m   10.244.0.14   aks-nodepool1-29573508-vmss000000   <none>           <none>

Concepts


API References