Integrating Distributed Cloud Mesh with Red Hat OCP

When Mesh is deployed outside the OCP cluster, the kubeconfig file is required to be loaded into F5® Distributed Cloud Console (Console) for service discovery to function. The kubeconfig enables Mesh to monitor kube-api, discover OCP services, and discover lifecycle of pods. It is recommenced to create a custom non-expiring kubeconfig with permissions limited to read-only. A kubeconfig file uploaded to Console is secured with Distributed Cloud Services Blindfold.

Note: For simplicity, in this deployment guide, the system:admin kubeconfig file is used instead of a token-based kubeconfig file.

The kubeconfig can be obtained in one of the following ways:

mkdir kubefile

cd kubefile

oc -n openshift-kube-apiserver extract secret/node-kubeconfigs lb-ext.kubeconfig

lb-ext.kubeconfig
lb-int.kubeconfig
localhost-recovery.kubeconfig
localhost.kubeconfig

Depending on your environment setup, use one of the kubeconfig files to log in. This example uses the internal kubeconfig file.

Note: You cannot use the <localhost>.kubeconfig to log in. Use lb-ext.kubeconfig or lb- int.kubeconfig.

• Set the KUBECONFIG environment variable:

export KUBECONFIG=./lb-int.kubeconfig

• Connect to the OCP and verify the node information:

oc get node

The following is a sample output of the above commands:

Figure: OCP Node List

Deploy your apps to OCP using the app manifest file. This example deploys an app called Hipster Shop, whose manifest is hosted here.

Note: Ensure that you update the service type to NodePort for the service you want Mesh to advertise. This is because the Mesh node is deployed outside the OCP cluster.

• Create a namespace in OCP that is same as your application namespace in Console. This example creates a namespace called foobang-chan:

oc create ns foobang-chan

• Deploy the apps:

oc -n foobang-chan create -f hipster.yaml

The following output is returned:

deployment.apps/emailservice created
service/emailservice created
deployment.apps/paymentservice created
service/paymentservice created
deployment.apps/productcatalogservice created
service/productcatalogservice created
deployment.apps/cartservice created
service/cartservice created
deployment.apps/currencyservice created
service/currencyservice created
deployment.apps/shippingservice created
service/shippingservice created
deployment.apps/recommendationservice created
service/recommendationservice created
deployment.apps/checkoutservice created
service/checkoutservice created
deployment.apps/frontend created
service/frontend created
deployment.apps/redis-cart created
service/redis-cart created

• Verify that the services started:

oc -n foobang-chan get svc

The following output is returned:

Figure: Deployed Services in OCP

Note: For OCP to send traffic to a pod, the application service type should be NodePort. In this example, the frontend service is planned to be advertised with Mesh, so its type is NodePort.

Ensure that Mesh is able to reach the OCP kube-api using the server URL present in the kubeconfig. The server URL can be obtained by viewing the contents of the kubeconfig file.

• Use a file reading command such as cat <kubeconfig-file> to view the kubeconfig contents.

Figure: API URL in Kubeconfig

• Log into Console.

• Click Multi-Cloud Network Connect.

• Click Sites > Site List.

Figure: Site List

• Click on your Mesh site to open its Dashboard.

• Click the Tools tab.

• Select Ping from the options of the Select Tool drop-down menu.

• Enter the API URL in the Destination field and then click Call ping.

Note: You can also execute ping using the site local UI. See Site Local UI for more information.

• Click Manage > Service Discoveries.

• Click Add Discovery.

• Set the metadata in the Metadata section.

• In the Where section, select Site from the Virtual-Site or Site or Network menu.

• Select your site from the Reference menu.

• Ensure that you select Site Local Network from the Network Type menu.

• Select K8s Discovery Configuration from the Select Discovery Method menu, and then click Configure.

• In the Access Credentials section, select Kubeconfig from the Select Kubernetes Credentials menu, and then click Configure.

• In the Secret section, ensure that the Text option is selected for the Type and enter the contents of the kubeconfig file in the form available beneath the Type field.

• Click Blindfold.

• Wait until the secret is encrypted and then click Apply.

Note: Use a terminal file reading utility, such as cat, to obtain the contents of the kubeconfig file. Use the kubeconfig file obtained in the previous steps.

• Click Apply and then click Save and Exit to complete creating the discovery object.

Note: For a full set of instructions on Kubernetes-based service discovery, see Service Discovery - K8s.

• Go to Manage > Service Discoveries and confirm that the service discovery object was created and that it lists discovered services under the Services column.

• Click the value displayed in the Services column to view the details of the services discovered.

• Search for the frontend service and note down the service name.

• Ensure that the Service Type value is displayed as NodePort.

• Check the nodes and node states in the OCP cluster:

oc get node

oc get node -o wide

• Confirm that no node has state Not Running or Not Completed:

oc get pod -A | egrep -vi 'Running|Completed'

Enter the following command to approve a pending certificate signing request:

oc adm certificate approve `oc get csr |grep Pending |awk '{print $1}' `

Label node or assigned role for which HugePages is to be configured. Alternatively, you can also configure or label HugePages for all nodes.

These examples show labeling for three worker nodes:

oc label node worker-1.ocp.f5-udf.com node-role.kubernetes.io/worker-hp=

oc label node worker-2.ocp.f5-udf.com node-role.kubernetes.io/worker-hp=

oc label node worker-3.ocp.f5-udf.com node-role.kubernetes.io/worker-hp=

• Verify that the labels created are using the oc get node command.

Note: The worker-1, worker-2, and worker-3 nodes are labelled as worker-hp - which is an arbitrary tag/label/name of choice.

• Create two files named hugepages-tuned-boottime.yaml and hugepages-mcp.yaml.

The following are sample YAML contents for both files:

hugepages-tuned-boottime.yaml

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: hugepages
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Boot time configuration for hugepages
      include=openshift-node
      [bootloader]
      cmdline_openshift_node_hugepages=hugepagesz=2M hugepages=1792
    name: openshift-node-hugepages

  recommend:
  - machineConfigLabels:
      machineconfiguration.openshift.io/role: "worker-hp"
    priority: 30
    profile: openshift-node-hugepages

hugepages-mcp.yaml

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker-hp
  labels:
    worker-hp: ""
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-hp]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker-hp: ""

• Enable HugePages using the newly created YAML files:

oc create -f hugepages-tuned-boottime.yaml

The following output is returned:

tuned.tuned.openshift.io/hugepages created

oc create -f hugepages-mcp.yaml

The following output is returned:

machineconfigpool.machineconfiguration.openshift.io/worker-hp created

Note: See Official OCP Documentation for more information on HugePages configuration.

• Wait for all worker nodes to have Ready status.

• Use the oc get node command to verify the state of nodes.

• Verify that HugePages is enabled. Repeat this command for all the worker nodes.

This example shows the verification command for the worker-1 node:

oc get node worker-1.ocp.f5-udf.com -o jsonpath="{.status.allocatable.hugepages-2Mi}"

Note: Output for the above command must be a non-zero value and should match the HugePages you specified in the hugepages-tuned-boottime.yaml file.

• Verify that the storage class or dynamic Persistent Volume (PV) is configured. This example shows verification for the storage class:

oc get sc

The following output confirms that storage class is configured:

Figure: Storage Class Validation

• Click Multi-Cloud Network Connect.

• Select Manage > Site Management > Site Tokens.

• Click Add site token.

• In the Name field, enter the token name.

• Click Add site token.

• Click > to expand the token details in JSON format and note down the value of the uid field.

Figure: Site Token

• Download the sample manifest template file.

• Edit the configuration in the Vpm section of the manifest file per the following guidelines:

  • In the ClusterName field, type your cluster name.

  • In the Latitude and Longitude fields, type the latitude and longitude values.

  • In the Token field, type the site token.

Figure: Manifest Template

• Save your changes.

• Deploy the Distributed Cloud Services site using the site manifest file.

This example uses the ves-ocp.yaml site manifest file:

oc create -f ves-ocp.yaml

The following is sample output of the above command:

namespace/ves-system created
daemonset.apps/volterra-ce-init created
serviceaccount/vpm-sa created
role.rbac.authorization.k8s.io/vpm-role created
rolebinding.rbac.authorization.k8s.io/vpm-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/ver created
configmap/vpm-cfg created
statefulset.apps/vp-manager created

• Run the following command to watch the progress of the site creation:

watch 'oc -n ves-system get pod -o wide'

• Check the PV and PVC information:

oc -n ves-system get pv

oc -n ves-system get pvc

Note: Log into Console to accept the site registrations. Only vp-manager-0 pod will be displayed as running until registration of the site is approved.

• Log into Console.

• Click Multi-Cloud Network Connect.

• Click Manage > Site Management > Registrations.

• Click the Pending Registrations tab.

• Find your node name and then click the blue checkmark.

• In the form that appears, fill in all required fields with the asterisk symbol (*).

• Click Save and Exit.

The watch command you set earlier will start showing the ver-0 pod as well. This indicates that the site is registered and getting provisioned. Upon successful provision, the status of all pods changes to Running.

It may take a few minutes for the site health and connectivity score information to update.

• Click Sites > Site List.

• Click on your Site name. The Dashboard tab appears, along with many other tabs to inspect your Site.

• Click the Site Status tab to verify the following:

  • The Update Status field has a Successful value for the F5 OS Status section.

  • The Update Status field has a Successful value for the F5 Software Status section.

  • The Tunnel status and Control Plane fields under the RE Connectivity section have up values.

Note: At this point, the site is ready to provide service to apps deployed on OCP. Due to the site pods being inside OCP, you do not need to create service discovery from Console. The pods can query kube-api for services and lifecycle of pods.

Deploy the Hipster shop app using the app manifest. Download the manifest from here.

• Create a namespace with the same name as that of your app namespace in Console:

oc create ns foobang-chan

• Deploy the apps:

oc -n foobang-chan create -f hipster.yaml

Note: When Mesh deploys as pods inside OCP, the frontend service can be of type ClusterIP.

• In Console, click Multi-Cloud App Connect.

• Select your namespace from the drop-down menu. Ensure that you have the namespace with the same name as the namespace for the OCP cluster.

• Click Manage > Load Balancers > Origin Pools.

• Click Add Origin Pool.

• Enter a name in the Metadata section.

• In the Basic Configuration section, click Add Item.

• From the Select Type of Origin Server menu, select K8s Service Name of Origin Server on given Sites.

• Enter a service name in the <servicename>.<namespace> format in the Service Name field.

• Select Site from the Site or Virtual Site menu.

• Select your Mesh site from the Site drop-down menu.

• From the Select Network on the site menu, select Outside Network.

• Click Add item.

• Click Save and Exit.

Note: See the Origin Pools guide for more information.

• Click Manage > Load Balancers > HTTP Load Balancers in your app namespace.

• Click Add HTTP Load Balancer.

• Enter a name in the Metadata section.

• In the Basic Configuration section, enter a domain name in the List of Domain field.

• Enable the Automatically Manage DNS Records option.

• In the Default Origin Servers section, click Add Item.

• From the Select Origin Pool Method drop-down menu, select the origin pool created in the previous step.

• Click Add Item.

• Scroll down and click Save and Exit.

• Click Virtual Hosts > HTTP Load Balancers in your app namespace.

• Click on the load balancer you created in the previous step. The General Monitoring view with the Dashboard tab loads by default.

• Click the Origin Servers tab.

• Verify that the page shows the pod running your apps as the origin server.

Note: When Mesh is deployed as the ingress/egress Gateway outside the OCP, it discovers services using the service discovery created. When deployed as pods inside the OCP, it auto-discovers the services using the service name and namespace name as specified in the origin pool.

Send a request to your domain using a web browser or a utility, such as cURL, and confirm that the request is successful and your web page is loaded or an HTTP response has data with a 200 code.