Continuous Delivery & Verification

Continuous Delivery

F5® Distributed Cloud App Stack integrates with customer’s existing CI/CD pipelines to deploy applications across the fleet, ensuring minimal operational disruption. App Stack Virtual Kubernetes provides a single endpoint to the CI/CD tool, representing the customer’s fleet of sites. Users can use standard deployment manifest files with annotations to select the sites on which the application needs to be deployed.

Figure: App Stack Continuous Delivery

The workflow to integrate Virtual Kubernetes (vK8s) with CI/CD pipelines is as follows

  1. The user creates a Virtual Kubernetes (vK8s) object in their specific namespace. App Stack provides them a kubeconfig file that represents the entire fleet of customer sites in that namespace.
  2. The user then creates segments of the fleet using virtual sites. For example, the user segments the fleet into three virtual sites - dev/test/prod.
  3. Next, the user links the kubeconfig file to their CI/CD tool. The user might choose to have different CI/CD pipelines for dev, test, and production. The user links the same kubeconfig file to each pipeline and leverages annotations in the deployment manifest file to specify the virtual sites (dev/test/prod) on which to deploy applications. This scenario can be visualized in the diagram shown next.
Figure: CI/CD Using vK8s

App Stack provides complete flexibility for users to manage their applications and deployments. For example, users might choose to have three different namespaces for dev, test and prod respectively. In this scenario, they will have a Virtual Kubernetes (vK8s) and Kubeconfig file per namespace representing the three specific segments of their fleet - dev, test, prod. In this scenario, users can link respective kubeconfig file with each of their specific pipelines - dev, test, prod. This can be visualized as shown in the diagram next.

Figure: CI/CD with vK8s and Fleet

Azure DevOps Integration

The next diagram shows an example of the App Stack kubeconfig file linked to the Azure DevOps CI/CD pipeline. The kubeconfig file was provided in the “KubeConfig” field as part of the “kubectl apply” job in the Azure DevOps CI/CD pipeline. Information about the Kubernetes cluster, such as cluster-name, namespace, and the user is derived from the Kubeconfig file and shown in the “Cluster Context” field.

Figure: Azure CI/CD Example

Note: Ensure that you remove the certificate-authority-data entry from the Kubeconfig file.

Continuous Verification (CV)

Virtual Kubernetes (vK8s) is not only used for configuration, but it is also used to aggregate the health across distributed sites in the fleet. This is a really powerful tool compared to other solutions where operators would have to write tooling to get status from each and every site one by one and aggregate the information. Virtual Kubernetes (vK8s) aggregates the health status across all sites, in a single pane of glass, reducing operational complexity for the customer’s operations team. Health status collected includes application deployment status, site health, and application health.

CV of app deployment and site health

The aggregated information on site health shows the distribution of CPU and memory load across the sites. This information is useful to IT or site administrators to identify which sites are being heavily utilized and in danger of running out of resources. Site administrators can set up alerts to be notified when the resource consumption exceeds thresholds. Users use continuous verification to view the status of deployment - how many PODs were deployed or are in-progress or have failed. Once deployed, users can also view the health status of the pod whether it is running, waiting for an image, out of memory, etc.

Figure: Continuous Verification of App and Health

CV of app health

The complexity of troubleshooting rises exponentially when debugging failures in a thousand instances of the application across a thousand sites. Traditionally the user would have to first determine which site and which application’s pod is failing. Next, the user has to log onto the device, and hope that debugging was turned on and debug logs are available for him/her to determine the root cause. And the user would have to compare the logs across healthy and unhealthy pods to determine what went wrong. This manual workflow does not scale beyond a few sites.

App Stack service offers the ability to continuously analyze customer’s application logs to identify differences/anomalies between normal and misbehaving instances of the application. Once anomalies are spotted, App Stack offers the ability to alert on the anomalies so that users get early notifications when the first deviation from normal occurs, reducing mean time to resolve issues. Continuous analysis of customer’s application logs is performed using supervised AI learning models ingesting logs from all the sites where the application is deployed.


The following topics are used by App Stack’s Continuous Delivery and Verification features. Click on each one to learn more.