Release Changelogs
Objective
This document covers:
- New features or functionalities
- Enhancements to existing features or functionalities
- Open issues or known issues
- Fixed issues
Note: This document covers full list of changes each SaaS release introduces. See Release Highlights to view detailed information of a specific set of significant features for each release.
To follow up on the latest improvements and updates, subscribe to RSS Feed.
October 07, 2024
Last Updated: October 07, 2024.
New Features
BRM Data Intelligence (Data Delivery) 3.0
Following are added:
- Data Dictionary (all datasets)
- Allow customer on-boarding
- Add Premium Dataset - Aggregation and Anomaly Detection
- Custom Dataset Templates
Cloud Connect BIGIP Next - Central Manager and Instances to Distributed Cloud
Two new site types (BIG-IP Central Manager and BIG-IP Instance) are introduced in Multi-Cloud Network Connect workspace to onboard the Central Manager and BIGIP Next Instances as sites on F5 Distributed Cloud. Once these sites are created, the site registrations for both Central Manager and BIGIP Instances can be triggered from the Central Manager console. The JWT tokens for these sites must be created and supplied as input to the onboarding form in Central Manager while triggering the site registration. Once the sites are registered, applications deployed behind the BIGIP instances can be accessed by creating Load Balancers on F5 distributed cloud. Use of this feature requires BIG-IP Next and Central Manager version 20.3.1.
Milliseconds in Timestamp for Transactions on Traffic Analyzer Page
Displaying milliseconds in timestamp field for transactions on Traffic Analyzer page, providing greater precision when tracking and analyzing traffic patterns. This enhancement allows customers to better distinguish between events that occur in quick succession, enabling more accurate troubleshooting and performance analysis.
MyF5 Usage Information
MyF5 is updated to present entitlement and usage information for F5 Distributed Cloud Services. Users can quickly access MyF5 from the support dropdown and the Usage Details page. MyF5 provides access to subscriptions, entitlements, and usage information.
Restriction for Listen Ports Removed for Non SLO/SLI VIP Addresses
When an LB is advertised on a CE SLO/SLI interface, certain ports cannot be used as they are reserved for system processes. This restriction was also applied to any user-specified VIP which was incorrect. This is now removed and users can use any listen port while configuring LB with non SLO/SLI VIP.
Adding Line Chart Graph Type to Consumption Report
A new line chart graph type is introduced to enhance consumption report, providing users with a more dynamic and intuitive way to visualize data trends over time.
Sensitive Data Policy and Compliance Detection Enhancement
With this release, you can now select relevant compliance frameworks such as GDPR, HIPAA, CCPA in the new Sensitive Data Policy screen. This feature highlights API endpoints containing sensitive data governed by the chosen frameworks in API discovery, simplifying compliance management and providing enhanced visibility into data exposure within your API Inventory.
JA4 TLS Fingerprints for Client Matching and User Identification
The JA4 TLS Fingerprint can be utilized in User Identification Policies, and Service Policy rules to match clients.
Revised Default Role Names and API Group Names
Some role names and API group names are revised for consistency and clarity in relation to the service names in console. More roles and API groups will be revised by the same standard in upcoming releases. All role names will follow this format: f5xc-<service>-<tier>-<role>
. API group names will follow this format: f5xc-<workspace>-<role>
.
Introducing Requests to Origin Metric to Observe Traffic Patterns
A new metric Requests to Origin
is introduced in the System and Namespace Performance Dashboards. This metric tracks the total number of non-blocked requests which have passed through the load balancer to an origin server. With this enhancement, users will have a clearer view of their app and API traffic as well as of WAAP features.
Editable Fields in Child Tenant Manager Object
The following fields that are part of a Child Tenant Manager
object can now be edited after creating the object:
- Labels
- Description
- Company Name
- Contact Details
- Customer Information
AI Assistant Updated UI
AI assistant UI is updated with expanded prompts and user feedback option. The updated UI has more natural and intuitive interactions. This release also includes many expanded prompts for Multi-Cloud Networking with Secure Mesh Sites and enhanced guidance on Web App and API Protection. Additionally, a user feedback option is added which allows you to rate the quality of the assistant's responses with a thumbs up or thumbs down option. These updates aim to provide a more seamless, secure, and user-friendly experience.
Replaced Term Service with Workspace in Console
The change of term Service
to Workspace
in Console clarifies user experience by aligning terminology across all platform components, specifically in the navigation and onboarding sections of the UI.
API Security Posture now Detects Configurations with GRAPHQL
The API Security Posture now detects configurations where the GraphQL query depth is not limited and where the number of batched GraphQL queries is not restricted. The API Security Posture now detects configurations where the GraphQL query depth is not limited and where the number of batched GraphQL queries is not restricted. A new security enhancement that detects when the depth of GraphQL queries is not limited, which can lead to potential performance degradation or security vulnerabilities. In addition, the system now identifies configurations where the number of batched GraphQL queries is not restricted, helping to mitigate risks from excessively large or complex batched requests that could overwhelm the server.
Expose Tenant Add-on Services as a Metric
This release exposes tenant add-on services as a metric, so that it can be queried and sent to downstream telemetry systems.
Bot Defense Advanced Customers able to view Detailed Bot Policies
Bot Defense Advanced customers will be able to view their detailed bot policies (Endpoint, Allowlist, and Network) and associated bot infrastructure within the console. Customers will be able to view previous policy configurations and versions.
Additional Transaction Details in Bot Defense Traffic Analyzer Report
Additional Transaction details now available in the Bot Defense Traffic Analyzer Report. Users can now click on a transaction in the Traffic Analyzer report to reveal additional fields that are not included in the default view. These fields can be added to the page-level filter for deeper investigation. Additionally, users have the ability to incorporate any of these second-level fields into their transactions table if they choose. This enhancement allows users to access more detailed information without cluttering the default view.
Workspaces Locked Until all Required Services are Subscribed
A feature to lock workspaces until all required services are subscribed. Users will only have access to the About page of a workspace until they enable the necessary services. This feature ensures that all mandatory services are in place for the workspace to function correctly. Applies to all workspaces except the following: Application Traffic Insight, System Management Workspaces (Administration, Audit Logs & Alerts, Billing, Shared Configuration).
Code-based API Discovery
Code-based API discovery for seamless integration with code repositories, allowing users to scan for API endpoints and manage them within the load balancer. This release introduces code-based API discovery, enabling users to create integrations with their code bases (GitHub, GitLab, etc.) to scan and identify API repositories. Once API repositories are selected within the load balancer, the code will be scanned to detect API endpoints. Detected API endpoints will have their schemas and be marked with the source of detection (Code/Traffic/Code & Traffic), providing comprehensive visibility and management of APIs within your infrastructure.
Expanded Alert System API Security Detections
Expanded the alert system to include configurable alerts for API Security detections, including severity-based alerts and API Discovery alerts for Shadow APIs and Unused APIs. The feature enhances the existing alert mechanism by adding new options for API Security alerts. Customers can now configure alerts for API Security Posture detections based on selected severities. Additionally, API Discovery alerts are available for detecting Shadow APIs and Unused APIs allowing for better security management.
Discover, Onboard, and Monitor your Cloud VPCs via Cloud Connect Dashboards
Adding dashboards that will help you discovery your cloud VPCs, then onboard them via Cloud Connect to a Customer Edge (CE) Site. As well as visualize vital network data for the onboarded VPCs. Once VPCs are onboarded, services offered by F5 Distributed Cloud can be extended to the workloads in these VPCs - such as Load Balancing, WAAP, Bot, and more.
Cacheability Metrics to HTTP Load Balancer Dashboard
Adding cacheability metrics to the HTTP Load Balancer dashboard. New metric in the HTTP LB Dashboard: cache-ability
, i.e. a widget that shows what percent of content is cacheable and nons-cacheable, and could benefit from using F5 Distributed Cloud CDN capacities.
Enhanced Customer Edge (CE) Routing Table
The Enhanced Customer Edge routing table aims to immensely simplify our routing table for our users. Customers can simply look at the routing table to understand if the next hop is the Regional Edge or another Customer Edge or can directly distinguish a local route from a remote route. The available route types are BGP, Static, Local and remote site route. Local route refers to routes that are directly connected whereas remote site routes are generally learned via other sites.The feature requires upgrading the CE to the latest F5 Software version.
Support for Assigning Service Credentials to User Groups
Service credentials can now be assigned to user groups. This allows for more granular control and management of access permissions. Service credentials assigned to a user group will automatically have their permissions updated when the associated group has its permissions or roles changed, reducing administrative overhead.
Improved DNS Dashboard
The DNS Dashboard now shows traffic across all DNS zones by default. It is possible to filter on up to 5 DNS zones at the same time. This filtering applies across all DNS widgets of the dashboard, as well as DNS Request Logs.
Enhanced DNSLB Dashboard
Revamped the DNSLB Dashboard, bringing a lot more details at a glance on DNSLB health, and also displaying the health check that failed.
F5 Distributed Cloud CDN Support of WAAP
CDN is adding native support of Web App and API Protection as a preview within the CDN product. For the July release, CDN supports WAF and Service Policies with the remaining security features to be enabled during the subsequent releases. These features can be configured with the CDN workspace and dashboards for individual CDN distributions are available for this release.
Early Access to Customer Edge Serviceability Workflows
Early Access to Customer Edge serviceability workflows for log bundle collection and ability to run site commands directly from Site Tools. Customers can now collect log bundle for Customer Edge Sites directly from the console streamlining log collection workflows while interacting with F5 support. In addition to log bundle collection from Site Tools, customers can now run troubleshooting commands directly from the Site Tools to quickly identify root cause issues. Here is an (early access)[https://docs.cloud.f5.com/docs-v2/multi-cloud-network-connect/reference/ea-sitecli-ref] guide with more details on these serviceability enhancements.
RBAC for Web App Scanning (WAS)
RBAC is now required for Web App Scanning, and must be configured in both Distributed Cloud Console and WAS service. To provide granular access to WAS, Admin, User, and Monitor Roles have been added. Please see the Web App Scanning section of the Distributed Cloud Technical Knowledge Hub for additional details.
On-Prem Deployments of Web App Scanning (WAS)
Web App Scanning is now available for on-premises deployments. App Registration via Microsoft Entra ID is required, and the required Docker images can be obtained from your F5 Technical Account Team. Please see the Web App Scanning section of the Distributed Cloud Technical Knowledge Hub for additional details.
Caveats
The following caveats apply:
-
When multiple spokes are attached to a cloud connect, sometimes spoke status is not updated as per the status in Azure portal. In some cases the vnet marked for retrying are not added in the DB because of that background job is not able to pick it up and update the status, even though its status is updated in Azure portal. As a workaround, one can insert the vnet entry for retrying to make it work. Make sure to modify cloud connect name, uid, vnet marker, etc.
-
Azure network segmentation only supported for CEs with Two Interfaces. After deletion of cloud connects related to Azure CEs with override default route option actual route in the spoke VNets are not being deleted. Users need to delete default route manually.
-
May 2024 release OS (9.2024.8) requires May 2024 release OS. When CE is installed with May 2024 OS (9.2024.8 and later) and older SW than May 2024 release, vK8s pod got stuck in
ContainerCreating
state. Updating to May 2024 SW version, pods will start successfully. -
Cloud alerts can take up to 45 minutes to be displayed. Cloud resources are periodically polled and validated. When an issue is found in the configuration or state of a resource, there are checks and procedures that occur between this initial issue discovery and the propagation of the alert to the client. If certain services in the process are under load, there can be a delay in the time it takes for alert to be verified and displayed.
-
Service responsible for creating status object for cloudlink may initially have an error for DescribeDirectConnectGatewayAttachments. This error will eventually clear in 10-15 min once the job responsible for updating the status runs again. This is due to a performance issue identified in the service responsible for the status object and will be addressed in release.
-
Intra-Cluster connectivity check failed (Node-To-Node) alert is generated on sites that are configured to be in L3 Enhanced Mode with Jumbo Support. Due to a limitation in the forwarding plane of F5DC, intra node communication in a multi node CE site does not work. As a result of this the user sees
Intra-Cluster connectivity check failed(Node-To-Node)
alert on the Alert page. -
Multiple Azure Sites get created sometimes there seems to be a leak for BGP ASN. We are still root causing the leak for ASN, but if customer faces this issue there is a command to clean up ASN which can be run to cleanup leaked resources. This will be fixed for upcoming release.
-
Network connectivity to a node may be lost while using Mellanox PF/VF along with RHEL image based CE. Problem happens because Network Manager assigns same IP on two interface namely vhost0 and physical interface like eth0 causing networking stack to start dropping packets. Only way to recover is by rebooting the node.
-
When multiple origin servers in TCP load balancer (LB) origin pool are setup while one server is serving a TCP request with consecutive requests are expected to go to any other origin server with no active connection but isn't happening. Consecutive requests going to same origin server that has active connection results in TCL LB hash algorithm method
LeastActive
not functioning as expected. -
With UDP Proxy configured and continuous traffic running, additional deletion of endpoint may result with temporary user-traffic loss for a few seconds.
-
TLS parameters of only one of the load balancers (LBs) is applied if multiple LBs share same certificate info. When multiple LBs share same certificate info, in some specific combinations of the configuration, the virtual-host configurations are merged in the backend load balancer configuration. Possibility of TLS parameters of one of the LBs is applied even if they are different between the LBs.
-
Containerized Data Importer only works on a single node CE. Currently Containerized Data Importer only works on a single node CE. When this feature is needed, please avoid to use multi node CE.
-
Direct Connect Status for AWS site won't be create for site's using cloud credential of type assume role. AWS sites that are deployed using cloud_credential, assume role, fail to get Direct Connect Status.
-
L3 focussed mode requires L3 focussed mode enabled on all CEs. L3 focussed mode is not enabled on all CE sites in Site Mesh Group. MTU configured on CE-CE tunnel interfaces would not be consistent. While using this new feature, it is recommended to enable L3 focussed performance mode on all sites participating in Site Mesh Group.
-
Argo pod doesn't come up when CE mode is changed from L7 to L3 or viceversa. Due to known limitation of DPDK, hugepage reservation may fail. Recommended to reboot node to recover.
-
AWS VPC and AWS TGW sites are not getting provisioned if created by cloning other site object. If new aws_vpc or aws_tgw site is created by cloning other site object, system generated labels are also getting cloned. This result in site provisioning failure. Workaround is to remove all system generated cloned labels while creating new site object.
August 13, 2024
Last Updated: August 15, 2024.
New Features
Introducing CE on F5 rSeries (Early Access)
Introducing CE (Secure Mesh Site) on F5 rSeries. With this release you will be able to create a CE site on F5 rSeries running F5OS 1.8.0 and above. F5 5000 and up are supported.
Renamed DDoS Service to Routed DDoS
The DDoS tile on the main home screen had its name changed to DDoS Service
. This is updated to show the intended title, Routed DDoS
.
New Parameters Added to HTTP Header Processing
The following new parameters are added to the existing list of fields that can be added as a header:
$client_ssl_cipher
: Cipher used by client$client_ssl_serial
: Client certificate serial number$client_ssl_issuer
: Client certificate issuer$client_ssl_subject
: Client certificate subject$client_tls_version
: Client certificate TLS version$client_ssl_cert_validity_start
: Client certificate validity start date$client_ssl_cert_validity_end
: Client certificate validity end date
Updating Bot Defense Chart Colors
Creating consistent colors for chart items in Bot Defense. To reduce confusion and enhance your experience, we have standardized the colors for chart items across all widgets and pages. This update ensures a consistent visual representation throughout the application.
Updating Bot Defense Menu
Changing the Bot Defense left-navigation menu ordering. To enhance usability in Bot Defense application, we have reordered the items in the Report section of the left-navigation menu. Frequently used reports are now positioned higher in the list, making it easier for you to access the information you need most often.
Added Validation for HTTP Header Value
The validation prevents users from configuring invalid header values.
Transaction Usage Report Name Updated to Consumption Report
The name of the Transaction Usage Report is changed to Consumption Report. To enhance clarity and better reflect its purpose, we have renamed the Transaction Usage Report
to Consumption Report
. This update aims to provide a clearer understanding of the report's contents and its relevance.
AWS AssumeRole Support in Global Log Receiver
It is now possible to use AssumeRole with a Global Log Receiver S3 target. This comes in addition to the existing Programmatic Access Credentials.
Longer DNS TXT Records Allowed
We removed the previous limitation of 255 characters on TXT records. It is now possible to have TXT records longer - answers will be cut up into 255 characters chunks for transmission to the client.
Retiring Application Infrastructure Protection (AIP) Tile and Functionality
The F5 Application Infrastructure Protection (AIP) tile and functionality are no longer available for use as of the August 13, 2024 release.
Allow Load Balancer on CE to Listen on a Port Range
This feature allows the user to configure one or more port ranges (maximum ports in range is 64) on which the load balancer must listen. This is allowed only for load balancer advertised to CE Site or Virtual Site.
MTLS with Client Certificate
Support MTLS with client certificate optionally absent.
New Time Frame for Default Page Load
Updated the default time frame on page load to the last six hours to improve the performance of page loads. This change ensures faster access to the most recent and relevant data, enhancing your overall user experience.
Maximum Date Range for Bot Defense Advanced Tier Customers
To provide a more responsive experience and reduce timeouts caused by large data volumes, adjusted the maximum date range for our Bot Defense Advanced Tier customers. The new maximum date range is now 14 days.
Data Delivery on Phase 2
Enables customers to configure and manage data sinks to receive Data Intelligence feeds in Google GCS buckets and Microsoft Azure Blob Storage.
New Dataset Support:
- Bot Defense and Data Intelligence - Basic (Mobile): Includes basic elements, device IDs, automation
- Bot Defense and Data Intelligence - Advanced (Mobile): Includes basic elements, device IDs, behavioral and device related elements, automation
- Bot Standard and Data Intelligence
Data Dictionary: Additionally, supporting DI Advanced (Bot Defense based) aggregated features and anomaly detection engine.
Introducing Simplified Secure Mesh Sites (Early Access)
We have introduced a simpler workflow to configure and manage Secure Mesh Sites. With this release, you will be able to create a CE site anywhere. This feature is in Early Access (EA) and can be used for PoC/PoV deployments. This will be made Generally Available (GA) over the next couple of releases. Your old Secure Mesh sites now reside in the Legacy Configurations: Secure Mesh Sites
page within Network Connect.
This release brings in the following features:
- Removal of certified hardware (making it easy to deploy in any provider)
- Flexibility to choose F5 Distributed Cloud Regional Edges (REs)
- Removal of unnecessary user inputs (for example, latitute/longitude for the CE site)
- Support for dynamic interfaces
- Using a single endpoint for registration and upgrades
- Support to deploy CE sites in any region
- Introducing CE support on F5 rSeries (5K and onwards running F5OS-A 1.8.0 and above)
To get started, refer to the documentation here.
Introduction of Custom Parameter Management for Kubelet
Introducing new functionality for managing custom parameters in Kubelet. Users can now add custom parameters with execcli kubelet-add-param < param >
and remove them using execcli kubelet-remove-param < param >
. To view the list of current parameters, use execcli kubelet-get-params
.
Please note that after adding or removing parameters, a restart of the vpm service is required for changes to take effect with the command execcli systemctl-restart-vpm
. Only specific custom parameters are supported, including but not limited to --kube-reserved=cpu=200m, --max-pods=50, and --system-reserved=cpu=100m.
Site Dashboard Enhancement to Show TGW Routing Tables
The Site dashboard is enhanced to show the TGW related routing tables under the TGW
tab. The user can view the routing tables in F5 Distributed Cloud Console itself, instead of switching to the AWS Console.
Orchestrated Cloud Sites Deployment Workflow
This feature allows user to configure admin password as part of site deployment workflow for orchestrated cloud sites. When deploying cloud sites using F5 orchestration, users can now configure admin password for their CE nodes using F5 Distributed Cloud Console.
CSP Routes
The feature allows viewing the Cloud Service Provider Routes (AWS, Azure, GCP) without having to navigate to the CSP Console thus customers can troubleshoot and visualize end to end routing from within F5 Distributed Cloud Console. In addition, AWS TGW tab has been revamped with improved information around TGW attachments, propagations, and route tables.
Announcing F5 Distributed Cloud Web App Scanning
Announcing the release of F5 Distributed Cloud Web App Scanning, an automated penetration testing and reconnaissance solution for web applications and APIs. With this release, we now offer an External Attack Surface Management solution (Recon), and a Dynamic Application Security Testing solution (Scan) to help you discover your exposed apps and APIs and scan them with the purpose of uncovering hidden vulnerabilities (such as SQL injection, cross-site scripting, vulnerable and outdated dependencies, etc.). Using these insights, you can proactively protect and secure your assets using the Web App and API Protection services in F5 Distributed Cloud.
Announcing Threat Mesh as a Service
Threat Mesh is F5 Distributed Cloud's Threat Intelligence provided as a service. Threat Mesh provides F5 Distributed Cloud customers with an additional layer of protection against web application attacks. It leverages cross-customer correlation, i.e correlation of client attacks across different customers, to identify malicious intent of the client. When a client is flagged due to malicious intent by our WAAP decision engines, the client's IP address will be added to the Threat DB. Customers with Threat Mesh enabled are protected as they will get this intelligence update automatically.
Introducing Client-Side Defense Script Inventory Justification Commenting
To assist in demonstrating compliance with the Payment Card Industry Data Security Standard (PCI DSS) v4.0 Requirement 6.4.3, users can now add comments containing written justifications for the specific utility of each script in Client-Side Defense.
Introducing New Sensitive Data Detections
Introducing new sensitive data detections, easier management of custom data types, and shared sensitive data policies between load balancers for enhanced data protection. Introducing new sensitive data detections, easier management of custom data types, and shared sensitive data policies between load balancers for enhanced data protection.
Announcing AI Assistant
AI Assistant helps streamline operations by providing detailed analysis of requests, security events, and customer edge sites with the click of a button. With AI Assistant, you can quickly get additional information about CE Site operational status, potential attack traffic sources, as well as break-downs of WAAP violations, mitigation actions taken, and any recommended configurations or follow-up actions.
Custom Domain Name Support for Existing AWS VPC
For CE deployment in an existing AWS VPC, a custom, non-empty domain is now supported in addition to the currently supported defaults of \*.ec2.internal or \*.compute.internal
.
CE Upgrade Visibility Improvements
The objective behind this feature is to provide more visibility into the CE upgrade process for end users to figure out the progress in terms of completed, in-progress, and upcoming stages. Note that this is only a visibility improvement that doesn't solve any issues with how upgrades are being processed. As part of this feature, end users will have a clear banner when the upgrade seems to be taking longer than expected prompting them to open a support case that is pre-filled for them.
Updates to User Group Deletion Flow with Referenced Object
Caution is advised when deleting user groups utilized by Delegated Access or Managed Service Providers (MSPs). These groups may hold hidden connections within the system, and their deletion could inadvertently disrupt functionalities relied upon by Delegated Access or MSPs. To ensure a smooth removal process, it's crucial to first identify all instances where these groups are referenced. Once all such references are addressed and removed, the user groups can then be safely deleted using the standard procedures.
Introducting Cloud Connect, Connecting Cloud VPCs and VNets to the Cloud
Introducing Cloud Connect! Cloud Connect allows customers to seamlessly discover and connect their Cloud VPCs and VNets to the F5 Distributed Cloud network. This feature is in "Early Adopter (EA)" and will be made "Generally Available (GA)" soon. Cloud Connect allows easy onboarding of customer VPCs/VNets from their cloud accounts. Customers can then apply policies such as l3 networking, application delivery, application security, and more to these VPCs/VNets. In this release, Cloud Connect support is extended to Azure and now supports onboarding AWS VPCs as well as Azure VNets onto a F5 Distributed Cloud - Customer Edge (CE). This functionality is part of F5 Distributed Cloud Network Connect.
CE Image Published to all AWS and Azure Regions
CE images previously unavailable in some regions are now made available in all AWS and Azure, Gov and regular/non-Gov regions.
F5 Distributed Cloud CDN Support for WAAP
CDN is adding native support of Web App and API Protection as a preview within the CDN product. For this release, CDN supports WAF and Service Policies with the remaining security features to be enabled during the subsequent releases. These features can be configured with the CDN workspace and dashboards for individual CDN distributions are available for this release.
Global Log Receiver Now Supports DNS Request Logs
DNS Request Logs can now be sent to all the existing Global Log Receiver targers (e.g., Splunk, Datadog, S3, ...), allowing customers to have their DNS logs fed into their SIEM systems.
Ability to Search Across Multiple Wizard Table Pages
For resources with many records (e.g. DNS zones), it is now possible to search for records and find occurrences across multiple pages (as opposed to results only being displayed for the current page before).
Allow Configuration of Different TLS Key Caching
Allow Configuration of custom, default, or disable TLS key caching. Exposes the configuration in Origin Pool to configure TLS key caching to be able to either choose default number of keys to cache, disable key caching, or choose a custom value for number of keys to be cached.
JA4 TLS Fingerprint Available in Access Logs
To improve troubleshooting, monitoring, and forensics, Access Logs now contains client JA4 fingerprint. JA4 is a method of generating a unique fingerprint for a TLS (Transport Layer Security) connection. This fingerprint is based on specific characteristics of the TLS handshake, the order of cipher suites, extensions, and other parameters that can uniquely identify the client's software or device.
JA4 TLS Fingerprint Used in Auto DDoS Mitigation
To improve DDoS identification and auto-mitigation, JA4 TLS fingerprint is now used to identify clients. JA4 is a method of generating a unique fingerprint for a TLS (Transport Layer Security) connection. This fingerprint is based on specific characteristics of the TLS handshake, the order of cipher suites, extensions, and other parameters that can uniquely identify the client's software or device.
App Stack Customer Edge's Now Supports L4 GPU
App Stack Customer Edge's now supports L4 GPU when running on KVM hypervisor environment with GPU passthrough. App Stack Customer Edges now support Nvidia L4 GPU on KVM with GPU pass through to support inference workloads using L4 GPU. App Stack VMs can deployed on KVM or KVM equivalent environment with L4 GPU passthrough to allow inference workloads on App Stack utilize L4 GPU capabilities.
Fixed Issues
Centralized Controller and Regional Edge
Fixed Issue in Load Balancer Configuration
Fixed issue in Load Balancer configuration that gets uninstalled in data path. Fixed a race condition where Load Balancer configuration can get uninstalled, when multiple Load Balancers share same VIP.
TLS Certificate Expiration Update
TLS Certificate expiration previously shown in browser time, now in UTC. TLC Certificate expiration dates were previously shown in local browser time leading to confusion about the precise date and time of expiration. This has been changed to show UTC to avoid confusion.
Unable to Download Debug-info from Web GUI
Fixed issue where debug-info in a site cannot be downloaded 10 days after the site creation.
Direct Response Body Size Internally Configurable for HTTP and HTTPS Proxies
Earlier the max response body size of HTTP and HTTPS proxies was restricted to 4096 bytes. Now this limit can be increased up to 16384 (16K) bytes. You can also have a maximum of 4 proxies (within a tenant) whose limits can go up to 65636 bytes. If more than 4 proxies need limits more than 16384 bytes, they need to get their quota increased by SRE for their tenant. This configuration is not exposed to user and only internal services creating proxies can increase the direct response body size. On the 4 proxies with higher size limit, this change enables a higher limit on the number of Bot Defense endpoints i.e. approximately 128 endpoints.
Fix TCP Session Setup Failure Due to Flow Revaluation
RE's have multiple bond interfaces on which packets can be transmitted. Due to asymmetry, it's possible that the packet can be transmitted on one bond interface and the reverse traffic from destination comes back on other bond interface, and in such case flow revaluation is triggered to ensure all packets in session are transmitted on same interface. As part of flow revaluation, packets were redirected to wrong RE causing drop. Fix has been added to ensure flow revaluation doesn't change egress RE and session setup and traffic flows fine.
Internal URL Exposed in Alert JSON
Within the JSON of F5DC alerts, various internal URLs were being exposed. No change in functionality however these have now been removed.
Multiple TLS Certificate Alerts Update
Customers received multiple TLS certificate alerts due to the alert status incorrectly cycling between Detected
and Resolved
.
Support Ticket Error Updated
After creating a support ticket some users were seeing extra characters added to the ticket. In the body of the support ticket text line breaks were being presented in the UI as
. This has been corrected.
Fixed Failure in Route Withdrawal on Custom VIP Removal/Uninstallation
When Custom VIP advertised on Site Local Inside network, routes advertised for the VIP were not getting retracted when the VIP is uninstalled. Fixed this by withdrawing routes on uninstall of Custom VIP.
Customer Edge Site
Fix Fragmentation of UDP Packet
Issue: When packet is transmitted to origin server some UDP packets might undergo fragmentation.
Symptoms: MSS can be different between client to Load Balancer (LB) traffic and LB to origin server.
Conditions: When packet is transmitted to origin server some UDP packets might undergo fragmentation, if packets were not fragmented but dropped since payload exceeded MTU.
Fix: Fix to fragment the packet and forward accordingly.
Reflection of Storage Configuration Settings to CE
Issue: Notification loop observed when etcd cluster breaks for some period resulting in high CPU utilization.
Symptoms: When user configures multiple static routes on storage interfaces, only the last one is actually configured on a CE.
Conditions: Issue is seen everytime user configures multiple static routes on storage interfaces.
Fix: Fixed static route configuration to apply all the storage interface static routes in UI.
Node Running at High CPU
Issue: Fix issue when user configures multiple static routes on storage interfaces.
Symptoms: When etcd becomes unreachable for sometime due to network issue, ver creates a new interface object as it doesnt find an existing interface.
Conditions: Results in two copies of object once etcd cluster recovers, causing a notification loop, resulting in CPU hogging.
Fix: Issue is reconciled and any duplicate object is deleted.
Problems with CE Connection with AWS ROSA
Issue: Disabled BFD for CE on k8s on RE side.
Symptoms: BFD was disabled on CE while running on K8s, but was enabled on the connected RE.
Conditions: Causing continuous connection resets from RE side when connection is type SSL.
Fix: Disabled BFD on RE side also for CE on K8s to have a stable connection.
Name Resolution Failure for K8s Service in App CE
Issue: Improve query success rate of name resolution for K8s service.
Symptoms: Name resolution for K8s service fails irregularly on CE.
Conditions: The issue is observed irregularly on a CE where a pod attempts to resolve a K8s service.
Fix: Improve name resolution to retry queries upon failure.
CE IPsec Packet Loss
Issue: TCP session to origin server is source IP and source port translated. Source port is allocated when session is setup and it is released when session is torn down. After the session has been deleted in forwarding plane due to TCP reset or four-way teardown, if CE receives a packet from origin servers belonging to same session which was deleted, then a new session entry was created which was leaking port internally and future TCP session would get affected due to this. TCP packet reception after teardown is rare and happens mostly in race case. Fix ensures that in such scenarios we dont leak ports.
Symptoms: CE ipsec packet loss.
Conditions: If CE receives a packet from origin servers belonging to same session which was deleted.
Fix: Fix ephemeral port leak during session teardown to origin server.
GPU node information missing in UI
CE Node Status now reflects GPU status related information when supported Nvidia GPUs are detected. With this release Nvidia GPU status information is displayed in the UI Node Status dashboard when system detects supported Nvidia GPUs.
Caveats
The following caveats apply:
-
A CE with GPU need to be upgraded to SW version crt-20240618-2805 or higher. Software(SW) upgrade will fail on a CE with GPU because of crashing 2 DaemonSets: nvidia-dcgm-exporter and nvidia-device-plugin-daemonset when using SW version that is lower than crt-20240711-2869 and higher than crt-20240618-2805. Upgrade SW version to crt-20240711-2869 or higher will start these DaemonSets successfully and then resolve failed upgrade.
-
May/2024 release OS (9.2024.8) requires May/2024 release OS. When CE is installed with May/2024 OS (9.2024.8 and later) and older SW than May/2024 release, vk8s pod got stuck in
ContainerCreating
state. Updating to May/2024 SW version, pods will start successfully. -
Cloud alerts propagation from olympus to Akar is somewhere between 35 to 45 minutes, because multiple form factors. The topology job runs for three clouds. It runs tenant based. And the metrics which are needed for the alerts are based on the topology objects, so there's no way to determine which tenant will run and which objects will be created.
-
AWS cloud link olympus operational status failed with Error. The service responsible for creating the status object for cloudlink may initially have an error for
DescribeDirectConnectGatewayAttachments
. This error will eventually clear in 10-15 min once the job responsible for updating the status runs again. This is due to a performance issue identified in the service responsible for the status object and will be addressed in the August release. -
Intra-Cluster connectivity check failed(Node-To-Node) alert is generated on sites that are configured to be in L3 Enhanced Mode with Jumbo Support. Due to a limitation in the forwarding plane of F5 Console, intra node communication in a multi node CE site does not work. As a result of this the user sees
Intra-Cluster connectivity check failed(Node-To-Node)
alert on Alert page. -
When multiple Azure Sites get created sometimes there seems to be a leak for BGP ASN. We are still root causing the leak for ASN but if customer faces this issue there is a command to clean up ASN which can be run to cleanup leaked resources. This will be fixed for upcoming release.
-
Network connectivity to a node may be lost while using Mellanox PF/VF along with RHEL image based CE. Problem happens because NetworkManager assigns same IP on two interface namely vhost0 and physical interface like eth0 causing networking stack to start dropping packets. Only way to recover is by rebooting the node.
-
TCL LB hash algorithm method
LeastActive
not functioning as expected. In a TCP LB origin pool, when multiple origin servers configured and one of the servers is serving a tcp request the next subsequent request, they should go to any other origin server which has no active connection, but currently it not. The subsequent requests going to same Origin server which already has an active connection. -
With UDP Proxy configured and continuous traffic running, addition/deletion of endpoint may result in user traffic disruption for few seconds.
-
Conflicting container name due to migration, causing
CreateContainerError
. Delete container to resolve it. On rare occasions, there may be some containers that are failed to be replaced by Kubernetes during container runtime migration from docker to CRIO. As a result, two pods with same name are trying to start, and are causing the pod to getCreateContainerError
error. To resolve it, delete the problematic pods through kuberneteskubectl delete
or container clicrictl stop && rm
. -
TLS parameters of only one of the Load Balancer (LB) is applied if multiple LBs share the same certificate info. When multiple LBs share the same Certificate info, in some specific combinations of the configuration, the virtual-host configurations are merged in the backend loadbalancer configuration. This has a possibility where the TLS parameters of only one of the LBs are applied (even if they are different between the LBs).
-
Currently Containerized Data Importer only works on a single node CE. When this feature is needed, please avoid to use multi node CE.
-
Direct Connect Status for AWS site wont be created for site's using cloud credential of type assume role. AWS sites which are deployed using cloud_credential of type assume role are failing to get the Direct Connect Status.
-
While using L3 focused mode feature, if L3 focused mode is not enabled on all CE sites in Site Mesh Group, the MTU configured on the CE-CE tunnel interfaces would not be consistent. While using this new feature, it is recommended to enable L3 focused performance mode on all sites participating in Site Mesh Group.
-
Argo pod doesnt come up when CE mode is changed from L7 to L3 or vice versa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
-
If new aws_vpc or aws_tgw site is created by cloning other site object, system generated labels are also getting cloned. This result in site provisioning failure. Workaround is to remove all the system generated cloned labels while creating new site object.
May 28, 2024
Last Updated: July 10, 2024.
New Features
AppStack and Secure Mesh site not displaying Latest Version in Software Version panel
A newly provisioned AppStack and Secure Mesh site with LTS SW version is displaying the latest CRT version instead of latest LTS version. To find out and upgrade to the new LTS version, user go to https://docs.cloud.f5.com/docs/changelog/node-lts-changelog, copy the version number (In lts-20240528-< 4 digit number> format) and paste it into the pop-up SW version text box to upgrade to the correct LTS version.
This release introduces a custom header for Request Start Time
Users can now configure custom headers in the load balancer to include the request start time. As part of the request headers to add
or response headers to add
configuration options, users can specify any custom key and use the value req_start_time
to capture the exact time when the request started. This feature enhances the ability to monitor and log precise timing information for requests.
Consistency in permissions for f5xc-console-user and f5xc-console-admin
To provide a more secure and consistent experience, we have refined multiple default roles. Previously, users with the f5xc-console-user role had the ability to add roles to users, and update/edit roles in user groups using the API. This access will now be exclusively available to users with the f5xc-console-admin role.
Enable source ip persistence quota in origin pool
By default, source ip persistence is disabled and default resource quota for origin_pool.sip_persistence is 0. Tenant can enable/disable source ip persistence in origin pool UI through other settings. The sip persistence quota can be configured to restrict the number of origin pools with sip persistence enabled per tenant. Beyond the configured value, the sip persistence quota resource exhausts and user gets an error message regarding the same.
Renaming the Good Bot traffic type to Benign Bots
To further refine our traffic categories and enhance clarity, we have updated the name of the "Good Bots" traffic type to "Benign Bots." This change is designed to alleviate any customer confusion regarding the classification of what constitutes a Good Bot.
Adding an additional field in the traffic analyzer table for action taken on traffic
We are introducing an extra field in the traffic analyzer table to display actions taken on traffic by XC Bot Defense. This new field will also support filtering for enhanced analysis capabilities.
Improvements in AWS S3 Global Log Receiver target
We have fixed the "compression always enabled" issue, and changed the filename to clarify what kind of logs is being stored.
For HTTP loadbalancers, the route match for a path can now support a regex pattern
While creating Route objects for HTTP Loadbalancers, the "Path Match" option now supports a "Regex" match along with the existing "Prefix" and "Path" match options
Adding a tool tip to indicate the TLS Certificate type and Renewal Status
We created a status object for each load balancer with auto cert and the status object will now show the reason for Auto Generation failure.
The API Security Posture now detects configurations where the GraphQL query size is not limited and where GraphQL endpoints support introspection.
We have introduced a new security enhancement that detects when the size of GraphQL queries is not limited, potentially exposing systems to performance issues or security risks. Additionally, another critical update is the detection of GraphQL endpoints that support introspection. By identifying these, our security posture helps you mitigate risks associated with exposing detailed schema information that could be leveraged in malicious attacks.
Adding 'System' Namespace configurations for Distributed Apps workspace
The Distributed Apps workspace now supports a new namespace called "System", to provide easy configuration and visibility of Managed K8s as well as Service Discovery. Please note that only specific roles having access to 'system' namespace would be able to view this.
Added TCP Proxy Protocol support to convey original connection parameters, such as the client IP address, to the back-end servers for L4 Load Balancers (TCP)
Protocol extensions such as “X-Forwarded-For” header for HTTP require knowledge of the underlying protocol (such as HTTP). For layer 4 applications, F5 Distributed Cloud Load Balancers now support versions 1 (human-readable format) and version 2 (binary format) of the PROXY protocol (PROXY protocol spec), which conveys the original connection parameters, such as the client IP address, to the back-end servers.
Kubernetes 1.29 upgrade.
Kubernetes version will be upgraded to v1.29 during this release.
WAAP Scheduled Reports now support trends
WAAP Scheduled Reports have been enhanced to show trends, that provide context into the change in percentage of attacks detected and mitigated with the previous time period.
Sampling of Packets Dropped via Fast ACL in DDOS
Sampling has been enabled for DDOS related fast ACL. The sample will carry following info - source IP, destination IP, source Port, destination Port, sampled packet, fast-acl and fast-acl-rule name. These logs are visible in Network Logs of Multi Cloud Network.
X-Forwarded-For Header Behaviour Update
The F5XC load balancer will create or append (if it exists) the downstream client IP address to the header "X-Forwarded-For". When the header "X-Forwarded-For" is configured in the user identification, the updated value is currently being used. As part of this feature, the header value received from the downstream client will be used for user identification.
IP Prefix as Selection Criteria for DNS Load Balancing Rule
This release adds the ability to use an IP Prefix as criteria to match incoming DNS queries, and taking DNS Load Balancing decisions based on that IP Prefix. It comes in addition to the existing GeoIP location set and AS Number.
Simplified CE Site Dashboard
The new site dashboard for Customer Edge (CE) sites provides an improved and simplified view of the CE Site. At a high level, some changes include:
- Manage Configuration capability for users to be able to edit the configuration from the dashboard.
- Provides linkages to Topology and Flow Analysis tools.
- Provides improved contextual information such as the type of Site, Infrastructure Provider, as well as the number of Control & Worker nodes.
- Being able to figure out all connectivity from the CE be it connectivity to Regional Edges (RE), other CE sites or Private Connectivity
- Consolidating all Metrics (System, Node, Interface, Pod) under a single view).
- Infrastructure Page which displays all the details about the CE Site Nodes & their relevant interfaces.
The new Site dashboard is available across all CE Site Types. As part of this effort, there are a few page URLs that have been updated, a few pages that have been removed and a few that have been added.
The URLs to the following pages have been updated. This information is only relevant to customers with previous browser bookmarks.
- System metrics
- Node metrics
- Interface metrics
- Flow table
- DHCP
- Status Objects.
Infrastructure page is a new page part of this enhancement which merges the information within the Node & Interface tabs into a single tab.
The list below mentions the pages that have been deleted and where to find the same information in our console.
- Application metrics: available under App Connect on a per Load Balancer basis.
- Nodes tab: This has been merged as part of the infrastructure tab
- Interfaces tab: This has been merged as part of the infrastructure tab
- Site Status: Information is available on the site dashboard and within infrastructure pages of the site.
- Requests: The information is available on a per Load Balancer basis within App Connect under Overview > Performance, and choosing the load balancer of interest.
- Top talkers: There is the Traffic Flows link on the main dashboard that will redirect to Flow Analysis which has the information of top talkers on a site and further capabilities.
- Connections: For Dynamic Reverse Proxy (DRP) monitoring this information is available within App Connect under Overview > Applications there is a tab HTTP Connect & DRP where you can get stats about a particular DRP.
Showing Bot Customers Additional Traffic Types
This feature is designed to provide power users with more detailed traffic insights across all Bot Defense pages by showing more granular traffic types such as allow listed and unevaluated traffic.
Config Drift Support for LB/Origin/Service Policy
Terraform Config drift support is supported for specifically Load Balancer, Origin, Service Policy & IPPrefixSet. Prior to this behavior, customers who have created these resources via F5 Distributed Cloud Terraform Provider and had to make modifications through the UI (out-of-band) had major issues as the Terraform Provider didn't recognize the out-of-band change and kept overriding it. Now with the feature enabled for the mentioned resources, customers using Terraform will be able to detect the out-of-band change, and either modify the Terraform to incorporate this out of band change or have Terraform override the out-of-band change to ensure the infrastructure automation continues to have a single source of truth.
Proactive Discovering of Issues on Cloud Sites
The feature alerts to customer by proactively discovering issues on cloud sites. These include missing security group rules or default routes.
Explicit Annotation in App Stack Deployment Manifest for Wingman
Until now Wingman sidecar is injected by default when customers deploy applications onto App Stack. With this release, Wingman sidecar container will not be injected automatically for workloads in non-F5xc/customer namespaces on CE sites. Although existing workloads in non-F5xc namespace will continue to run wingman even after upgrade until there are some changes in workloads which triggers re-creation of pods. If such workloads have dependency on wingman for App Secrets service, then they must include following annotation in pod template of the workload: "ves.io/wingman-injection-mode": "enable"
.
Hiding of Mobile Reload Header Field
Bot Defense users don't need to configure Reload Headers for their Mobile SDK on the XC UI. This value is being randomly generated per customer now.
Improved Maintenance Banner
This release improves the banner displayed during F5 Distributed Cloud maintenance windows, giving customers more information about the ongoing status.
Console Only Plan for a Select Subset of Customers Leveraging Specific Point Solutions
The XC Console Only Plan is now available, providing access to the XC Console and select capabilities, including UAM and SSO. This plan is ideal for customers who do not require the full portfolio of the XC Base Package. Upgrades to the full XC Base Package are facilitated through F5 sales for expanded feature access. Additionally, the Console Plan permits non-entitled exploration of the XC Portfolio through limited quotas across various functionalities.
Data Intelligence Service
Enable customers to configure and manage data sinks to receive Data Intelligence feeds in Splunk and AWS S3.
IPv6 Functionality for Features on non-orchestrated CEs as Early Acces
IPv6 functionality for non-orchestrated CEs - Baremetal, KVM, VMware can be turned on for testing purposes as part of the early access pilot. For more details on the features and functionality enabled on CEs for IPv6, please refer to the IPv6 Early Access Support for CEs article.
Ability to Save and Share Bot Defense Filters
We are enhancing Bot Defense with a new feature: Filter Save and Share. This allows users to save their customized filter configurations for later use and access filters created by others for insights and analysis. Additionally, these filters will be specific to individual pages for targeted data examination.
DNS Load Balancer Health Check Billing Visualization Improvement
This release improves the DNS LB Health Checks billing page, by giving more details on the utilization and making it more clear what the usage was.
Network Segmentation Support for AWS TGW and On-premises Secure Mesh Sites
A segment is a F5XC enforced network of spoke VPCs, where VPCs in the segment can communicate with each other, and VPCs not in the segment cannot communicate with each other.
Introducing DNS API Groups
This release introduces new specific API groups for DNS management, allowing more granular permissions. There are 3 groups available: monitor, user, admin.
Import BIND DNS configuration
This release adds the ability to import BIND DNS configuration, allowing for an easier onboarding for customers migrating their DNS into F5 Distributed Cloud.
F5 Distributed Cloud Customer Edge Software Long Term Support (LTS)
Customers running LTS version of CE software will get 12 months of support with critical bug fixes and high severity security vulnerabilities will be made available on a regular basis during the 12 months support window. CE LTS software versions will be labeled in the lts-<date>-<build_no>
format while standard CE software version will continue to be labeled as crt-
.
In this release, lts-20240528-0001 will be available for customers looking for 12 months of support, it's recommended to use RHEL OS 9.2024.11 with this LTS software.
Tenant Level Dashboards for WAAP Workspace
The WAAP workspace now supports a new namespace called "System" , to provide visibility into security and performance metrics aggregated across the tenant. Customers would now also be able to export the security, performance metrics and load balancer configuration details in a csv format from the Security and Performance dashboards. These capabilities drastically simplify the configuration review and attack investigation for the user, by providing a single unified view of all load balancers. In addition, The Tenant Search page provides the ability to search for example a source ip or request id across the tenant and navigate to that specific load balancer. Please note that only specific roles would be able to view these dashboards.
CRD Support for Apps on Virtual Kubernetes
Many enterprise web and AI applications, and API gateways rely on k8s Custom Resources(CRs) to pass the initial and run-time configurations to the application controllers on the k8s clusters. F5XC now supports Custom Resource Definitions (CRDs) to be created on managed k8s and be used for CRs that can be created using the vk8s API. This enables customers to configure the CRs once using vk8s API and apply the changes to apps across all clusters in the vk8s. The feature is supported only for vk8s applied to the virtual site containing CE sites. This release only supports CRDs and CRs for Seldon (an platform to deploy AI/ML models) and Ambassador's Emissary ingress.
Workspace Focused Roles Streamlining User Access Management and RBAC Assignment
This release introduces new workspace-focused roles, enhancing User Access Management (UAM) and Role-Based Access Control (RBAC) assignments. Each role is designed to outline clear expectations for user access and the boundaries of what they can and cannot do within those roles. These roles are closely aligned with Services that exist within a Workspace, ensuring that API groups within each Service are accurately reflected in the Workspace roles. The pattern for Workspace Roles and Service API Groups is as follows:
f5xc-<workspace/service name>-monitor
: Read APIsf5xc-<workspace/service name>-user
: Write APIsf5xc-<workspace/service name>-admin
: Privileged APIs
Console UI Enhancements Showing Tunnel Details between CEs in Site Mesh Group
The CE-CE link in an SMG supports multiple tunnels (3 tunnels for 3 node CE to 3 node CE link or 1 node CE to 3 node CE link, and 1 tunnel for 1 node CE to 1 node CE link). This feature brings UI enhancements to show the details of the tunnels on connectivity graphs and views to observe individual tunnel metrics between CEs in a Site Mesh Group and between CE and REs.
Introducing Catalog Page, Workspace Focused RBAC, and Workspaces with Integrated Add-On Services
This release debuts a new Catalog View, offering a streamlined and intuitive interface where users can browse, learn about, and enable a wide range of Add-On Services. Each workspace clearly delineates the add-on services required for its enablement. Additionally, this update introduces workspace-centric RBAC roles, enhancing security and providing precise access control tailored to specific users, personas, or teams. Roles for each workspace are defined with monitor, user, and admin patterns. Add-On Services also adhere to these patterns, with each workspace role incorporating all necessary API groups from the integrated Add-On Services.
Introducing Cloud Connect
Cloud Connect allows easy onboarding of customer VPCs from their cloud accounts. Customers can then apply policies such as l3 networking, application delivery, application security, and more to these VPCs. In this release, Cloud Connect will support onboarding AWS VPCs onto a F5 Distributed Cloud - Customer Edge (CEs). This functionality is part of F5 Distributed Cloud Network Connect.
Note: Cloud Connect allows customers to seamlessly discover and connect their Cloud VPCs to the F5 Distributed Cloud network. This feature is in "Early Adopter (EA)" and will be made "Generally Available (GA)" soon.
KVM Multi NIC Mesh Support for Secure Mesh Certified Hardware
kvm-multi-nic-voltmesh has been added into secure mesh certified hardware list.
Enhanced Header Transformation Configuration with Expanded HTTP Protocol Options
Enhanced the Header Transformation configuration for HTTP/1.1. Header Transformation options now display exclusively when HTTP/1.1 is selected, simplifying the configuration process. Moreover, the HTTP Protocol configuration is now accessible at the Load Balancer level, facilitating the configuration of the Preserve Case option for both request and response headers.
HPE CSI Upgraded to v2.4.2
HPE CSI software has been updated to a new version, specifically version 2.4.2.
Upgrade Webroot IP/URL Reputation SDK to the latest 5.36.3
New release includes functionality to force the SDK to return unused local DB and RTU memory to the operating system and improves handling of SIGPIPE messaging to prevent SDK crashes.
Fixed Issues
Centralized Controller and Regional Edge
Deleted Sites Showing up in Topology
Sites that have been deleted are in rare cases showing up in topology, delete notification problem has been fixed.
Suggest Values are Missing for Global Network When Adding/Modifying a Network Connector
When adding/modifying a network connector under Network connector configuration if options "Direct, Site Local Inside to a Global Network" or "Direct, Site Local Outside to a Global Network" are selected no suggest values are shown for associating global network. This fix would allow the suggest values to be shown.
TLS Certificate Expiration Date is not Updating in Load Balancer View for Custom Certificates
TLS Certificate expiration date is not updating in the Load Balancer view. Updates to the expiration were not being accurately shown in the load balancer view. The expiration is now correctly shown.
Errors While Configuring Secure Mesh Global Network
Some advanced configurations in the Secure Mesh Global sites cause errors. For example, certain configurations when adding a Secure Mesh Site and choosing Select item of Global Virtual Network result in a "Request Error" with no useful details. The handling of this form has been fixed.
Enhancements to AWS CE sites
AWS CE sites will have AWS Network Load Balancer cross-zone load balancing enabled by default now. Data transfer charges will be incurred to customer by AWS when AWS CE multi-node site is configured as per AWS pricing.
Customer Edge Site
DRP - NoHealthyUpstream Burst
Description: A reserved port 6080 is not used anymore for user's data traffic
Symptoms: User's data traffic(getting NAT-ted) being sent on the wire could potentially be dropped if the random source port chosen was 6080
Conditions: The source port for sending NAT-ted traffic can be from a range of ports. In a very rare scenario, port 6080 can also be chosen randomly, due to which , such traffic would then be dropped by the CE
Fix: Port 6080 is now excluded from the range of source-ports to be used while doing NAT of traffic
About SiteCLI Command Issue
Description: Update execcli commands to improve user experience
Symptoms: automic
and kubectl
commands do not work properly and traceroute
command should be tracepath
on RHEL
Conditions: when using RHEL
operating system, automic
and kubelet
commands are not supported in execcli
and traceroute
command is renamed to tracepath
Fix: update execcli
command tree in SiteCLI to reflect corresponding findings
Pod does not Start with "No space left on device" on a Specific Node
Description: In CENTOS 7, /run inode count is equal to half of the number of physical RAM pages, but this value is harcoded to 800k in RHEL 9. It gets full quickly if /run is mounted as hostPath and used in application.
Symptoms: /run
inode becomes full in RHEL 9.
Conditions: In RHEL 9 CE, managed k8s pod mounts and uses /run
.
Fix: /run
is remounted with 63M inodes in RHEL 9 once the OS boots up.
Name Resolution via Internet from Pods Fails in Certain CEs
Description: VER pod not able to resolve DNS after OS reboot, or after DNS update on host it is not reflected to VER pod
Symptoms: VER pod not able to resolve DNS after OS reboot, or after DNS update on host it is not reflected to VER pod
Conditions: This issue would happen after OS reboot, or when user updated the DNS config
Fix: An fix has been implemented to periodically copy the DNS config from the host to VER pod to address this issue.
VER Pod Restarts Irregularly (High)
Description: When calling collect-debug-info
, vpm
writes vega dumps into /run
temporally, which is running on RAM disk. This may cause vega
to OOM.
Symptoms: Vega OOMs during collect-debug-info
.
Conditions: Running low on RAM.
Fix: We have changed this to write into a physical disk instead of ram disk in this version.
Increasing Trend in Memory Usage of CE Nodes - Webroot OOM
Description: Upgrade Webroot IP/URL Reputation SDK to the latest 5.36.3
Symptoms: New release includes functionality to force the SDK to return unused local DB and RTU memory to the operating system and improves handling of SIGPIPE messaging to prevent SDK crashes.
Conditions: NA
Fix: Upgrade Webroot IP/URL Reputation SDK to the latest 5.36.3
Kubernetes 1.28 Removes hostPort in Workload Manifests when hostNetwork is Set to True
Description: Before Kubernetes 1.28, if hostNetwork is set to true and hostPort is not specified, deployments/statefulset/daemonset or other workload management objects will add hostPort inside PodTemplate Object (.spec.template). The adding of hostPort is not present in Kubernetes 1.28 onwards.
Symptoms: A one-time pod restarted when upgrade to Kubernetes v1.28 onwards.
Conditions: HostNetwork is set to true and HostPort is not specified in the k8s manifest.
Fix: No fix is required. You will not see any changes in behaviour of using hostNetwork other than triggering a redeployment; your application functionality will remain as is. This is a one time change and you will only see this behaviour once during this transition.
Caveats
The following caveats apply:
-
- A newly provisioned AppStack and Secure Mesh site with LTS SW version is displaying the latest CRT version instead of latest LTS version. To find out and upgrade to the new LTS version, user go to https://docs.cloud.f5.com/docs/changelog/node-lts-changelog, copy the version number (In lts-20240528-< 4 digit number> format) and paste it into the pop-up SW version text box to upgrade to the correct LTS version.
-
When a new site is created there may be a delay in displaying Topology for the site. Topology for a site is gathered through periodic asynchronous tasks, so this may not be displayed until the next Interval, currently this interval is 5 minutes. Also there is a possibility that the topology could take longer than 5 minutes when the service is overloaded
-
Cloud alerts propagation from olympus to Akar is somewhere between 35 and 45 minutes, because multiple form factors. The topology job runs for 3 clouds. It runs tenant based. And the metrics which are needed for the alerts are based on the topology objects. So there's no way to deterministically say which tenant will run and which objects will be created. And this takes time.
-
The service responsible for creating the status object for cloudlink may initially have an error for DescribeDirectConnectGatewayAttachments. This error will eventually clear in 10-15 min once the job responsible for updating the status runs again. This is due to a performance issue identified in the service responsible for the status object and will be addressed in the july release.
-
Intra-Cluster connectivity check failed(Node-To-Node) alert is generated on sites that are configured to be in L3 Enhanced Mode with Jumbo Support. Detail: ** Due to a limitation in the forwarding plane of F5XC, intra node communication in a multi node CE site does not work. As a result of this the user sees
Intra-Cluster connectivity check failed(Node-To-Node)
alert on the Alert page. -
When multiple Azure Sites are getting created sometimes there seems to be a leak for BGP ASN. We are still root causing the leak for ASN but if customer faces this issue there is a command to clean up ASN which can be run to cleanup leaked resources. This will be fixed for upcoming release.
-
Kubevirt VM subnets created with connect to SLO & isolated interfaces data missing in
All network Interfaces overlay
, no issues in functionality of the kubevirt VM interfaces. -
Network connectivity to a node may be lost while using Mellanox PF/VF along with RHEL image based CE. Problem happens because NetworkManager assigns same IP on two interface namely vhost0 and physical interface like eth0 causing networking stack to start dropping packets. Only way to recover is by rebooting the node.
-
TCL LB hash algorithm method 'LeastActive' not functioning as expected. In a TCP LB origin pool, When multiple origin servers configured and while one of the server is serving a tcp request, the next subsequent request should go to any other origin server which has no active connection. But currently, it is not happening now. The subsequent requests going to same Origin server which already has an active connection.
-
With UDP Proxy configured and continuous traffic running ,addition/deletion of endpoint may result temporary user-traffic loss for few seconds. With UDP Proxy configured and continuous traffic running, addition/deletion of endpoint may result in user traffic disruption for few seconds.
-
Conflicting container name due to migration, causing
CreateContainerError
. Delete container to resolve it. On rare occasions, there may be some containers that are failed to be replaced by Kubernetes during container runtime migration from docker to CRIO. As a result, 2 pods with same name are trying to start and are causing the pod to getCreateContainerError
error. To resolve it, delete the problematic pods through kuberneteskubectl delete
or container clicrictl stop && rm
. -
TLS parameters of only one of the LB is applied if multiple LBs share the same certificate info. When multiple LBs share the same Certificate info, in some specific combinations of the configuration, the virtual-host configurations are merged in the backend loadbalancer configuration. This has a possibility where the TLS parameters of only one of the LBs are applied(even if they are different between the LBs).
-
Containerized Data Importer only works on a single node CE. Currently Containerized Data Importer only works on a single node CE. When this feature is needed, please avoid to use multi node CE.
-
Direct Connect Status for AWS site wont be create for site's using cloud credential of type assume role. AWS sites which are deployed using cloud_credential of type assume role are failing to get the Direct Connect Status
-
While using L3 focussed mode, it is important to have L3 focussed mode enabled on all CEs. While using L3 focussed mode feature, if L3 focussed mode is not enabled on all CE sites in Site Mesh Group, the MTU configured on the CE-CE tunnel interfaces would not be consistent. While using this new feature, it is recommended to enable L3 focussed performance mode on all sites participating in Site Mesh Group.
-
Argo pod doesnt come up when CE mode is changed from L7 to L3 or viceversa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
-
In case of network segmentation, the destination segment match will not work in ADC case for EFP & service policies.
-
The aws_vpc and aws_tgw sites are not getting provisioned if it is created by cloning other site object. If new aws_vpc or aws_tgw site is created by cloning other site object, system generated labels are also getting cloned. This result in site provisioning failure. Workaround is to remove all the system generated cloned labels while creating new site object.
March 26, 2024
Last Updated: March 26, 2024.
New Features
Renaming Field Automation Type to Bot Reason
This change renames the field Automation Type and widget title Reason Code to Bot Reason. This will ensure consistency across F5 Bot products. The underlying data will not be impacted.
Service Policy Custom Rules Support for Invert Match for HTTP Path
This functionality provides flexibility to create advanced match criteria to address specific use cases, as invert match is introduced for HTTP Path, in addition to HTTP Methods and HTTP Headers.
Easy Access to Log Fields Reference Documents via Distributed Cloud Console
Security Analytics and Requests pages in the Console now provide quick access to the reference documents via links. The documents provide explanation of log fields for security events and requests (access logs), enabling users to review and understand the name and description for each field in the log.
JWT Validation Enhancements
The JWT Validation feature is enhanced with the following key updates:
- Mandatory Claim Validation for custom JWT claims
- Enhanced User Identification using JWT claims
- JWT Claim Matchers for Service Policy Rules
These improvements bolster security by providing more granular control and flexibility in authentication and access control.
Advanced Exclusion Criteria for Signatures and Attack Types in WAF Exclusion Rules
Introduced following enhancements to WAF Exclusion rules:
-
Ability to exclude all signature IDs for a specific context
-
Ability to exclude attack types for a specific context
These enhancements enable customers to address specific scenarios and provide more flexibility to tune their WAF policies
New Validation Workflow for Cloud Sites
AWS VPC Site, Azure VNET Site, AWS TGW Site, and GCP VPC Site will have a new status workflow after a Site is created which will verify cloud-specific conditions. If validation fails, user will be able to re-validate after making changes on cloud console or updating cloud Site configuration.
Enhancements to Customer Edge Execcli Utility for Troubleshooting and System/Kernel Tuning to Support 5GC Data Plane NFs
With enhancements to execcli utility, customers can run network and file operation troubleshooting commands to troubleshoot issues related to Site. Additionally, for use-cases where 5GC Network Functions (UPF) require system/kernel tuning for optimized performance, customers can leverage F5 validated system/kernel tuning commands using execcli.
CloudLink Dashboard Support for GCP Monitoring
CloudLink dashboard now includes monitoring for Google Cloud Platform (GCP).
Update to Workspace Tiles Order on the Console Main Menu
As part of enhancing the visibility and accessibility of key services on the F5 Distributed Cloud Console, the most frequently used services are relocated to the forefront of the service list, ensuring a seamless and efficient user experience.
Customer Edge (CE) Intra-Cluster Communication Checks
Intra-cluster communication checks are meant to ensure that the communication between nodes within a Customer Edge (CE) Site is not interrupted. Each node will send ICMP pings towards the other nodes in the cluster every minute. The ICMP will be sourced from the Site Local Outside (SLO) Interface on each node targeting the IP addresses of the SLO on the other nodes. Within a period of 10 minutes, if all the pings were to fail, an alert named Intra-Cluster connectivity check failed (Node-to-Node)
with critical severity will be triggered. For the checks to run successfully, ensure that ICMP is allowed between nodes on the SLO interfaces. Intra-Cluster Communication Checks is supported in all Customer Edge (CE) Site types.
New HTTP/TCP Load Balancer Status Object
If HTTP load balancer and TCP load balancer are created with manual certificates or automatic certificates, then a status object will get populated with more information. A show status workflow is added for http load balancer and tcp load balancer objects in Console.
Kubernetes 1.26 upgrade.
Kubernetes version is upgraded to v1.26. Following deprecation from Kubernetes, the Pod Security Policy (PSP) is deprecated and replaced by Pod Security Admission (PSA). AWS, Azure, and GCP Cloud Container Storage Interface (CSI) is migrated from in-tree CSI to out-of-tree CSI.
Support to Manage Security Posture Vulnerabilities Status
Introduced a dynamic way to track and manage API Endpoint vulnerabilities with the Security Posture Vulnerabilities Change State feature. This feature allows users to categorize vulnerabilities into the following four statuses:
- Open
- Under Review
- Resolved
- Ignored
This categorization aids in identifying new issues, monitoring ongoing reviews, and recognizing resolved items. Once vulnerabilities are addressed or set to ignored, they are automatically moved to the archive tab.
Load Balancer SNAT IP Persistence Between Regional Edge and Customer Origin
Regional Edge load balancer SNAT IP persistence ensures a consistent source IP from the F5 Distributed Cloud network to the origin for a given client session. This feature ensures that the Source IP from the F5 Distributed Cloud load balancer to origin is a consistent F5 Distributed Cloud IP, and remains unchanged for the duration of the session. To enable this functionality, raise a support ticket and specify the load balancer on which you wish to enable it.
Expansion of API Rate Limiting Capabilities
The API rate limiting capabilities are expanded to include advanced request conditions, wider client conditions, and a new duration period option. This update provides more flexibility and control over API usage.
-
Advanced Request Conditions: Users can now implement rate limiting conditions based on specific Query parameters, Headers, or Cookies, allowing for more granular control over API access and usage.
-
Client Condition Enhancement: Improved the way client conditions are defined and managed, making it easier to customize rate limiting rules that match your specific application needs.
-
New Duration Period - Hours: Added "hours" as a new duration period for rate limiting. This option complements existing range of time-based restrictions, providing additional flexibility for managing API traffic.
Enabling SSH management Option for NFV Service
For managing NFV service, if you need SSH access, users can now enable it while creating/managing NFV service. NFV service dashboard also displays SSH command that users can copy and execute.
Documentation for Bot Signatures
New documentation is added for bot signatures along with change logs, see Reference for bot signatures reference document.
Shared Objects for Root CA Certificates
Root CA certificate can be uploaded once in the tenant and shared across load balancers in namespaces. In addition, mutual TLS configuration at the load balancer and origin pool supports ability to select a shared object (certificate).
Customer Edge (CE) Site Manual Mode Deployment for AWS
Manual mode is another method of deploying Customer Edge (CE) Sites that provides greater flexibility and deployment customization that caters to varying customer needs. This feature allows customers to improve control on how they orchestrate their cloud resources, catering to their architectural and security requirements, especially in brownfield environments. In addition, manual mode is an option for a limited set of customers who do not wish to input their Cloud Service Provider (CSP) credentials in any Console. This feature is now available on AWS via the AWS Console and the AWS Terraform Provider.
Azure, AWS, GCP Cloud Site CSI migration to Out-of-Tree CSI
The in-tree Container Storage Interface (CSI) for cloud Sites will be migrated to out-of-tree CSI and this is a one-time migration during upgrade in which, workload will be drained node-by-node. Azure, AWS, GCP cloud Site CSI will be migrated from in-tree CSI to out-of-tree CSI as part of Kubernetes effort to phase out in-tree CSI. Out-of-tree CSI storage class default-csi
will be introduced and in-tree default
storage class will be deprecated. To support workload migration, existing workload using in-tree storage class will continue to function, Kubernetes will internally reroute to out-of-tree CSI.
AppStack Sites Support SR-IOV Interfaces for Container Workloads Using DPDK Driver along with VM Workloads
Customers can now attach SR-IOV interfaces to VM workloads and containers (DPDK-based) workloads running on AppStack Sites that require bare-metal like high performance networking while connecting directly to the underlay network.
API Inventory Management
Introducing API Inventory Management, a feature designed to enhance your API ecosystem by simplifying the management of your API inventory. It allows for easy managing of discovered APIs, marking of non-API discoveries, removal of outdated endpoints, and seamless updates to API schemas. This tool keeps your API inventory organized, current, and secure, catering to your dynamic requirements.
Fixed Issues
Centralized Controller and Regional Edge
Code Optimizations in User Deletion Workflow
Code optimizations are added in user deletion workflow while handling a high number of namespaces.
Azure VNET Site Configuration Enhancements
The Azure Machine Type for Node
field is now required in Azure VNET site configuration.
Site Hardware Changed Alert was Triggered on a Service Restart
In situations where a Centralized Control and Management service restarts, the "Site Hardware Changed" alert is triggered. This alert is responsible for alerting user in situations where the underlying "Node Flavor" or "Certified hardware" changes. This false positive alert condition is fixed.
Customer Edge Site
Deprecate Pod Security Policy (PSP)
Issue: Pod Security Policy is removed, migrate to Pod Security Admission.
Symptoms: NA
Conditions: PSP is removed from k8s version 1.25. If custom PSP is used in managed k8s, it has to be migrated to custom PSA.
Fix: Find out equivalent PSA security standard for the custom PSP and configure custom PSA for managed k8s. See Kubernetes Migration from PSP page for more information.
Status of hpe-csi Pod Changes to ImagePullBackOff
Issue: Fixed an issue that causes registry change on K8s community.
Symptoms: A third party csi driver image could not get pulled.
Conditions: When a specific storage feature enabled from the Console.
Fix: Correct the container registry which will make this application to not depend on the upstream change.
Caveats
The following caveats apply:
-
Intra-Cluster connectivity check failed(Node-To-Node) alert is generated on Sites that are configured to be in L3 Enhanced Mode with Jumbo Support. Due to a limitation in the forwarding plane of F5 Distributed Cloud Services, intra-node communication in a multi-node CE Site does not work. As a result, user will see
Intra-Cluster connectivity check failed(Node-To-Node)
alert on the Alert page. -
Kubevirt VM subnets created with connect to SLO & isolated interfaces data missing in
All network Interfaces overlay
, no issues in functionality of the kubevirt VM interfaces. -
Network connectivity to a node may be lost while using Mellanox PF/VF along with Site deployed using RHEL image. This problem happens because network manager assigns same IP on two interface namely vhost0 and physical interface like eth0, causing networking stack to start dropping packets. Only way to recover is by rebooting the node.
-
With UDP Proxy configured and continuous traffic running, addition/deletion of endpoint may result temporary user-traffic loss for few seconds.
-
On rare occasions, there may be some containers that are failed to be replaced by Kubernetes during container runtime migration from docker to CRI-O. As a result, 2 pods with same name trying to start will cause the pod to get
CreateContainerError
error. To resolve, delete the problematic pods throughkubectl delete
or container CLIcrictl stop && rm
. -
When multiple load balancers share the same certificate information, in some specific combinations of the configuration, the virtual host configurations are merged in the backend load balancer configuration. This results in a possibility where, the TLS parameters of only one of the load balancers are applied, even when they are different between the load balancers.
-
Containerized Data Importer only functions on a single-node Site. If this feature is required, avoid using multi-node Site.
-
AWS Sites deployed using cloud credential of type assume role fail to get the Direct Connect status
-
While using L3 focussed mode feature, if L3 focussed mode is not enabled on all CE Sites in Site Mesh Group, the MTU configured on the Site-to-Site tunnel interfaces will not be consistent. Therefore, it is recommended to enable L3 focussed performance mode on all Sites participating in Site Mesh Group.
-
Argo pod does not come up when Site mode is changed from L7 to L3 or vice-versa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
-
Daemonset needs to be manually deleted and paused during CSI migration. During cloud CSI migration, node drain will happen to detach storage from pods to allow CSI migration to happen without disruption. However, daemonset will not be drained. User is required to manually detach and remove the pod from consuming the storage before start of migration to prevent any data corruption.
January 16, 2024
Last Updated: January 16, 2024
New Features
Default Advanced L7 DDoS Detection and Auto-Mitigation for HTTP Load Balancers
Advanced L7 DDoS detection and auto-mitigation is enabled by default on all existing and new HTTP load balancers to provide default protection for all customer origins against large scale volumetric L7 DDoS attacks. Users will have the option to choose a different mitigation action. However, disabling either detection or auto mitigation capabilities is not supported.
CSRF Policy Configuration per Route
The CSRF policy was defined at HTTP load balancer level and did not provide an option to disable/configure CSRF enforcement for specific match criteria. This release provides the option to configure CSRF policy per-route, which allows overriding the global CSRF policy configuration definition. The feature can be configured under Routes
> Advanced Options
> Security
section.
Enhanced Range for Alert/Audit Logs in Bot Defense
The Alert/Audit Log feature within Bot Defense has been upgraded to support a comprehensive 30-day range, allowing for extended visibility and analysis. This enhancement enables more robust monitoring and investigative capabilities over a full month's period.
Improved Handling of Empty Data States in Bot Defense
Enhanced the handling of empty data states in Bot Defense, incorporating additional indicators and resolution options. This improvement ensures a smoother experience when dealing with unavailable data in widgets and charts.
Enhance Scroll Query APIs (V2) to Use POST Request for Log and Event Access
Enhance Scroll Query APIs (V2) to use POST request to pass the scroll_id
in the body. This resolves the scroll_id
length limitation in Scroll Query (V1) APIs.
Site Debug Information Collector Support for Collecting More Local Data
The collect-debug-info
now gets more data locally for cases where a brain split might happen, and the node is unable to reach the master node. It now collects argo, vega, envoy, kube-proxy, kubelet-proxy, openvpn, voucher from crio directly. In addition, vega tracebuffers, envoy dump, vega db-dump, argo o/p data are also collected locally from crio.
UX Enhancements to Primary Navigation Menu in WAAP Workspace
The navigation menu for Overview
section in WAAP workspace has been updated, to make it easy for users to navigate to security and performance dashboards with few clicks. Users can also switch between performance and security dashboards at a load balancer level, by selecting the dropdown at the top of the page.
Enable Accelerated Networking for Azure Site
Azure Accelerated Networking (SR-IOV) is a native functionality within Azure that our CE sites can benefit from. We can now enable accelerated networking for CE sites deployed in Azure. This option is available at the site level and when enabled will enable Accelerated Networking across all interfaces on all member nodes provided that the selected Azure Virtual Machine supports Accelerated Networking. The default setting for newly created Azure sites is to have accelerated networking enabled. For existing sites, there is no option to turn this feature on. After the site is created, the setting for accelerated networking can not be changed. This applies to all Azure site types (Mesh/Stack) and for ingress as well as ingress/egress modes on recommended or alternate regions. The feature will be available through the Console and through Terraform. In this release, when enabling Accelerated Networking on a virtual machine that does not support it, the user will get the error VMSizeIsNotPermittedToEnableAcceleratedNetworking
during the apply stage suggesting other instance types that support the feature.
The GraphQL Discovery Enhancement to Show GraphQL Endpoint in Native Format
Enhanced the GraphQL discovery process by incorporating the ability to present the GraphQL endpoint in its native format. This enhancement provides application owners with a more intuitive and insightful experience, fostering a deeper understanding of the API structure, ability to download and facilitating streamlined interactions.
Cloud Sites Monitoring Improvement
Added a new state in the site workflow called QUEUED
. The state will be set to QUEUED
when user performs any actions and remains in QUEUED
state until terraform at the backend starts executing user actions (PLAN/APPLY/DESTROY).
Introducing Detailed Events in the Synthetic Monitoring Service
Users can triage issues faster and dive deeper into critical events with Detailed Events
in synthetic HTTP and DNS monitors in the Events
table.
New Action for L7 DDoS Auto Mitigation
L7 DDoS now supports JavaScript Challenge as one of the mitigation options in addition to blocking. This option provides flexibility for customers, to choose an action of their choice to mitigate volumetric DDoS attacks.
AS Number as Selection Criteria for DNS Load Balancing Rule
This release adds the ability to use an ASN (Autonomous System Number) as a criteria to match incoming DNS queries, and taking DNS Load Balancing decisions based on that ASN. It comes in addition to the existing GeoIP location set.
Fixed Issues
Centralized Controller and Regional Edge
Upgrade to Latest Version Option is not Available for Site
By using the old Software version, if user creates a Site, the upgrade option is not showing up to the newer version which is happening because of Maurice version object in old code. This fix in the Maurice code with AvailableVersion
resolve the issue provides the option in Site.
Customer Edge (CE) Site Health Accounts for any Nodes in Not Ready State
Customer Edge health status reported 100% healthy even when if a node in the Site transitions to NotReady
state. With this fix, customers can keep track of the Site health when a node transitions to NotReady
state. This is especially helpful for AppStack Sites.
Fixed Access Issues for Tenants with Tenant Access Policy set
Resolved issue with Incorrect client IP extraction resulting in failure of Tenant access.
Customer Edge Site
Site with Small Disk may Get Disk Pressure
Issue: CE with small disk may get disk pressure.
Symptoms: CE runs with small disk may get disk pressure. It is recommended to use a larger CE to host application data.
Conditions: When any CE running with 40G disk and has been running software upgrade multiple times.
Fix: After every software upgrade, vpm will run a docker system prune to clear up unused docker image. Crio image will get cleared based on K8s garbage collection process.
Argo Issue with Specific Memory Setting
Issue: Fixed an issue where argo may not come up with specific memory setting.
Symptoms: With a specific argo memory setting, argo and socket memory allocation may result in memory init failure due to K8s limits. This issue is only seen at the software upgrade. This release has fixed this issue.
Conditions: When customer starts Software upgrade, argo pod may stay in crashloopbackoff or not became Running state.
Fix: This release has added logic to allocate memory to avoid above issue.
CE Admin CLI Unable to Show VPM Logs for RHEL9
Issue: CE admin CLI was not able to show vpm logs for RHEL9 OS.
Symptoms: The log vpm command will not show any output.
Conditions: When the user logs in to CE admin CLI in RHEL9.
Fix: Improved the log collector code to make it able to read format on RHEL9.
Broken Node Requires Deletion or Decommission
Issue: When the customer node is broken, customer may need to delete or decommission the node. When any node is broken or suspect an issue, delete the node.
Symptoms: Node is broken and stuck.
Conditions: NA
Fix: This release allows the customer to run kubectl delete node command with kubeconfig
Caveats
The following caveats apply:
-
With UDP Proxy configured and continuous traffic running, addition/deletion of endpoint may result temporary user-traffic loss for few seconds.
-
Connectivity issues might be observed in the cluster if SR-IOV VF interface is renamed as eth0 by VF driver in some certified hardware. This issue is not seen in Dell server. In case connectivity issue is noticed in certified hardware, appropriate VF driver udev rules must be created to restore the connectivity.
-
On launching new vK8s pods, it may sometimes take few (10 to 15) minutes to show the correct pod status. Users are expected to wait for that period.
-
On rare occasions, there may be some containers that are failed to be replaced by Kubernetes during container runtime migration from docker to CRI-O. As a result, 2 pods with same name trying to start will cause the pod to get
CreateContainerError
error. To resolve, delete the problematic pods throughkubectl delete
or container CLIcrictl stop && rm
. -
MAC address data for some interfaces is missing in UI for MULTUS pods. However, this does not affect functionality of the pod interfaces.
-
When multiple load balancers share the same certificate information, in some specific combinations of the configuration, the virtual host configurations are merged in the backend load balancer configuration. This results in a possibility where, the TLS parameters of only one of the load balancers are applied, even when they are different between the load balancers.
-
Containerized Data Importer only functions on a single-node Site. If this feature is required, avoid using multi-node Site.
-
AWS Sites deployed using cloud credential of type assume role fail to get the Direct Connect status
-
While using L3 focussed mode feature, if L3 focussed mode is not enabled on all CE Sites in Site Mesh Group, the MTU configured on the Site-to-Site tunnel interfaces will not be consistent. Therefore, it is recommended to enable L3 focussed performance mode on all Sites participating in Site Mesh Group.
-
Argo pod does not come up when Site mode is changed from L7 to L3 or vice-versa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
-
Exporting public endpoint routes happens only from subset of RE Sites instead of all RE Sites. The Endpoints that are advertised with "Where" field as 'Virtual Network' (public virtual network) in RE, were previously advertised from every RE where the discovery succeeded. This behaviour has been modified in such a way that only a subset of RE sites export these routes. If the user requires that traffic reaches endpoints from a particular RE or a specific set of REs, endpoint configuration needs to be modified in such a way that "Where" field is either Site or Virtual Site. Choosing the "Where" field as Site or Virtual Site disables this optimisation and restores the original behaviour.
December 12, 2023
Last Updated: December 12, 2023
New Features
Introducing DNS Monitor response string validation in the Synthetic Monitoring Service.
Users can now validate that the correct DNS records are returned in DNS query answers through their Synthetic DNS Monitors via regular expressions in the receive string
field.
Access Controls for Support Requests by a Managed Tenant User in an Operating Tenant
RBAC-based access controls have been updated to allow users from an Operating Tenant to submit and view support requests when accessing a Managed Tenant with Delegated Access.
Improved Handling of Empty data states in Bot Defense
Refined the handling of empty data states in Bot Defense, incorporating additional indicators and resolution options. This improvement ensures a smoother experience when dealing with unavailable data in widgets and charts.
Support for Seconds in Dashboard Date-Time Picker Filter
Customer can use custom time range filter to review the monitoring data to the seconds level.
Pausing New Signups for Free, Individual, and Teams Plans
F5 Distributed Cloud is pausing new signups for the Free, Individual, and Teams plans. This change removes signup capabilities in the console and the API. This does not impact existing Free, Individual, or Teams plan customers.
Flexible Caching and Purging support in CDN
Distributed Cloud CDN has added support for flexible caching to enable more control over how to cache assets within your CDN Distribution. Distributed Cloud CDN has also added support to provide more granular control when purging content from your cache.
Disable the Creation of new Delegated Domains
With this release, ability to create new Delegated Domains is disabled. This is because it is now possible to leverage F5 Distributed Cloud Primary DNS to do so, and also benefit from having the capacity to manage the content of the DNS zone (in addition to having the DNS records for the HTTP load balancer created).
Improved Visibility when No Data is Present
The empty data states feature ensures that customers have clear visibility, even when data is absent, preventing confusion and improving user experience. It provides intuitive indicators and messages to signify when data is not available, maintaining transparency and facilitating better user understanding.
DNS Query Logs
This release adds the ability to see the DNS query logs in the F5 Distributed Cloud Console, giving customers more visibility on their DNS traffic.
Detect and Label Unused APIs (Zombies) After 45 Days of Inactivity
The API discovery is enhanced to automatically discover APIs that are part of inventory and are inactive for more than 45 days. Unused APIs are labeled in the API Attributes column on the API Endpoints dashboard.
Enhancements to Malicious Users Detection
Malicious User detection is enhanced to consider Bot Defense and Rate-limiting activity from the client, when these features are enabled for the HTTP load balancer, to effectively detect bad actors.
Support Automatic Certificates for Load Balancers Advertised to the Internet on a Customer Edge (CE) Site
This release introduces support for automagtic certificates for load balancers advertised on a Customer Edge (CE) Site. This release will support VIPs that are advertised to the internet via CE Internet VIP option on AWS site. Internet VIP option must be enabled on the Site before you enabled the custom advertise network option on HTTP load balancer.
Validate Authenticity, Integrity, and Expiry of JSON Web Token
As a pivotal component in modern web application security, this feature ensures the integrity of JSON Web Tokens (JWTs), commonly utilized in authentication. By cryptographically verifying incoming JWTs, the platform mitigates the risks of replay attacks and tampering, fortifying your API against unauthorized access. Additionally, JWT Validation prevents requests with expired or invalid tokens, elevating the overall security posture of your application.
Peer Comparison Report Page
A new report that allows customers to see how their protection data compares to that of a cohort of their peers.
DNS Load Balancer Fallback Pool
This release adds the notion of "fallback pool", allowing customers to define a pool that will match if no other pool matches.
Ability to Manipulate Single DNS Records Using the API
It is now possible to create, delete or update single DNS records using the F5 Distributed Cloud Services API. Previously, one had to resend the whole DNS zone to update it, and that is no longer needed.
CloudLink for Customer Edge Sites
First, users provision a virtual connection into public clouds using a network provider of their choice. Then configure a CloudLink to provide required cloud network orchestration such as orchestration of the VIFs, Direct Connect Gateway and association to one or more Customer Edge Sites (CEs) for AWS. Optionally, customers can configure a Private ADN network to establish private connectivity with F5XC Regional Edges (REs) from the Customer Edge Sites (CEs). This release also adds support for GCP, where GCP Cloud Router connectivity is already set up.
Support for GCP network orchestration and Azure is coming soon.
Fixed Issues
Centralized Controller and Regional Edge
HTTP Load Balancer with Automatic Certificate Allows Domains Greater than 64 Characters
Previously, creating load balancers was not allowed if any of the domains is greater than 64 characters. This limitation is removed, and users can create load balancers as long as one of the domain is less than 64 characters in length.
Site Upgrade State Stuck in UPGRADING if Worker Node Added During Upgrade
Issue: Site Upgrade State Stuck in UPGRADING if Worker Node Added During Upgrade
Symptoms: NA
Conditions: During upgrade, if user introduces new worker node to a Site cluster, the Site upgrade status gets stuck in UPGRADING, and the new worker node cannot join the cluster when upgrade is happening.
Fix: This fix resolves the issue where site is stuck in UPGRADING state, and new worker node can subsequently join the cluster once upgrade is completed.
Connection with RE Repeatedly Flapping
Issue: Tunnel flaps are happening because of BFD flaps.
Symptoms: BFD container is getting killed due to OOM. When this happens, the `veth`` interface for communication between BFD and Argo is deleted. New interface gets created upon BFD restart, but it is not updated to Argo.
Conditions: NA
Fix: Fix delivered by ensuring that the interface is not deleted if BFD is killed.
Rewriting Response Headers to Lowercase
Issue: Updating header transformation settings for both upstream and downstream configurations.
Symptoms: NA
Conditions: NA
Fix: Header Transformation settings are applicable only for HTTP 1.1, and this information has been added as a tooltip in the UI. For upstream connections (i.e., request headers), users can configure this option in the origin pool. However, for downstream connections (i.e., response headers), this option is hidden for both HTTPS Automatic Certificate and HTTPS Custom Certificate configurations, requiring users to contact F5 Distributed CLoud Services SRE team for assistance.
Customer Edge Site
VPM Tool File Edit Issue for RHEL
Issue: Some VPM CLI tools used for edit configuration does not work properly with RHEL9 OS.
Symptoms: NA
Conditions: Some VPM CLI tools used for edit configuration does not work properly with RHEL9 OS.
Fix: This release fixed the VPM tool file edit issue.
New Worker Node Became NotReady
Issue: New worker node became NotReady
after adding it to cluster.
Symptoms: NA
Conditions: After adding new worker node to cluster Site, there is a rare chance that the node will stay in a NotReady
status due to the order of process.
Fix: This release has fixed the issue by ensuring the process order.
Debug Info Collector Stops when NFS Not Responding
Issue: When there are any issue with NFS client, debug-info collector will wait and not continue to finish.
Symptoms: NA
Conditions: NA
Fix: This release fixe the issue.
Caveats
The following caveats apply:
-
An extra load balancer is shown in the WAAP performance dashboard. Due to a metrics collection defect, an extra load balancer may be displayed in the WAAP performance dashboard in the health section.
-
PersistentVolume is not created with version
crt-20231106-588
. -
After adding new worker node to CE cluster, there was a rare chance that the node will stay in a
NotReady
status due to the order of process. To temporary recover, restart VPM or reboot any other working node. -
Connectivity issues might be observed in the cluster if SR-IOV VF interface is renamed as eth0 by VF driver in some certified hardware. This issue is not seen in Dell server. Detail: In case connectivity issue is noticed in certified hardware, appropriate VF driver udev rules must be created to restore the connectivity.
-
On launching new vK8s pods, it may sometimes take few (10 to 15) minutes to show the correct pod status. Users are expected to wait for that period.
-
On rare occasions, there may be some containers that are failed to be replaced by Kubernetes during container runtime migration from docker to CRI-O. As a result, 2 pods with same name trying to start will cause the pod to get
CreateContainerError
error. To resolve, delete the problematic pods throughkubectl delete
or container CLIcrictl stop && rm
. -
When multiple load balancers share the same certificate information, in some specific combinations of the configuration, the virtual host configurations are merged in the backend load balancer configuration. This results in a possibility where, the TLS parameters of only one of the load balancers are applied, even when they are different between the load balancers.
-
Containerized Data Importer only functions on a single-node Site. If this feature is required, avoid using multi-node Site.
-
AWS Sites deployed using cloud credential of type assume role fail to get the Direct Connect status
-
While using L3 focussed mode feature, if L3 focussed mode is not enabled on all CE Sites in Site Mesh Group, the MTU configured on the Site-to-Site tunnel interfaces will not be consistent. Therefore, it is recommended to enable L3 focussed performance mode on all Sites participating in Site Mesh Group.
-
Argo pod does not come up when Site mode is changed from L7 to L3 or vice-versa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
November 07, 2023
Last Updated: November 09, 2023
New Features
Update to Terraform Resource Approval Arguments
Argument private_network_name
is added for the Terraform resource volterra_registration_approval
.
Update to API Credentials Managed Through Terraform
API Credentials managed through the Terraform were not deleted during destroy action. This release updates destroy action to delete the API credentials.
Cloud credential Field Order Change for Azure VNET Site
Cloud credential field is moved up and based on this cloud credential, suggest values get populated for existing objects such as Azure virtual network, resource group, etc.
Additional Policy on Availability Zones for AWS Sites
The AWS Sites need additional policy ec2:DescribeAvailabilityZones
. If this is not set, suggest value for the availability zones will not be populated.
General Availability of F5 Distributed Cloud Content Delivery Network (CDN)
The F5 Distributed Cloud CDN service is now out of public preview phase and is generally available.
Azure Site Worker Node Scaling and Updating Load Balancer Backend Pool with Scale Set
This release fixes Azure Site worker node scaling issues and updates load balancer backend pool with scale set.
GCP Site Replace Validation for Subnet/VPC
Fixed GCP configuration validation failure during replace/upgrade. The failure was observed when the Site configuration had new subnet configuration for existing Inside or Outside VPC. This validation is applicable only for new Sites and not existing Sites.
Fixed Failure in Route Withdrawal on Custom VIP Removal/Uninstallation
When custom VIP advertised on Site Local Inside network, routes advertised for the VIP were not getting retracted when the VIP is uninstalled. This is fixed by withdrawing routes on uninstallation of Custom VIP.
Distributed Cloud Services Node Software Support for Dell R650.
Distributed Cloud Services Node software is certified to run on Dell R650 server for new Customer Edge (CE) Sites. See Create Baremetal Site page for the detailed hardware specifications.
Update to Suggested GCP Regions
The following regions are added to the GCP region suggestions:
- southamerica-west1
- northamerica-northeast2
- us-south1
- us-east5
Revamped Security Incidents
Security incidents are updated to provide additional insights such as attackers intent and description which includes attacker source IP address and TLS fingerprint.
Multiple Custom TLS Certificates/Keys per Load Balancer on Customer Edge (CE) Site
HTTP and TCP load balancers now support the ability to refer to more than one custom (Bring Your Own) TLS certificate. You can upload your TLS certificates and intermediate certificate chains to the F5 Distributed Cloud Services platform once, and refer those objects from multiple load balancers. This new capability is available under Manage > Certificate Management section of Web App and API Protection service.
BGP MD5 Authentication
Added support for configuring MD5 password for authenticated BGP sessions while configuring peers in BGP configuration.
Alerts for L7 DDoS Auto Mitigation
A new alert is introduced to provide visibility into the auto-mitigation of L7 DDoS attacks. The alerts are generated in the Console, when an auto mitigation rule is created or deleted. The L7 DDoS auto mitigation feature can be enabled in the DoS Protection section of HTTP Load Balancer.
Supportfor Bond and VLAN Interfaces on Non-Dedicated Physical Interface in Secure Mesh Site
Secure Mesh site with bond Ethernet interfaces and VLAN interfaces is supported both on dedicated physical interface and non-dedicated physical interface. This enhancement does not support ISV and IGW devices.
Generic Webhook for Alert Reciver
Generic webhook provides the ability to send F5 Distributed Cloud alerts to any endpoint of your choice that supports webhooks. The webhook feature can be configured in the Alert Receivers section in the Console.
Export Security and Performance Dashboards to PDFs
Exporting Security and Performance dashboards in WAAP service as PDFs is introduced so that you can archive, print, or distribute them. This makes it easy for you to share the security and performance insights for your namespaces and HTTP load balancers with stakeholders in your organization.
Insights into Workload Traffic Using Flow Analysis
The Flow Analysis tool provides a graphical way to visualize the volume of data flow between your workloads across the F5 Distributed Cloud fabric. You can choose individual entities or get a top ten list based on the amount of data transferred and plot it. You can also gain additional insights using the metadata provided with every node in the graph and the link connecting them. Additionally, you can view and search through individual records in a tabular format.
Introduction of Health Policies in the Synthetic Monitoring Service
Users can now set health policies on HTTP(s) and DNS monitors based off dynamic and static thresholds on response time to determine when applications are unhealthy.
Addition of New Widgets in Authentication Intelligence
The following enhancements are made:
-
Updates one existing dashboard API, friction histogram, and adds new fields
/sr/{version}/dashboard/friction_histogram
. -
Users Login Transactions
-
Recognized Users: device category is private
-
Non-Recognized Users: device category is shared or unknown
-
Total: recognized and non-recognized users
-
Top Reason Codes
-
Time Period: Last 7 days (startTime, endTime)
-
Total Users: 100K (MUD + SH + LOE + LSBH)
-
MUD: 50,000 (MUD)
-
SH: 35,000 (SH)
-
LOE: 10,000 (LOE)
-
LSBH: 5000 (LSBH)
-
Reset Password Without Login Attempt
-
X-axis: date field of epoch in millisecond (actual browser date)
-
Y-axis: users percentage
-
Failed Login Attempt
-
Directly Clicking Forget Password
-
Number of Users: includes failed login attempts and direct attempts to reset the passwords
Deprecate HMAC-MD5 TSIG Algorithm
The hmac-md5
algorithm used for TSIG keys in secondary DNS is insecure, and disallowed in FIPS 140-2. This is deprecated and it is no longer possible to create secondary DNS zones using this algorithm for TSIG. Existing zones using it continue to work, but while editing the zone, it is not possible to save it unless the algorithm is changed to a more secure choice.
AXFR DNS Import (GUI)
This release adds the ability to import DNS zones from a Primary DNS zone, using a zone transfer (AXFR), from the F5 Distributed Cloud Console.
Note: Support using API was released in September 2023 release.
Dynamic Suggestion of Values for Azure Site VNET Fields
F5 Distributed Cloud Platform is enhanced to make synchronous call to find field values such as Existing VNET Resource Group
and Existing VNET Name
from Azure. Ensure that you pre-select credential and region, and in case of VNET name, select the resource group.
Licensing Server Details for APM on Bare Metal App Stack Sites
This release enables users to specify the licensing server details when creating APM on Bare Metal App Stack Sites so that BIG-IP instance can obtain a license.
MSPs are going to run an instance of License Server (BIG-IQ) so that when users are creating APM instances for Bare Metal App Stack Sites they can use it to license the BIG-IP instance. This feature will need a Licensing sever running and configured with licenses and a TCP Load balancer that is pointing to the Licensing server. This TCP load balancer is chosen during the APM creation so that BIG-IP is licensed during creation.
Dynamic Suggestion of Real Values for Config Fields for AWS VPC Site and AWS TGW Site
For AWS VPC Site, support for suggesting real time values for Existing VPC ID, Availability Zones, and NAT GW ID is introduced. For AWS TGW Site, support for suggesting real time values for Existing VPC ID, VPC ID to be used for attachment, TGW ID, and NAT GW ID is introduced. Suggesting real values will work only when you select valid cloud credential with correct access and region in the configuration fields.
Upstream Default Policy to Retry
HTTP load balancer has a "Do not retry" policy added newly. When this option is not chosen, the load balancer will program route with system default retry policy. This default retry policy will retry upstream request once if a 5xx error is seen in the first attempt. This means, load balancer will automatically attempt a retry if the upstream server responds with any 5xx response code, or does not respond at all (disconnect / reset / read timeout).
TCP Load Balancer Port Range Limit Update
TCP load balancer is updated to support up to 1000 port with port range limit.
Distributed Cloud Services Node Software Support for Dell R660.
Distributed Cloud Services Node software is now certified to run on Dell R660 server for new CE sites. See Create Baremetal Site page for the detailed hardware specifications.
DNS Load Balancer Support for SRV RR Type
This release adds the ability to create DNS Load Balancer records of type SRV, bringing customers more flexibility in how they use the DNSLB.
Kubernetes 1.24
F5 Distributed Cloud Services is updated with new Kubernetes main version 1.24. K8s version will automatically upgrade to this version via software upgrade. From this release, the CE Site also starts to use the CRI-O
runtime. You are required to use the OS version 7.2009.45
(or above) before software upgrade. If you are using the old OVA/ISO and building the new CE site, click OS upgrade to bring up the Site.
Note: There are multiple changes from docker to
CRI-O
. For example, pod needs to addNET_RAW
tosecurityContext.capabilities
to be able to run ping command.
GCP Instance types with GPU can now be used to provision Appstack in GCP
GCP Sites now support the ability to spin up instance types with GPU to bring up Appstack on GCP. The following instance types needs to be added into our GCP VPC Site:
- T2D machine types: t2d-standard-4, t2d-standard-8, t2d-standard-16
- A100 GPU machine types: a2-highgpu-1g, a2-highgpu-2g, a2-highgpu-4g
Support for SR-IOV Interface for Container Workloads in AppStack Site
Customers can now attach SR-IOV interfaces to container workloads running on AppStack Sites that require baremetal-like high performance networking while connecting directly to the underlay network.
DNS Dashboards
This release brings observability to the F5 Distributed Cloud Services DNS, with different widgets available to get a better understanding of DNS traffic.
RHEL OS for New Site Deployments
F5 Distributed Cloud Services introduced new Customer Edge OS with RHEL. The OS version will start with 9.2023.xx
. For the new CE Site, use RHEL image with the current latest software version.
Default Region when Opening Bot Dashboard
This feature automatically configures the default region to align with the predominant regions of the protected applications set up in the system.
Fixed Issues
Node Became Not Ready When Adding New Node
Issue: When adding a new node, the nodes changed to NotReady
status.
Symptoms: After adding new worker node to CE cluster, node becomes not ready.
Conditions: After adding new worker node to CE cluster, there is a rare chance that the node will stay in a NotReady
status due to the order of process.
Fix: To temporary recover, restart VPM or reboot any other working node.
NTP Configuration Issue on Customer Edge Site
Issue: NTP Configuration on CE Site is not functioning as configured.
Symptoms: Custom NTP configuration does not work after registration of a Site.
Conditions: Custom NTP can be configured for a Site during registration. There was an issue with the Site that, after registration, NTP server will use the Regional Edge (RE) infra instead of custom NTP configured by the user.
Fix: This release fixed this issue that local configured NTP will take priority.
Update of Labels on Origin Server Does not Work
Issue: Update of labels on origin servers does not work as expected.
Symptoms: Attempt to update labels on origin servers is not working.
Conditions: Addition or removal of labels of origin server of type IP Address or DNS Name was not getting updated in data path.
Fix: This fix ensures data path is updated correctly on edit of labels of origin server.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
Connectivity issues might be observed in the cluster if SR-IOV VF interface is renamed as
eth0
by VF driver in some certified hardware. This issue is not seen in Dell server. In case of connectivity issues in certified hardware, create appropriate VF driver udev rules to restore the connectivity. -
After adding new worker node to CE cluster, there was a rare chance that the node will stay in a
NotReady
status due to the order of process. To temporary recover, restart VPM or reboot any other working node. -
On launching new vK8s pods, it may sometimes take few (10 to 15) minutes to show the correct pod status. Users are expected to wait for that period.
-
On rare occasions, there may be some containers that are failed to be replaced by Kubernetes during container runtime migration from docker to CRI-O. As a result, 2 pods with same name trying to start will cause the pod to get
CreateContainerError
error. To resolve, delete the problematic pods throughkubectl delete
or container CLIcrictl stop && rm
. -
When multiple load balancers share the same certificate information, in some specific combinations of the configuration, the virtual host configurations are merged in the backend load balancer configuration. This results in a possibility where, the TLS parameters of only one of the load balancers are applied, even when they are different between the load balancers.
-
Containerized Data Importer only functions on a single-node Site. If this feature is required, avoid using multi-node Site.
-
AWS Sites deployed using cloud credential of type assume role fail to get the Direct Connect status
-
While using L3 focussed mode feature, if L3 focussed mode is not enabled on all CE Sites in Site Mesh Group, the MTU configured on the Site-to-Site tunnel interfaces will not be consistent. Therefore, it is recommended to enable L3 focussed performance mode on all Sites participating in Site Mesh Group.
-
Argo pod does not come up when Site mode is changed from L7 to L3 or vice-versa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
-
The ISO image with version 7.2009.27 or prior does not have the required package to run Kubernetes v1.24 and CRI-O. Customer deploying node with old ISO image will need to upgrade OS version to 7.2009.45 immediately to complete the node setup and bring site health to healthy state.
-
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
September 12, 2023
Last Updated: September 19, 2023
New Features
Support to Disable Wingman Secrets Management Service
Customers leveraging 3rd party secrets service like HashiCorp Vault can now optionally disable F5 Distributed Cloud Services native secrets management provided by Wingman side car service by disabling side car injection using annotations.
New Diagnosis Command to Debug Network Configuration
New implementation of diagnosis introduced ability to run network configuration check in admin CLI. The command dumps the information on gateway connectivity, nodes connectivity, and information on domains and DNS.
Customizable DNS Refresh Interval for Origin Pool
Users can now set the DNS refresh interval (in seconds) for specific origin server types, granting enhanced control over DNS caching and resolution.
Applying Storage Interface MTU Configuration
The Storage interface MTU setting is appropriately honored, allowing improved performance in external storage connectivity for deployments that support jumbo MTUs.
Trends for Performance Dashboard
Performance dashboard now supports trends for Traffic Overview
and Throughput
widgets. This will enable users to view the change in metrics (up or down) for the selected date time range compared with the previous period, along with the sentiment (positive, negative or neutral). This allows clear understanding of applications performance evolution, over time.
Custom Security Groups for AWS Sites on Existing VPC
This release introduces the capability for AWS Sites located on existing VPCs to utilize user-provided security groups. Users can now specify their own existing security groups for AWS VPC Sites and AWS TGW Sites.
OpenAPI Validation Expansion to Allow Validation for Responses
The OpenAPI Validation feature is expanded to validate not only requests but also responses with precision and confidence. This expansion encompasses the validation of crucial response properties, including Content-Type
, HTTP Headers
, HTTP Body
, and Response Code
.
Enhanced Error Description and Suggested Action for Site Errors
The feature transforms the error output in terraform to a user-friendly error description, and also provides a suggested action based on the error. If the error can not be interpreted, a generic internal error
is displayed, and directs the users to check the error_output
section of terraform_parameters
.
Global Log Receiver Support for Google Cloud Storage.
This release adds support for Google Cloud Storage, allowing users to send F5 Distributed Cloud logs and events to their Google S3-compatible Object Storage.
Extension to DNS Resource Records Types
This release extends support for the following DNS Resource Records (RR) types:
- DNSKEY
- CDNSKEY
- SSHFP
- TLSA
- CERT
- DLV
These records can be created using the API, or through the F5 Distributed Cloud Console with validation forms.
Enhanced Monitoring and Billing Support for MSPs of BIG-IP APM Instances
The Managed Service Providers (MSPs) can use the F5 Distributed Cloud to deploy BIG-IP virtual appliances on their AWS VPC or Bare Metal App Stack Sites for their customers. This feature provides the MSPs with the metrics and alerts allowing them to monitor the health of APM instances and bill their customers based on utilization (maximum number of concurrent APM sessions per instance).
Default Limit on Source IPs for Fast ACL Filtering the Source for DDoS Attack
A limitation of 512 source IPs per tenant is imposed as default. If there is a requirement for more than 512 source IPs, contact support to increase the limits.
AXFR DNS Import (API)
This release adds the ability to import DNS zones from a Primary DNS zone, using a zone transfer (AXFR), using the F5 Distributed Cloud API.
Note: GUI support will be introduced in next releases.
Change of Geo-IP Provider
F5 Distributed Cloud Services changed the Geo-IP provider for the Distributed Cloud platform a more accurate one, providing more detailed information in future releases. In addition, the database is unified with the expanded F5 product portfolio (including BIG-IP), so that Geo-IP decisions made by any F5 product are from a common database. No user action is required for this enhancement, and any existing Geo-IP rules continue to operate as they do now.
DNS Dashboards
This release brings observability to F5 Distributed Cloud DNS, with different widgets available to get a better understanding of DNS traffic.
Automated Threat Briefing Integration with Report Scheduler
The Automated Threat Briefing email is integrated with F5 Distributed Cloud Services Report Scheduler. This enhancement allows customers to seamlessly manage both users and scheduling for this report.
Fixed Issues
Idle Timeout Problem for HTTP Proxy Load Balancers
Issue: The idle timeout not working for HTTP load balancers.
Symptoms: The idle timeout configured at HTTP load balancer is not getting applied.
Conditions: When PortMatch
is specified on HTTP load balancer or route, the idle timeout configured at the load balancer is not getting applied. Instead, default timeout of 30 seconds is used as idle timeout.
Fix: This is fixed to pick the configured value at HTTP load balancer for idle timeout even when the PortMatch
is configured.
Pod Created with a MULTUS Interface Displayed as Down in Console
Issue: Vega is not exporting status metrics for veth-interface
created on bridge,
Symptoms: If a pod is created with a MULTUS interface, interface is showing as Down
on UI.
Conditions: This issue is seen only for the VLAN MULTUS Layer 2 Interfaces.
Fix: This is resolved by exporting the status metrics for the veth-interface
created on the bridge.
Unable to Login to Console
Issue: The Original SSO configuration did not have the proper SCOPE
set.
Symptoms: Users were unable to log into Console due to the improper SCOPE
. The Update account information
was shown to update user information because some of the scopes were not properly set.
Conditions: The user's default scopes
is set with empty value.
Fix: Recommended default scopes (openid
, profile
, email
) are automatically set when configuring SSO. Contact F5 support to update the default scopes in case you can not configure the SSO again. This ensures that the Update account information
form within the Console will not be displayed if the user's ID token already contains claims such as family_name
, given_name
, and email
as per their Identity Provider (IDP) configuration. Ensure that your IDP is configured to include the recommended OIDC scopes (openid
, profile
, email
).
Note: The
profile
and
Mastership Switchover Causing Vega Restart Due to Race Condition
Issue: Vega restarts during mastership switchover due to a race condition.
Symptoms: NA
Conditions: In case of mastership switchover, Vega obtains the MUTEX lock over ETCD, causing a go routine crash.
Fix: This crash is caught using recovery so that Vega does not restart.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
Containerized Data Importer only functions on a single-node Site. If this feature is required, avoid using multi-node Site.
-
There is a limit on number of port and domain combinations in case of port range configuration. For a newly created load balancer, a combination of ports and domains where the number of ports in port range multiplied by number of domains not exceeding 256 is supported. For example, the combination of (4 domains and 64 ports) or (32 domains and 8 ports) is supported.
-
NFV status is intermittently not working, the status fails to show up some times, and this is a known issue being worked on.
-
While using L3 focussed mode feature, if L3 focussed mode is not enabled on all CE Sites in Site Mesh Group, the MTU configured on the Site-to-Site tunnel interfaces will not be consistent. Therefore, it is recommended to enable L3 focussed performance mode on all Sites participating in Site Mesh Group.
-
Argo pod does not come up when Site mode is changed from L7 to L3 or vice-versa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
-
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
August 08, 2023
Last Updated: August 08, 2023
New Features
Ability to Specify a Name for DNS Load Balancer Pool Members
It is now possible to enter a name and description when adding/modifying a member inside a pool. This allows easy identification of resources that are part of a DNS load balancer pool.
Description Field for DNS Records
It is now possible to enter a description for each DNS record in F5 Distributed Cloud Primary DNS. This allows to qualify specific records in a better manner.
Removal of Built-in Policy Rules Page
As part of enhanced RBAC user experience, the built-in policy rules page is removed.
Mandatory SSH Public Key for All Site Configuration
This enhancement mandates all cloud Sites to have SSH public key while creating the Site. This is required to debug any Site which fails before registration.
BIG-IP APM on Bare Metal App Stack Sites
The MSPs can deploy a BIG-IP virtual appliance on their bare metal App Stack Sites, use native BIG-IP UI to configure APM policies tailored based on the requirement, monitor the health of APM instances, and bill customers based on maximum number of concurrent APM sessions per instance.
Introducing Threat Types to Bot Defense
This release introduces a new feature termed Threat Types
, which will be accessible on both the Bad Bot
tab on our monitor page and in the Bad Bot
report page. This feature is designed to help users comprehend the various attack strategies that malicious bots are attempting to deploy.
Trusted Client Rules Enhancement to Skip DDoS Protection
L7 DDoS Protection can be bypassed for one or more clients (identified by IP prefixes) using the trusted client rules.
Support more DNS Resource Records Types
This release adds support for the following DNS Resource Records (RR) types:
- NAPTR
- DS
- CDS
- EUI48
- EUI64
- AFS
- LOC
Those records can now be created using the API and also through the Console, with validation forms.
API Attributes Column for API Endpoints Monitoring
This release introduces enhanced discovery capabilities, along with the introduction of a new API attributes column on the main API endpoints monitoring page. This feature provides improved visibility and monitoring of API endpoints, including the detection of API types such as GraphQL, gRPC, SOAP, XML-RPC, and login endpoints. This helps users proactively identify potential weaknesses in their API endpoints, allowing them to take appropriate actions to mitigate the risks.
Alerts and Notifications Support for Content Delivery Network Service
To configure alerts, go to the CDN service and select Manage
> Alerts Management
. To view any active alerts, go to Notifications
> Alerts
.
F5 Distributed Cloud Bot Defense Mobile SDK Integrator Management
This release introduces Mobile SDK integrator access in F5 Distributed Cloud Console. This requires subscription to enable under the Organization plan. It provides no-code integration of customer’s mobile applications and F5’s mobile SDK and is designed to reduce the friction of the manual integration process. For more information, see Bot Defense.
F5 Distributed Cloud Bot Defense Native BIG-IP, SFCC, Adobe Connector General Available(GA)
This release introduces F5 Distributed Cloud Bot Defense General Available(GA) connector types for Native BIG-IP, SFCC, Adobe with F5 Distributed Cloud Console onboarding and integration. For more information, see Bot Defense.
Fixed Issues
Fix for Route Configuration Update Causing Panic
Issue: Route configuration update causes panic in system
Symptoms: NA
Conditions: This sometimes occurs when the route object configuration is updated with less number of routes than earlier.
Fix: This is fixed and the root cause for this issue was a race condition between two Go routines. One of the Go routines produced data that was consumed by the other. If the consumer runs before, it is resulting in this panic. However, system used to recover from the panic. The issue is fixed so that the panic will not occur.
Bonding Issue in Storage Interface
Issue: When bonding is created with storage interfaces, failover between those bond interfaces does not work
Symptoms: Bonding in storage interfaces does not work
Conditions: It occurs when bonding is created with storage interfaces.
Fix: In this release, this issue is fixed by tuning bonding parameters created by the platform manager.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
There is a limit on number of port and domain combinations in case of port range configuration. For a newly created load balancer, a combination of ports and domains where the number of ports in port range multiplied by number of domains not exceeding 256 is supported. For example, the combination of (4 domains and 64 ports) or (32 domains and 8 ports) is supported.
-
NFV status is intermittently not working, the status fails to show up some times, and this is a known issue being worked on.
-
While using L3 focussed mode feature, if L3 focussed mode is not enabled on all CE Sites in Site Mesh Group, the MTU configured on the Site-to-Site tunnel interfaces will not be consistent. Therefore, it is recommended to enable L3 focussed performance mode on all Sites participating in Site Mesh Group.
-
Argo pod does not come up when Site mode is changed from L7 to L3 or vice-versa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
-
TLS Coalescing will fail between load balancer with inline certificate and load balancer with explicit certificate object, when their certificate contents do not match, even if the certificate from one of these load balancers have newline at the end and other one does not have. The workaround is to edit the configuration and ensure that both certificates either end with newline or both do not have newline at the end.
-
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
July 11, 2023
Last Updated: July 11, 2023
New Features
Direct Connect Status for AWS Site for Assume Role
AWS sites which are deployed using cloud credential of type assume role does not display the direct connect status.
Node OS Version Change when Adding New Node to Multi-Node Cluster
When adding new node to multiple node cluster, if the newly added node's default OS is different from other node on the cluster, based on the timing of process, it may result in the change of existing node OS. This release fixes it so that the newly added node OS version is set to the cluster OS version.
Scheduled WAAP Report for All Namespaces
Scheduled reports now support the ability to generate a report which aggregates WAAP metrics across all namespaces in the tenant.
Support for AWS Sites Using Existing VPC with Route Tables Attached to Outside or Inside Subnet
Deployment of AWS VPC Sites using existing VPC with route tables attached to outside or inside subnet is now supported. This supports only one custom route table attached to all the outside or inside subnets. For Workload subnets, F5 Distributed Cloud will create separate route tables for each subnet, and no custom route table is supported. Custom route table with default route or no default route pointing to internet gateway is supported with this release.
Removal of Debug Users
The ability to create debug users is removed from the F5 Distributed Cloud Console, and any existing debug users are deleted.
Support for MSPs to Deploy and Manage BIG-IP APM VE Using F5 Distributed Cloud Services
MSPs can deploy a BIG-IP virtual appliance on their AWS VPCs, associate it with the AWS TGW Sites for their customers, use native BIG-IP user interface to configure APM policies tailored based on customers' needs, monitor the health of APM instances, and bill their customers based on maximum number of concurrent APM sessions per instance.
Support for IAM Assume Role to Deploy Sites on AWS
Users can create an AWS assume role and delegate to F5 Distributed Cloud Services account. The F5 Distributed Cloud Services uses its own account credentials to assume the role delegated by the user and then deploy the Site in user account. Users are required to request for F5 Distributed Cloud Services account and IAM role details via support ticket.
Automatic Certificate Generation for Load Balancers Advertised on Customer Edge (CE) Sites.
F5 Distributed Cloud can auto-generate certificates for load balancers that are advertised on Customer Edge (CE) Sites. This includes support for load balancers that are advertised on private networks such as Site Local Outside (SLO), Site Local Inside (SLI), and advertised to the internet directly from the CE Site.
Note: DNS domain delegation is not supported in this release.
Enable Native JavaScript Tag Injection for Dedicated Bot Defense and Fraud Protection
Enable native JavaScript Tag injection for both dedicated Bot Defense and Fraud protection under unified route configuration through an HTTP Load Balancer. Please contact Bot Defense and Fraud support teams for dedicated Bot Defense onboarding, route configuration and Fraud protection integration Preview. For more information, see Bot Defense.
Traffic Loss Due to Stale Disovered Endpoint
A race condition in the handling of discovered endpoint delete was resulting in the object being added back as part of Audit. This race condition is now addressed.
New Target for Global Log Receiver
This release adds support for IBM QRadar as target for the global log receiver. This allows customers using those vendors to send their logs more easily, rather than having to use the generic HTTPs endpoint.
Rich Dashboards for Multi-Cloud App Connect
The Multi-Cloud App Connect service gets a refresh with rich dashboards focused around application delivery. Application and network operators can now observe and take action on applications delivered across their multi-cloud network fabric with a dashboard focused on applications and performance.
Export Security Events and Incidents from the Console
Users now have the ability to export security events and incidents (upto 500 each ) from the Distributed Cloud console in csv format, which enables seamless investigation of security logs
Optimization of cloud loadbalancer which is created
We are optimizing the internal cloud loadbalancer deployment. It will be only be created in cases when its needed. In AWS case its created when AllowedVIPPortConfig is defined for inside or outside network and in cases where there are more than 3 nodes in the cluster
Support for Multiple Ports on HTTP and TCP Load Balancers
HTTP and TCP load balancers now provide the ability to configure multiple ports (port ranges), to serve applications which listen on multiple ports.
Display of DNS Load Balancer Healthcheck Failure Details
This release adds more information in the DNS load balancer dashboard about the reason why a health check failed, such as connection refused, or received string not matching the configured string. This gives users more information for troubleshooting.
Changes to RBAC Policies on Various Console Pages
RBAC policies have been changed which may impact a user's ability to access pages or configurations that they previously could access. Locks are displayed on primary navigation entries and other areas in the Console if a user does not have the correct permissions to access them. As a result, user permissions may need to be changed to restore access.
Announcing Trends for Security Dashboards
Security dashboards (namespace and HTTP load balancer) now support trends for metrics such as security events, threat campaigns, IP reputation, etc. This will enable users to view the change in metrics (up or down) for the selected date time range compared with previous time period along with the sentiment (positive, negative or neutral).
Migration of the Delegated Domain Functionality into Primary DNS Management
With this release, the Delegated Domain feature is moved into the Primary DNS section of F5 Distributed Cloud Console. This allows easier management, as it is now possible to have DNS records for an HTTP load balancer automatically created, while letting users manage the content of DNS zone. Automatically created records appear in a new RR set group called x-ves-io-managed
which is in read-only mode.
Fixed Issues
Argo Hotfix 20230606 Does not Support E810 NIC
Issue: Upgrade firmware version of ICE NIC.
Symptoms: NA
Conditions: RE Site/CE Site with ICE NIC does not function with June 06 release until ostree is also upgraded.
Fix: DPDK version was upgraded in June 06 release, and it requires newer 1.7 firmware version for ICE NIC. Newer firmware and kernel driver are now picked to ensure compatibility is maintained.
Unknown Unicast Packet is Sent to Fabric
Issue: Flooding of traffic with Connect to Layer 2 VM configuration.
Symptoms: Traffic congestion may be observed on the SLO/SLI interface due to traffic loop.
Conditions: When a VM is spawned using Connect to Layer 2 feature, promiscuous mode is enabled on interface and in such case, packets with VLAN ID unknown to datapath were forwarded in SLI or SLO VRF causing loop.
Fix: In such case, Argo drops packets with VLAN ID that are not configured in Argo. And in promiscuous mode also, it is ensured that packets are not forwarded in case destination MAC does not match our MAC.
Existing Node OS Upgraded After Adding a Node to the Cluster
Issue: When new node is added to multi-node CE Site, sometimes other node's OS gets upgraded to OS version of the new node.
Symptoms: When new node is added to multi-node CE Site, sometimes other node's OS gets upgraded to OS version of the new node.
Conditions: This sometimes occures when new node is added to multi-node CE Site.
Fix: This is fixed and after the fix, new node's OS is always updated to the same version that the other nodes use.
Retain Basic Configuration in Argo on Soft Reset.
Upon restart, Vega does a soft-reset of Argo and at that point Argo used to delete all configuration, routes and interfaces. Node would not be reachable until vega reprograms the interface and required routes. This is changed now. In the new model, upon soft reset Argo would not delete the bootstrap interface and will cross connect traffic, so that even when Vega doesn't reprogram interface or route, node will still have connectivity.
Issue: Upon restart, a soft-reset of Argo is performed and at that point Argo deletes all configuration, routes, and interfaces. Node will not be reachable until system reprograms the interface and required routes.
Symptoms: Node will not be reachable until system reprograms the interface and required routes.
Conditions: NA
Fix: Upon soft-reset, Argo will not delete the bootstrap interface and cross connects traffic, so that even when system does not reprogram interface or route, node will still have connectivity.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
There is a limit on number of port and domain combinations in case of port range configuration. For a newly created load balancer, a combination of ports and domains where the number of ports in port range multiplied by number of domains not exceeding 256 is supported. For example, the combination of (4 domains and 64 ports) or (32 domains and 8 ports) is supported.
-
NFV status is intermittently not working, the status fails to show up some times, and this is a known issue being worked on.
-
While using L3 focussed mode feature, if L3 focussed mode is not enabled on all CE Sites in Site Mesh Group, the MTU configured on the Site-to-Site tunnel interfaces will not be consistent. Therefore, it is recommended to enable L3 focussed performance mode on all Sites participating in Site Mesh Group.
-
Remove
accept-encoding
header in compression feature is not working. A workaround is to addRemove Request Header
to removeaccept-encoding
header to be sent to upstream. -
Argo pod does not come up when Site mode is changed from L7 to L3 or vice-versa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
-
On existing CSP deployments, custom tags for objects cannot be added/modified. In CSP deployments, increase of custom tags for objects is only supported for new deployments as tags cannot be added/modified on existing deployments.
-
TLS Coalescing will fail between load balancer with inline certificate and load balancer with explicit certificate object, when their certificate contents do not match, even if the certificate from one of these load balancers have newline at the end and other one does not have. The workaround is to edit the configuration and ensure that both certificates either end with newline or both do not have newline at the end.
-
Secure Mesh Site provisioning fails when bonded Ethernet interfaces are specified before Site is registered. The Secure Mesh Site with bond ethernet interfaces is not yet fully supported. Create the Secure Mesh Site with just the list of bond devices but not the corresponding Ethernet interface configurations. Contact the support team for assistance with provisioning the Site with the bonded interfaces.
-
Layer 3 Performance Mode in Azure is supported only for default D flavours (for example,
Standard_D5_v2
). Azure flavours have a different number of CPU NUMA nodes. The B flavours have 1 NUMA node and D flavour has 2 NUMA nodes. The current release is optimised only for D flavours to achieve the best performance. -
NGINX Management Suite APIs
deployments
,agent-certs
, andsubscription
are not supported for usage in the F5 Distributed Cloud Services Platform. -
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
June 06, 2023
Last Updated: June 06, 2023
New Features
Ability to Use an Existing NAT Gateway as Egress in AWS VPC Site.
An existing NAT gateway can now be added as egress gateway in an existing VPC during AWS VPC Site deployment. The user needs to provide existing NAT gateway ID as input during Site creation workflow.
Kubernetes Version Upgrade for vK8s
The Kubernetes version for Virtual Kubernetes is upgraded to v1.23.
Authentication Detection from API Definition
Authentication detection from API definition enhances the visibility and understanding of authentication mechanisms within your API endpoints. This feature allows you to import authentication state and type information from the uploaded inventory OpenAPI specifications (v2/3), and presents it in a clear and intuitive manner in the API endpoint list.
The API Authentication Detection feature provides insights into the authentication status of API endpoints based on uploaded OpenAPI specifications. It identifies the following three states:
Authenticated Endpoints - If the OpenAPI specification references security schemes in the endpoint's operation-level or API-level security requirements, the endpoint is marked as "authenticated". The specific authentication type and location are determined based on the referenced security scheme.
Unauthenticated Endpoints - Endpoints without explicit security requirements at the operation or API level are labeled as "unauthenticated". No specific authentication types or locations are displayed.
Unknown Authentication State - When the OpenAPI spec lacks security schemes, the authentication state of the endpoint is labeled as "unknown".
These capabilities help developers and security analysts understand the authentication requirements of each endpoint, ensuring proper security measures are in place for API integration.
VPM Option to Reduce Debug Information File Size
To reduce the debug information file size, a --terse
option is added to the vpm debuginfo-collector
. With this option, VPM will not include the VER (control plane pod) configuration dump in the debug information.
Auto Setup Mode for PAN External Service
As part of the auto setup mode, the F5 Distributed Cloud Services will be responsible for configuring the username and password on the PAN firewall. After provisioning, user can directly log into the PAN firewall console.
Jumbo Frame Support in L3 Enhanced Performance Mode
On Sites participating in Site Mesh Group (SMG), jumbo mode can be enabled under L3 performance. This will allow jumbo packets to flow through tunnel established between the Sites.
TCP Reset Attack Handling
A reset attack is possible by spoofing a TCP packet with reset flag enabled. With this fix, reset packets are validated by checking that their sequence numbers are in expected window. Flow eviction is done if the above validation succeeds.
Note: This is as described in RFC 5961.
Update DPDK to Release 21.11.3
Upgrade Argo and OpenVPN to use DPDK 21.11.3, along with private fixes. Added support for Mellanox ConnectX-6 Lx interfaces. Updated drivers for Intel IAVF and ICE are included in this upgrade.
Storage Parameter Comment Out to Address a Third Party Product Issue
In this release, the Customer Edge (CE) Site VPM will comment out override_kernel_check
in /etc/containers/storage.conf
to avoid a third party application issue.
Enable Dedicated Bot Defense Preview on Distributed Cloud
Enable dedicated Bot Defense on Distributed Cloud for selected enterprise Customers for Preview. Pilot enterprise customers migration and onboarding will be managed automatically and customers will be able to access the enhanced dashboard in the Console. For more information, see Bot Defense.
Test Connection Option for Global Log Receiver
This release adds the ability to send a test message to a configured log receiver, to make sure the connection works fine and ease the troubleshooting.
Custom Sensitive Data Detection for API Discovery
The custom pattern detector enables you to define unique patterns of characters to search for within API Requests and Responses. You can configure the custom pattern detector to search for and identify personal information based on specific data types or regional requirements. Supported data types include but are not limited to names, addresses, phone numbers, and unique social security numbers. By leveraging custom patterns, you can customize the detection process to align with your specific data protection needs, ensuring compliance with data privacy regulations. This empowers organizations to proactively identify and secure sensitive information traversing APIs.
Disabling DNS Load Balancer Objects
It is now possible to disable DNS load balancer, pool, and health check objects. This allows more flexible operations such as troubleshooting, disaster recovery, etc.
Support Ticket Updates in Console.
New fields and changes are introduced to support ticket management in the F5 Distributed Cloud Console.
Enable Good Bot Inference For WAAP
Enabling good bot inference for WAAP through HTTP load balancers is introduced. It allows the good bots continue to the origin or use the existing mitigation defined for all automated traffic. For more information, see Bot Defense.
Improved Display of Secondary DNS Zone Content
This release adds a new "view zone file" option which shows the content of a secondary DNS zone in a much easier to read manner.
Origin Server Subset Rules
Origin server subset rules provide the ability to create match conditions on incoming source traffic to the HTTP load balancer. The match conditions include Country, ASN, Regional edge (RE), IP address, and client label selectors for subset selection of destination (origin servers). This feature can be configured in the Origins
section (advanced field) in the HTTP load balancer.
Support of A2 GPU on Site Deployed in Commodity Server HP DL360 and Dell R650.
Deployment of Site now supports A2 GPU on the certified hardware units HP DL360 and Dell R650. Site deployment on KVM also supports A2 GPU using PCI passthrough mode. Applications can use the nvidia.com/gpu
as a resource in the pod manifests.
UI Improvements for Child, Local Status, and Global Status Objects
F5 Distributed Cloud Console replaced the current JSON based view with read-only UI display view for the details of the child objects and status objects. These objects are mainly used for debugging purposes.
Terraform Version v1.3.6 for Cloud Site Deployments
Site deployment code is updated to use terraform version v1.3.6. This will help F5 Distributed Cloud Services to use the latest terraform software to provide advanced functionalities. To get the new version of terraform for already deployed Sites, users can do a software upgrade of the Site, and execute terraform apply
.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
While using L3 focussed mode feature, if L3 focussed mode is not enabled on all CE Sites in Site Mesh Group, the MTU configured on the Site-to-Site tunnel interfaces will not be consistent. Therefore, it is recommended to enable L3 focussed performance mode on all Sites participating in Site Mesh Group.
-
Remove
accept-encoding
header in compression feature is not working. A workaround is to addRemove Request Header
to removeaccept-encoding
header to be sent to upstream. -
Argo pod does not come up when Site mode is changed from L7 to L3 or vice-versa. Due to known limitation of DPDK, hugepage reservation may fail. It is recommended to reboot the node to recover from this.
-
On existing CSP deployments, custom tags for objects cannot be added/modified. In CSP deployments, increase of custom tags for objects is only supported for new deployments as tags cannot be added/modified on existing deployments.
-
TLS Coalescing will fail between load balancer with inline certificate and load balancer with explicit certificate object, when their certificate contents do not match, even if the certificate from one of these load balancers have newline at the end and other one does not have. The workaround is to edit the configuration and ensure that both certificates either end with newline or both do not have newline at the end.
-
Secure Mesh Site provisioning fails when bonded Ethernet interfaces are specified before Site is registered. The Secure Mesh Site with bond ethernet interfaces is not yet fully supported. Create the Secure Mesh Site with just the list of bond devices but not the corresponding Ethernet interface configurations. Contact the support team for assistance with provisioning the Site with the bonded interfaces.
-
Layer 3 Performance Mode in Azure is supported only for default D flavours (for example,
Standard_D5_v2
). Azure flavours have a different number of CPU NUMA nodes. The B flavours have 1 NUMA node and D flavour has 2 NUMA nodes. The current release is optimised only for D flavours to achieve the best performance. -
DNS load balancer dashboard does not show how many objects are disabled. This capacity will be added as part of the next release. The objects can still be enabled/disabled, and their status is shown in the various lists. Only the dashboard does not reflect the status.
-
NGINX Management Suite APIs
deployments
,agent-certs
, andsubscription
are not supported for usage in the F5 Distributed Cloud Services Platform. -
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
May 09, 2023
Last Updated: May 09, 2023
New Features
Disable Advertising Azure Spoke VNET Routes to Route Server
New configuration option added for Azure VNET Sites with Express Route to disable spoke VNET route advertisement to route server. By default, spoke VNET route advertisement to route server is enabled. On disabling it, routes for Azure spoke VNETs will not be advertised to Azure route server.
Addition of Suggest Values for VPC IDs in Enhanced Firewall Policy
Enhanced firewall policy object on Console is enhanced to show list of VPCs to select. This is when selecting the source or destination filter as part of the custom rule.
General Availability of Synthetic Monitoring
This release includes the general availability of Synthetic Monitoring as part of Observability. For more information, see the Synthetic Monitoring guide.
Enhanced Dashboards for Multi-Cloud Network Connect
The Multi-Cloud Network Connect service is updated with rich dashboards for network operators. Network operators can now observe and take action on their multi-cloud network with a dashboard focused each for Networking, Performance, Network Security & Site management.
MTLS Enhancements
Mutual TLS (mTLS) now supports the ability to send client certificate details to origin server in x-forwarded-client-cert (XFCC)
request header.
Support UDP 53 port in Advertise policy
Users can now load balance their DNS servers with a UDP load balancer configured on Site on port 53 with a custom VIP.
Enable Edit of HTTP Load Balancer in Standalone Bot Defense Service Card
This release enhances WAAP user experience to edit Bot Defense configuration in F5 Distributed Cloud Bot Defense service. For more information, see Bot Defense.
Rename of Delegated Access within Administration
The Delegated Access
page within the Administration
workspace is renamed to Tenant Access
. To clarify Delegated Access
configuration for Managed Tenants
, the Delegated Access
menu within the Adminstration
workspace is renamed to Tenant Access
. Functionality within the Delegated Access
workspace is unchanged.
Block SSH and DNS Ports by Default for Cloud Sites
The SSH and DNS ports will be blocked by default on cloud Sites outside network.
F5 Distributed Cloud Console DDoS & Transit Services Self-Service Prefix Advertising
In the F5 Distributed Cloud Console, a DDoS & Transit Services user now will be able to self-serve advertise and revoke their prefixes via GUI or via API. This is in addition to existing Cloud Console functionality which allows portal users to create, delete, and modify their prefixes. Advertising a prefix allows the Always Available
users to quickly and easily route their network traffic to F5 DDoS mitigation service during a volumetric DDoS attack. Users can still request the support of F5 SOC if required to advertise their prefixes on their behalf. Users can also self-service to revoke/delete their prefix advertisement once the attack has ceased.
Expand Object Tags for CSP Deployments
In CSP deployments, the number of tags that can be added to objects have been increased to 50. Out of these 50 tags, 10 tags are reserved for internal use, and up to 40 tags can be created by user. This increase of custom tags are available for new CSP deployments.
API Vulnerabilities Detection and Risk Score
API Endpoints Risk Score feature provides users with a comprehensive measure of the risk associated with their API endpoints. The risk score is calculated using a variety of techniques, such as vulnerability discovery, attack impact, business value, attack likelihood, and mitigating controls. Risk score helps users evaluate the potential impact of vulnerabilities or threats to an API endpoint and prioritize efforts to mitigate those risks. You can view the content of the risk score by security posture that appears in the endpoint details of each of the API Endpoints, with instructions and evidence for each vulnerability.
Security Incidents Reporting
Security incidents simplify the investigation of attacks by grouping thousands of events into few incidents based on context and common characteristics. The incidents for HTTP load balancer are seen in the Incidents
tab of the Security Analytics
page.
OpenAPI Validation
OpenAPI Validation is a new feature that ensures API traffic complies with the specified schema and can block non-compliant traffic. It provides flexibility in validation and control over shadow APIs, allowed IPs, and authentication schemes, improving the security and integrity of the API.
Many issues on the OWASP API Security Top 10 are caused by the lack of input validation. OpenAPI Validation is a solution to enforce protection against such attacks. The validation ensures API traffic complies with the specified schema. Non-compliant traffic can be blocked or reported, improving API security and integrity. Validation can be configured on a per-endpoint, per-group, or per-base-path basis for flexibility. The fall-through mode feature allows identifying and handling shadow APIs by either blocking, reporting, or allowing them. Enforcing authentication schemes improves API security by restricting access to authenticated users before reaching the Origin server.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
On existing CSP deployments, custom tags for objects cannot be added/modified. In CSP deployments, increase of custom tags for objects is only supported for new deployments as tags cannot be added/modified on existing deployments.
-
Secure Mesh Site provisioning fails when tagged Ethernet interfaces are specified before Site is registered. The platform manager generates logical interface for VLAN interfaces in bootstrap configuration during Site bringup when it should not, resulting in the Site provisioning failure. The workaround is to create tagged Ethernet interface after the Site is registered and the provisioning is complete.
-
Secure Mesh Site provisioning fails when bonded Ethernet interfaces are specified before Site is registered. The Secure Mesh Site with bond ethernet interfaces is not yet fully supported. Create the Secure Mesh Site with just the list of bond devices but not the corresponding Ethernet interface configurations. Contact the support team for assistance with provisioning the Site with the bonded interfaces.
-
TLS Coalescing will fail between load balancer with inline certificate and load balancer with explicit certificate object, when their certificate contents do not match, even if the certificate from one of these load balancers have newline at the end and other one does not have. The workaround is to edit the configuration and ensure that both certificates either end with newline or both do not have newline at the end.
-
Layer 3 Performance Mode in Azure is supported only for default D flavours (for example,
Standard_D5_v2
). Azure flavours have a different number of CPU NUMA nodes. The B flavours have 1 NUMA node and D flavour has 2 NUMA nodes. The current release is optimised only for D flavours to achieve the best performance. -
NGINX Management Suite APIs
deployments
,agent-certs
, andsubscription
are not supported for usage in the F5 Distributed Cloud Services Platform. -
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
-
Whenever the public IP associated with HTTP/HTTPs load balancer is re-configured to have a new public VIP, traffic to the load balancer may fail. In such scenario, user must raise a support case with F5 and F5 support team will resolve the issue.
April 11, 2023
Last Updated: April 11, 2023
New Features
Introducing Multi-Cloud Network Connect and Multi-Cloud App Connect
The Multi-Cloud Network Connect
and Multi-Cloud App Connect
are introduced as products for Multi-Cloud Networking. In this release, Cloud & Edge sites
and Load Balancers
tiles are renamed to Multi-Cloud Network Connect
and Multi-Cloud App Connect
respectively, to align with product rebranding. Also, landing pages for both these offerings are added.
Increased Granularity of Synthetic Monitoring Timeout Thresholds
HTTP(s) and DNS Monitor timeout units have changed from seconds to milliseconds, enabling finer control over alerting thresholds.
Added New Execcli for MPLS Support
This enhacement allows you to access Site CLI and execute execcli mpls
command to check MPLS label and correspond entry. Use execcli mpls --help
for more help options.
New Targets for Global Log Receiver
This release adds support for New Relic and Sumo Logic as targets for the Global Log Receiver feature. This allows customers using those vendors to send their logs more easily, rather than having to use the Generic HTTPS endpoint.
Support for Summary view of CDN Access Logs
F5 Distributed Cloud CDN now supports the ability to view a summary of CDN Access Logs.
Display Number of DNS Records in the DNS zones Listing
This release introduces possibility to display the number of DNS RR contained in each DNS zone, on the DNS zones listing page. This is done through an additional field named Number of DNS records
that can be added as part of the listing.
Discovery of Header, Payload, and Signature Information in JWT
This release introduces ability to discover and analyze headers, payloads, and signatures within JWTs. The discovery capability helps in identifying indicators of compromise, validate signature algorithm, detect user role or user ID, and identify sensitive data in JWT payloads. This can be used to guide remediation efforts to secure the insecure endpoints.
Cookie Tampering Protection
The Application Firewall Cookie Tampering protection prevents attackers from modifying the value of session cookies. This feature can be configured by navigating to HTTP Load Balancer
> Web Application Firewall
> Cookie Protection
section in the load balancers configuration.
API Authentication Types Discovery
This enhancement adds capability that detects the authentication type and its location in the API call. F5 Distributed Cloud services associates this data with the endpoint, and present the information in the endpoint details. New table columnn is created that allows you to filter and sort by authentication state or type. This helps you to quickly identify APIs that require additional security measures.
New Dashboard for Malicious Users
The new Malicious Users
page provides a global view of attackers for a specific namespace, along with the ability to obtain specific malicious user details. This page is available in Web App & API Protection
service under Overview
> Threat Insights
.
WAF Signature Staging
Attack Signatures Staging is ability to put new and updated WAF attack signatures in monitoring mode for a period of time. The feature is introduced with this release and can be configured under App Firewall
> Detection Settings
> Security Policy
section.
F5 Distributed Cloud Bot Defense Cloudflare Connector
Introduces F5 Distributed Cloud Bot Defense Cloudflare Connector. Users will be able to configure the protection through a new Cloudflare Connector type. In addition, users will be able to manage and download the configuration through the Distributed Cloud console. Once the F5 Connector module is deployed on Cloudflare, users will be able to view traffic statistics and security report in the Console dashboard. For more information, see Bot Defense.
Enable Bot Defense Mobile SDK/Base Configuration Direct Access from WAAP Portal
This release enhances WAAP customers to access F5 Distributed Cloud Bot Defense Mobile SDK/Base Configuration from WAAP Service directly.
Secure Mesh Sites
Users can now deploy Mesh sites on Edge or DC using a simplified workflow using F5 Distributed Cloud Console. Prior to this release, for deploying Mesh on Edge or DC, users were required to follow complex provisioning workflow. Using Secure Mesh Sites, the deployment of Mesh on Edge or DC is made easy with a simplified workflow in the Console.
Multiple Custom TLS Certficates/Keys per Load Balancer
HTTP and TCP load balancers now support the ability to refer to more than one custom (Bring Your Own) TLS certificate. Users can upload their TLS certificates and intermediate certificate chains to the F5 Distributed Cloud Services platform once and refer those objects from multiple load balancers. This new capability is available under Manage
> Certificate Management
section of Multi-Cloud App Connect
service.
WAAP Scheduled Reports
WAAP Scheduled reports provide the ability to schedule reports (daily, weekly or monthly) and have the WAAP summary results (of one or more namespaces) emailed to the users specified in the user groups. The feature is available under Manage
> Reports
section of Web App & API Protection
service.
Site Topology and Monitoring for Azure VNET Site
This feature now allows user to view detailed site topology for Azure VNet Site - both Standalone VNet and Hub VNet type. Users can view details about VNet, subnets, number of Mesh instances deployed, route tables etc.
Deriving Service Network for Managed K8s Endpoint Discovery Using Namespace
When doing managed K8s endpoint discovery, the virtual network for the endpoint is chosen using namespace. If the service name is used for endpoint discovery, and it has namespace embedded in it, then namespace is picked from service name. Otherwise, it is picked from endpoint object.
Note: Prior to this release, namespace was always picked from endpoint object.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
TLS Coalescing will fail between load balancer with inline certificate and load balancer with explicit certificate object, when their certificate contents do not match, even if the certificate from one of these load balancers have newline at the end and other one does not have. The workaround is to edit the configuration and ensure that both certificates either end with newline or both do not have newline at the end.
-
Layer 3 Performance Mode in Azure is supported only for default D flavours (for example,
Standard_D5_v2
). Azure flavours have a different number of CPU NUMA nodes. The B flavours have 1 NUMA node and D flavour has 2 NUMA nodes. The current release is optimised only for D flavours to achieve the best performance. -
L7 DDoS Auto-mitigation does not work when load balancer is advertised in a K8s Site. In this case, the load balancer is accessed via an external load balancer. This will result in source NAT of the traffic. Hence, instead of the true source IP, the translated IP is seen as the
src_ip
in the logs. Therefore, the created Fast ACL will end up blocking the translated IP. This inturn blocks all incoming traffic to the load balancer. -
NGINX Management Suite APIs
deployments
,agent-certs
, andsubscription
are not supported for usage in the F5 Distributed Cloud Services Platform. -
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
March 14, 2023
Last Updated: March 21, 2023
New Features
Custom Error Response Body Support for Request ID
The request_id
can be added to the custom error response body of the HTTP load balancer to make troubleshooting easier.
DDoS & Transit Services Management of IP Networks and Autonomous System Numbers (ASNs)
In the F5 Distributed Cloud Console, a DDoS & Transit Services user will be able to manage their IP networks and Autonomous System Numbers (ASNs). The first phase of this functionality offers the following capabilities in the console:
-
Define ASNs
-
Define Network Prefixes.
-
View ASN and Prefix approval status from F5 support.
Note: When a user defines their ASN or Prefix, they will then be reviewed and approved or rejected by F5 Distributed Cloud Services Support. Once approved, the ASN or Prefix is marked as approved in the Console by F5 Distributed Cloud Services Support. In the first phase, it is recommended that users append the ASN number into the alphanumeric ASN description field during ASN creation.
CDN Log Field Updates
The rsp_size
field in CDN access logs will report number of bytes sent to the client (header and response body included) instead of only response body.
Customizable TCP Load Balancer Service Policies
TCP load balancer now supports configuration to not apply any service policies or to apply a custom list of service policies. This is in addition to the default behavior of applying the namespace service policies.
Improved Configuration for Malicious User Auto-Mitigation
Auto-mitigation for malicious users can now be configured with ease, by navigating to Common Security Controls
> Malicious User Mitigation and Challenges
in the HTTP load balancer configuration.
Grouping per Availability Zone in Site Topology for AWS VPC and TGW Sites
This feature enhances the Site topology of AWS VPC and TGW Sites by grouping subnets and instances per availability zone.
Support for Directly Connecting Multiple Interfaces from Different VLANS in PK8s Kubervirt VMs and Pods
This feature supports VMs/Pods hosted on AppStack/PK8s Site to have multiple interfaces each from different VLANs/subnets directly connected to the underlay through the Site Local Outside (SLO) interface. The VMI metrics for each VLAN interface can be viewed from the F5 Distributed Cloud Console.
New Dashboard for Threat Campaigns
The new threat campaigns page provides insights into the full context of the attacker along with their origin (source IP), threat campaign attributes, and description. This page is available in WAAP service under Overview
> Threat Insights
.
Update to TLS Score Details Link in Synthetic Monitoring
The link to the detailed report for HTTPs monitors in the Synthetic Monitoring
service is moved to the footer of the Global Summary
bar.
API Groups - Club Your Multiple APIs
The API Group functionality is added to the API Management menu in the WAAP service in F5 Distributed Cloud Console. This feature allows you to group APIs together, making it easier to manage security policies across multiple APIs.
Support VM Export on PK8s
This feature supports creating a snapshot of VMs hosted on Appstack/PK8s enabling VM exports.
Introducing Enhanced Firewall Policy for cloud sites and app stack site
Enhanced firewall policy enables user to create network level policies. User can write source and destination match rules based on label selectors (selecting multiple sources or destination based on AWS VPC level tags), VPC IDs, IP prefix, and IP prefix set objects. The supported action could be to allow, deny, and insert an external service.
Note: The PAN VM Series Firewall Provider is the only external service type supported.
Enhance Client-Side Defense Reporting for F5 Distributed Cloud WAAP
WAAP users can now use the enhanced dashboard experience to view statistics and detection report for Client-Side Defense and have a navigation redirection link in between individual HTTP Load Balancer and standalone dashboard. For more information, see Client-Side Defense.
Improved WAF Exclusion rules
WAF exclusion rules now support additional match criteria such as cookie, query parameter, header, etc. These help creating granular exclusion criteria.
DDoS & Transit Services Tunnel Management
In the F5 Distributed Cloud Console, a DDoS & Transit Services user will be able to create, read, update, and delete and view status of their GRE Tunnel interfaces. This first phase of this functionality offers the following capabilities in the console:
- Create, Read, Update, and Delete GRE Tunnels
- GRE Tunnel Health Status monitoring.
Note: When a new tunnel is added via the portal, it will have an initial inactive state which will change to active once tunnel provisioning is completed.
Announcing Slow DDoS Protection
HTTP load balancer now supports configuration to protect origin servers against Slow POST and Slowloris attacks. The configuration is available at DoS Protection
> Slow DDoS Mitigation
in the HTTP load balancer configuration.
Site Topology for AWS VPC Site
This feature now allows user to view detailed Site topology for AWS VPC Site. Users can view details about VPC, subnets, number of Mesh instances deployed, route tables, etc.
Service Insertion of Next-Generation Palo Alto VM-Series Firewall with AWS Transit Gateway Site
This feature allows user to deploy Palo Alto Next-Generation Firewall (VM-Series) in AWS Transit Gateway Site (Services VPC) and create enhanced firewall policies to steer traffic to PAN FW. The traffic is steered from mesh instances to PAN instances using GENEVE
. There are added monitoring and visibility capabilities for health of service and network traffic.
Support for Internationalized Domain Names (IDN)
F5 Distributed Cloud DNS now supports IDN (Internalization Domain Names), which allows creating internet domain name containing labels displayed in non-latin script or alphabet.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
Layer 3 Performance Mode in Azure is supported only for default D flavours (for example,
Standard_D5_v2
). Azure flavours have a different number of CPU NUMA nodes. The B flavours have 1 NUMA node and D flavour has 2 NUMA nodes. The current release is optimised only for D flavours to achieve the best performance. -
L7 DDoS Auto-mitigation does not work when load balancer is advertised in a K8s Site. In this case, the load balancer is accessed via an external load balancer. This will result in source NAT of the traffic. Hence, instead of the true source IP, the translated IP is seen as the
src_ip
in the logs. Therefore, the created Fast ACL will end up blocking the translated IP. This inturn blocks all incoming traffic to the load balancer. -
NGINX Management Suite APIs
deployments
,agent-certs
, andsubscription
are not supported for usage in the F5 Distributed Cloud Services Platform. -
The
Synthetic Monitoring
service does not support aregion down threshold
greater than 5. If a monitor has more than 5 regions, entering a value greater than 5 in the region down threshold will cause alerting/global health to not work correctly for a given monitor. Synthetic Monitoring will support "region down thresholds" greater than 5 in an upcoming update. -
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
February 14, 2023
Last Updated: February 14, 2023
New Features
Clearer Indication of No Data State for New Monitors and Regions Added to Existing Monitor
When viewing monitor detail page in Synthetic Monitoring
, the Availability
chart now indicates if no health history data existed for the selected time window. This is especially useful when viewing the monitor details page immediately after creating a monitor or adding a region to an existing monitor.
Additional Validation when Using Automatic Certificates
Validation is now performed across multiple load balancers to ensure that certificates can be issued without incurring issuance errors. As an example, if domain.com exists on one load balancer and a certificate has been issued, the system will no longer accept a competing domain such as *.domain.com to exist on a different load balancer with automatic certificates.
CDN Request Logs Available for Previous 7 days in Console
For performance reasons, the CDN Request Logs functionality in the F5 Distributed Cloud Console now supports a time window selection of the previous 7 days and a 24-hour selection range. Previously supported time window selection was the past 31 days. This change brings the CDN Request Log functionality in line with the HTTP load balancer request log functionality.
Roles Added for Synthetic Monitoring Service.
Three new roles were created for the Synthetic Monitoring
service: f5xc-synthetic-monitor-admin
, f5xc-synthetic-monitor-user
, and f5xc-synthetic-monitor-monitor
.
Announcing GraphQL Inspection
F5 Distributed Cloud App Firewall now supports inspection of GraphQL requests for attacks. The settings for the feature can be configured in the Web Application Firewall (WAF)
section of the HTTP load balancer configuration.
CDI - Enable Efficient Cloning
The Containerized Data Importer (CDI) and Kubevirt is able to utilize extended features of the storage interface to create clones of volumes efficiently.
Introduction of Cookie Protection
Cookie Protection provides the ability to modify response cookies by adding SameSite
, Secure
, and Http Only
attributes.
Disable WAF per Route
Routes now support the ability to disable App Firewall (WAF) in addition to inheriting App Firewall and enabling WAF settings. This feature is available under HTTP Load Balancer > Routes > Advanced Options > Security section.
Support for HPE Alletra 6030
Added support for HPE Alletra 6030 as external storage for the Customer Edge (CE) Site. Users can select HPE as a storage device while creating an App Stack Site on the Console. It is assumed to support expandable volumes and does not require additional software development from F5 Distributed Cloud Services.
HTTP Load Balancer Support for Direct Advertisement on Internet for Cloud Site
HTTP load balancer custom advertise policy now supports a new network type Outside Network with Internet VIP
and Inside and Outside Network with Internet VIP
for AWS cloud Sites which triggers creation of AWS network load balancer of type external. Users are provided with the CNAME per AWS Site in the DNS Info
link on the HTTP load balancer configuration page. Using the CNAME, user can configure their respective DNS to attract traffic for the respective domain name.
Improved Observability for WAAP Service
Dashboards and pages in the WAAP service (expect Requests
and API Endpoints
pages) now support ability to select last 7, 14, or 30 days in the date time range picker.
Layer 3 Mode Enhanced Performance
AWS, Azure, and GCP Sites allow selecting performance enhancement mode to optimise for Layer 3 or Layer 7 networking. By default, all Sites are optimised for L7 processing. User can select site optimised for L3 traffic processing and then, majority of computing resources are allocated to L3 packet processing.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
Layer 3 Performance Mode in Azure is supported only for default D flavours (for example,
Standard_D5_v2
). Azure flavours have different number of CPU NUMA nodes. The B flavours have 1 NUMA node and D flavour has 2 NUMA nodes. The current release is optimised only for D flavours to achieve the best performance. -
AWS Site destroy fails, if Site has Internet VIP enabled and HTTP load balancer with Internet VIP network type.
-
Automatic certificates are not supported with domains and wildcard domains split across multiple load balancers. There is a currently an issue where validation does not prevent an unsupported configuration when using automatic certificates. Multiple load balancers cannot be used with automatic certificates. For example, the Console allows
*.example.com
andexample.com
on separate load balancers with automatic certificates. However, this configuration is not supported and the certificates will not be issued. Validation will be added to prevent this configuration. -
L7 DDoS Auto-mitigation does not work when load balancer is advertised in a K8s Site. In this case, the load balancer is accessed via an external load balancer. This will result in source NAT of the traffic. Hence, instead of the true source IP, the translated IP is seen as the
src_ip
in the logs. Therefore, the created Fast ACL will end up blocking the translated IP. This inturn blocks all incoming traffic to the load balancer. -
NGINX Management Suite APIs
deployments
,agent-certs
, andsubscription
are not supported for usage in the F5 Distributed Cloud Services Platform. -
The
Synthetic Monitoring
service does not support aregion down threshold
greater than 5. If a monitor has more than 5 regions, entering a value greater than 5 in the region down threshold will cause alerting/global health to not work correctly for a given monitor. Synthetic Monitoring will support "region down thresholds" greater than 5 in an upcoming update. -
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
January 17, 2023
Last Updated: February 14, 2023
New Features
Resolution of Namespace-Based Labels
Resolution of namespace label was broken because of which, ACL using these labels does not get resolved and in turn, policy does not work. This issue is resolved.
Deprecation of the Valid Response Codes Field in Synthetic Monitoring Service API
When using the Synthetic Monitoring service API, the deprecated field Valid Response Codes should no longer be used; the Response Codes field should be used instead.
TGW Site Support for VIP Port Options
TGW Site support for Outside VIP port and Inside VIP port options is introduced. This will be used to attract ingress traffic coming to the F5 Distributed Cloud Site from the cloud load balancers.
CDN – Origin Request Timeout
CDN Distribution Origin Request Timeout is now configurable with a default value of 60s (previously 180s). Customers should re-check their CDN Distributions and set correct values.
Setting of Latitudes and Longitudes at Site Creation
In case of Cloud Sites, the latitudes and longitudes are automatically set as soon as the site is created, instead of waiting till the site is deployed.
Downstream Connection Idle Timeout Configuration Support
HTTP loadbalancer can be programmed with idle timeout for downstream connections. If there are no active requests in the configured idle timeout period, the connection will be closed. This can be configured for HTTP loadbalancer of type HTTPS Proxy.
Updates to F5 Distributed Cloud Node Onboarding CLI
Updated default language and settings during the local Site onboarding shell (VPM).
HTTP Protocol Configuration with ALPN Negotiation Support in Origin Pool
The Automatic
setting when selected, allows switching of HTTP protocol. The configuration is available in Other settings
section of the Origin Pool.
New and Improved Analytics Capabilities for API Protection
The API Endpoints dashboard is enhanced with new widgets which provide a summary of top attacked APIs, sensitive data types detected, total API calls by response codes, and most active APIs by traffic observed. Ability to filter the dashboard by one or more domains is introduced. Two new columns Domains
and Sensitive Data
are added to the tabular view along with a new date time picker.
New Inside Route Table for AWS Site with Two Interfaces
For any AWS Site with two interfaces, a new inside route table will be created and all the inside subnets will be associated with it. In case of direct connect enabled site, the VGW will also propagate the route to this route table.
CDN - TLS v1.3 Support
CDN Distributions now offer TLS v1.3 support. With this addition, the F5 Distributed Cloud Services CDN offers the highest level of security and performance available.
Kubernetes 1.23
F5 Distributed Cloud Services released new Kubernetes main version 1.23.x. It automatically upgrades all Sites to this version via software upgrade.
Announcing Request Constraints
Request constraints define the validation criteria for incoming requests by enforcing size limits on HTTP request attributes. The requests that have fields larger than the specified maximums are denied. Properly configured limits mitigate buffer overflow exploits, preventing Denial of Service (DoS) attacks. Request constraints can be configured with a service policy custom rule.
New Alert for TLS Custom Certificate
A new alert for TLS Custom certificate expiration is introduced.
Update to CDN Maximum Object Size
The file size limit for the F5 Distributed Cloud Services CDN service is increased. This change will allow users to upload and serve larger digital files.
CDN – Request Logs
CDN performance dashboard now shows CDN distribution request logs.
F5 Distributed Cloud Bot Defense Mesh Connector Endpoint Tagging with Flow Labels
This release introduces F5 Distributed Cloud Bot Defense protected endpoint tagging with flow labels for WAAP through an HTTP Load Balancer. You will be able to configure flow labels as preview in this release. The endpoint and flow reporting in the security dashboard will be available in the subsequent releases. For more information, see Bot Defense.
Kubernetes Certificate Renew Process Update
Certificates are rotated only on software upgrade. Previously it was automatic, now user is introduced with the option to update it with software upgrades.
Enhanced Multiple Interface Support for PK8s Kubevirt VMs.
Pk8s Kubevirt VMs now support multiple interfaces, each from different subnets to directly connect to specific VLANs in the underlay. This feature supports AppStack/PK8s Site Kubevirt VMs to have multiple interfaces each from different subnets connected to different VLANs in the underlay and gets IP addresses allocated for those respective VLAN interfaces via external DHCP server. VMI metrics for each VLAN interface can be viewed in the Console.
Cloud Sites Upgrade Notification
After any cloud Site is upgraded, there is an updated notification which will pop up. The notification will guide user to re-apply the site after the upgrade is successful.
AWS Site Direct Connect - Support for Hosted VIFs from 4 Different Regions
AWS Site Direct Connect Configuration is enhanced to support hosted VIFs from 4 different regions. The region may be the same region as that of the Site.
Auto Mitigation for Application Layer DDoS Attacks (L7)
The Layer 7 (L7) DDoS feature based on Machine Learning (ML) now supports auto mitigation mechanism. The configuration is available under DDoS detection in DOS protection section of HTTP load balancer.
Removing Services VPN Connection from TGW (AWS resource) to AWS TGW Site
For any existing Site/new Site, F5 Distributed Cloud Services will remove the services VPN connection (Site-to-Site connection) from TGW (AWS resource) to AWS TGW Site. For existing Site, upgrade to the latest version of the Site and re-apply the AWS TGW Site.
Enable WAAP Customers to Use Standalone Bot Defense Service Card
This release enhances WAAP customers experience to use F5 Distributed Cloud Bot Defense service. You will be able to view the Bot Defense configuration for HTTP load balancers under the Application panel (Mesh Connector type for WAAP) in the Bot Defense Service Card. WAAP customers will be able to view the enhanced Bot Defense security dashboard and have a navigation redirection link in between individual HTTP Load Balancer and standalone dashboard. For more information, see Bot Defense.
Enable Private Connectivity Including Registration with Backbone for Cloud Sites utilizing CSP Private Link Solutions
This feature enables any Cloud Sites configured with a CSP provided private link solution such as AWS DirectConnect (Hosted VIF option only) and Azure ExpressRoute. The following are supported:
- Register the Cloud Site privately with the F5 Distributed Cloud Services Backbone (REs)
- Establish SSL tunnels between Customer Edge (CE) and Regional Edge (RE) over these private links and not be exposed to the internet. This enables customers to connect any cloud and on-premise locations privately & securely using F5 Distributed Cloud.
Upgrade of Envoy to 1.22.5
Envoy is upgraded to stable version 1.22.5 based on Envoy Upstream codebase, from version 1.12.7. All features will continue to work as they were.
AWS VPC Site UI Enhancement to Suggest Only AWS Credentials
In case of AWS VPC Site, the UI will suggest AWS credentials only.
Increased Limit for Configurable Cloud Tags
In case of cloud Site, configurable cloud tags limit is increased to 10.
New Attack Types and Violations in App Firewall
The following are introduced in App Firewall:
The four new violations are added:
- EVASION_IIS_UNICODE_CODEPOINTS
- EVASION_IIS_BACKSLASHES
- EVASION_PERCENT_U_DECODING
- EVASION_BARE_BYTE_DECODING
Two new attack types added:
- Remote file include
- Malicious file upload
Note: These attack types and violations are enabled by default in App Firewall policy.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
Terraform apply fails on pre-existing AWS TGW site object that is in destroyed state. Workaround is to update the site object and apply terraform.
-
AWS Site Direct Connect: Status not generated for Site with multiple hosted VIFs in region different to that of the Site.
-
NGINX Management Suite APIs
deployments
,agent-certs
, andsubscription
are not supported for usage in the F5 Distributed Cloud Services Platform. -
The
Synthetic Monitoring
service does not support aregion down threshold
greater than 5. If a monitor has more than 5 regions, entering a value greater than 5 in the region down threshold will cause alerting/global health to not work correctly for a given monitor. Synthetic Monitoring will support "region down thresholds" greater than 5 in an upcoming update. -
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
December 06, 2022
Last Updated: December 06, 2022
New Features
Site Management
External Services Configuration to Show valid list of Availability Zones
While configuring the external service nodes, the UI will only show the list of AZs where the AWS TGW Site is deployed.
Soft Restart Support to Restart All Components.
The new soft restart implementation provides ability to restart all software components instead of just limited subset of core components.
Mesh
Enhancements for Synthetic Monitoring API.
F5 Distributed Cloud Observability service has improved the design of the API for Synthetic Monitoring by replacing the nested field valid_response_codes
with the un-nested field response_codes
.
Ignore Case of Server Name Indication (SNI) of Incoming Requests during Match
If SNI in the TLS packets of incoming requests is not in the lower case, TLS handshake fails and consequently request also fails. With this enhancement, the load balancer performs case-insensitive comparison of SNI of the incoming request with the configured SNI. As a result, incoming TLS handshakes will not fail if the SNI in the incoming requests differs only in the case.
Configuration was not Updated on VER
Issue while handling mastership change resulting in configuration update being blocked is resolved.
Improved Rate limiting Rules
Rate limiting rules now support new fields such as IP address, Country, and ASN for defining the request match criteria.
Log Collection in Offline Survivability
Log Collection in offline survivability stores 5 minutes or 8 MB of logs, after that time, some logs are going to be dropped. When logs get filled there can be occasionally crashes of fluentbit component.
New User Identifiers
This feature adds support for Client Country, Client Region, and Client City as user identifiers.
Delegated Access
This release allows one organization to “Delegate Access” to their tenant to users from another tenant and easily allow users from the second tenant to manage configuration in side the first organization's tenant.
OpenAPI Specification Enhancement
This release enhances Open API specification to include more learnt information for discovered Application API. The API Discovery feature allows downloading of generated OpenAPI Specification based on analyzed traffic per application. The OpenAPI Specification is extended to include application domains; request content-types and headers; detected response codes. The specification can be downloaded from Application Security Dashboard > API Endpoints tab.
Enhanced PII Detection in API Discovery
The Personal Identifiable Information (PII) detection capability is enhanced to detect Credit Card Number, US Social Security Number, Email, and Password for API requests and responses. The PII is detected for JSON and x-www-form-urlencoded formats, and can be monitored by navigating to API Endpoints > API Endpoint Details,
Offline Survivability Support for DC Cluster Group
With Site Offline Survivability enabled, a site continues to function with existing configuration even when it has lost its connectivity to Regional Edge (RE) Site. A Local Control Plane is implemented in a site when this is enabled, so local traffic load balancing for this site continues to work. Also, if two or more sites (having this feature enabled) are part of DC Cluster group, load balancing across local and remote endpoints in CE Sites continues to work as well, even when connectivity with the RE is lost.
With offline survivability, a site can continue to function as is with existing configuration for upto 7 days, even when the site is offline. The certificates needed to keep the services running on this site are signed using a local CA. Secrets would also be cached locally to handle the connectivity loss. When this feature is enabled/disabled on an existing site, the pods/services on this site will be restarted. If a site is running in offline state, it would not be able to communicate with REs (even if there is connectivity). Site would resume communication with REs/GC as soon as connectivity to GC is restored. A Local Control Plane is implemented in a CE site with this feature. So when a site loses its connectivity to RE, the local site load-balancing continues to work. If two or more such sites are in a DC Cluster Group (SLO/SLI), when a site loses its connectivity to RE, the load-balancing across local and remote endpoints continue to work.
New Alert for TLS Automatic Certificate
A new alert for TLS Automatic certificate renewal failure is introduced. The alert can be configured in alert policy rules.
Advertise Virtual IP or Endpoint Over Private Virtual Network
This feature allows advertising application virtual IP (VIP) or application endpoints over a private virtual network (known as private ADN network). When a customer on-premises location or Data Center is connected to F5 Distributed Cloud Global Backbone using Regional Edge (RE) site, the customer's on-premises network is mapped as a private virtual network on the RE site. The users or applications on this on-premises network needs access to applications residing on other cloud or edge sites. With this feature, the application's VIP or endpoint can be advertised into the private virtual network, thereby providing access across clouds.
Cross Site Request Forgery(CSRF) Protection
This release introduces the CSRF protection, enabling protection for your applications against CSRF attacks. This feature can be configured within the Web Application Firewall section in the HTTP Load balancer.
Ability to Exclude WAF Processing on One or More Paths
WAF exclusion rules now provide the ability to skip WAF processing on a path. This provides the flexibility for users to skip (disable) WAF on one or more paths.
DNS Load Balancer
Customers can now use F5 DNS Load Balancer (known as GSLB) as part of the F5 Distributed Cloud Services, allowing to improve the performance and availability of global applications by sending users to the closest or fastest endpoint. The DNSLB leverages F5 Distributed Cloud Anti-DDoS and Anycast architecture to distribute DNS delivery globally.
Stack
Ingress Support for Managed K8s
All Kubernetes distributions use an API object called Ingress
to manage external access to the services in a cluster. This object describes the way to terminate the Layer 7 request from the client (HTTP/HTTPs), and apply host and path-based routing policies. An ingress controller is responsible for fulfilling and enforcing what the ingress describes, usually with a load balancer.
Console
New Namespace-level Security and Performance Dashboards
Security and Performance Dashboards are introduced with a new set of features dedicated to the following:
- Identifying, analyzing, and blocking attack activity against web applications and APIs
- Identifying and analyzing the performance and health of applications
UX Enhancements when Blindfolding a Secret
The F5 Distributed Cloud Console has improved presentation when blindfolding secrets.
UX Enhancements to Configuration Forms
The F5 Distributed Cloud Console has introduced improved presentation across various forms. The "View Configuration" and "Clone" operations are now available for shared resources in application namespaces.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
In the
Synthetic Monitoring
service, updates to a monitor via API require thevalid_response_codes
andresponse_codes
sections to be in sync. -
NGINX Management Suite APIs
deployments
,agent-certs
, andsubscription
are not supported for usage in the F5 Distributed Cloud Services Platform. -
In the
Synthetic Monitoring
service, newly added regions may not display in theAvailability chart
temporarily. After adding another region(s) to an existing Synthetic Monitor, that region(s) may not be displayed on the Availability chart, depending on the selected time window (e.g. Last 1 hour, Last 24 hours, Last 7 days). Once the monitor has run for a period of time long enough for the selected time window step size, the region(s) will show. -
In the
Synthetic Monitoring
service,Global
andRegional
health event timestamps may differ slightly. As the Global monitor derives health from its Regional monitors, there may be a slight mismatch in health event timestamps. -
The
Synthetic Monitoring
service does not support aregion down threshold
greater than 5. If a monitor has more than 5 regions, entering a value greater than 5 in the region down threshold will cause alerting/global health to not work correctly for a given monitor. Synthetic Monitoring will support "region down thresholds" greater than 5 in an upcoming update. -
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
November 01, 2022
Last Updated: November 08, 2022
New Features
Site Management
Kubernetes 1.22
This release introduces support for Kubernetes main version 1.22.x. The F5 Distributed Cloud Services automatically upgrade all sites to this version via software upgrade.
Ability to Purge Orphaned Containers through CLI
This release adds the ability to purge orphaned containers through the Site Command Line Interface (CLI).
Mesh
Disallow of Duplicate CDN Origins in Different Tenants
Duplicate origins were allowed in CDN Distributions across different tenants. This would lead to the CDN Distribution remaining in Pending
status and not transitioning to an active CDN Distribution. Validation is added to prevent duplicated origins.
F5 Distributed Cloud Bot Defense AWS CloudFront Connector
Introduces F5 Distributed Cloud Bot Defense AWS CloudFront Connector. Users will be able to configure the protection through a new AWS CloudFront Connector type. In addition users will be able to manage and download the configuration through the Distributed Cloud console. Users will access AWS Console to deploy the F5 Lambda@Edge Connector module for CloudFront via the Serverless Application Repository(SAR). Once it is completed, users will be able to view traffic statistics and security report in the Console dashboard. For more information, see Bot Defense.
CDN Cache Options Enhancement
Cache TTL has changed to "Cache Options". A new Cache Setting to "Disable Cache" has been added. This option will disable caching by the CDN service. When caching is disabled, the x-cache-status header will show BYPASS.
HTTP Expected Status Codes Support on Health Check Object
This feature supports the user in specifying a list of HTTP response status codes to be considered healthy. To use this feature, user can configure the list of expected status codes in HTTP Health Check Parameters. If this field is configured, the configured status codes are used to determine health status rather than the default status code.
Logs Streaming Enhancements
This release adds the ability to send logs to Kafka receivers and AWS Cloud Watch receivers, allowing to cover a broader set of targets. It is now also supported to send Audit Logs. Users can select between Request Logs, Audit Logs, and Security Events.
Rule Exclusion for API Security Events Triggered by WAF Signatures
API security events can be triggered by detecting a WAF signature of type "information leakage" in the transaction's response. Detecting this kind of signature will trigger the API Security event with "sec_event_name":"App Security Misconfiguration". All detected signatures will be visible in the event's details in the event page. If users decide this is not an API security event (false positive), or for any other reason, users can set an exclusion rule for this detected signature and avoid future events on this signature.
Note: This exclusion mechanism is already supported for WAF events and now the support is extended to API security events.
Customer Tenant Access for Support Team
The support team by default has read and write access to all namespaces on all customer tenants. Customer can now change the level of access of the support team to read-only access to all namespaces or read-and-write access to all namespaces or read-and-write access to selected namespaces.
App Infrastructure Protection Service
This release introduces the App Infrastructure Service for cloud workload protection, which delivers deep telemetry and high-efficacy intrusion detection for cloud-native workloads.
Analyze and Generate Response Payload Schema in API Discovery
The API Discovery mechanism is extended to learn responses payload schema. The discovered schema and response examples are presented per path and method in API Endpoints dashboard and in downloaded swagger. Currently, 2XX/3XX
JSON and form-urlencoded
responses are analyzed.
External Service Object Deletion after Cloud Resources Deletion
When user initiates delete of external service object, the actual object deletion takes few minutes as it waits for cloud resources to be deleted.
F5 Distributed Cloud Bot Defense Scheduled Threat Briefing Report
F5 Distributed Cloud Bot Defense users now will receive monthly scheduled threat briefing report through email. Users can enable and disable this email report through Manage Application page. By default, this feature is enabled for all subscribed bot defense customers.
Site Offline Survivability
With Customer Edge (CE) Site Offline Survivability enabled, a site continues to function with existing configuration even when it has lost its connectivity to Regional Edge (RE) Site. A Local Control Plane is implemented in a site when this is enabled, so local traffic load balancing for this site continues to work. Also, if two or more sites (having this feature enabled) are part of Site Mesh Group (Full Mesh), load balancing across local and remote endpoints in CE Sites continues to work as well, even when connectivity with the RE is lost.
With offline survivability, a site can continue to function as is with existing configuration for upto 7 days, even when the site is offline. The certificates needed to keep the services running on this site are signed using a local Certificate Authority (CA). Secrets would also be cached locally to handle the connectivity loss. When this feature is enabled/disabled on an existing site, the pods/services on this site will be restarted. If a site is running in offline state, it would not be able to communicate with REs (even if there is connectivity). The Site resumes communication with REs and Global Controller (GC) as soon as connectivity to GC is restored.
Azure Vnet Site Support Connecting to Azure Express Route Circuit
Azure Vnet Site in a Hub VNET mode now supports multiple connection to express route circuit.
Console
Enhancements to CDN Dashboard Page
CDN Dashboard shows a country map widget that displays the country that visitors are coming from. CDN Dashboard shows the Top 5 countries that are visitors are coming from.
Enhancements to Date & Time Selection for HTTP Load Balancer Security & Performance Monitoring
The custom
selection in date and time picker for following HTTP Load Balancer Monitoring pages provides the ability to select maximum 24 hrs in the last 30 days.
- Overview dashboard
- Security dashboard
- Performance dashboard
- Malicious Users
- DDOS
- Security events
- Alerts
- Bot Defense
- Metrics
- Errors
- Origin Servers
- Traffic
User Experience(UX) Enhancements for Monitoring Pages
The brush component is replaced with Pan and Zoom component in the charts, for all monitoring pages, to simplify ease of use while viewing and investigating of logs.
Enhancement to Audit Log APIs to Extend Maximum Query Duration
Maximum query duration for audit log APIs is extended from 24 hours to 30 days. In the F5 Distributed Cloud Console, the custom
selection in date and time picker for Audit Logs
page provides the ability to select any duration in the last 30 days.
F5 Distributed Cloud Bot Defense 30 days Time Range Selection Support
F5 Distributed Cloud Bot Defense users now can select up to 30 days data range on Dashboard
page and Traffic Overview
page.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
In the
Synthetic Monitoring
service, newly added regions may not display in theAvailability chart
temporarily. After adding another region(s) to an existing Synthetic Monitor, that region(s) may not be displayed on the Availability chart, depending on the selected time window (e.g. Last 1 hour, Last 24 hours, Last 7 days). Once the monitor has run for a period of time long enough for the selected time window step size, the region(s) will show. -
In the
Synthetic Monitoring
service,Global
andRegional
health event timestamps may differ slightly. As the Global monitor derives health from its Regional monitors, there may be a slight mismatch in health event timestamps. -
The
Synthetic Monitoring
service does not support aregion down threshold
greater than 5. If a monitor has more than 5 regions, entering a value greater than 5 in the region down threshold will cause alerting/global health to not work correctly for a given monitor. Synthetic Monitoring will support "region down thresholds" greater than 5 in an upcoming update. -
Azure Vnet Peering with auto routing should only be enabled for spoke vnets without custom route table attached to subnets. User can enable custom routing option for spoke vnets with existing route table attached to subnet on the spoke net.
-
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
September 27, 2022
Last Updated: September 27, 2022
New Features
Site Management
Extend Site Local CLI with Debugging Options
This release adds execcli
subcommand, which provides options to provide advanced debug operations like dropstats, nh, systemctl-restart-kubelet, systemctl-restart-docker, systemctl-restart-vpm, vif, rt, flow, journalctl, curl-host, edit-etc-hosts, etc.
Mesh
Enhancements when Using CDN and WAF in Series
When using WAF as a CDN Distribution's origin, the WAF protected origin would occasionally respond with a set-cookie
header and cause a cache miss on corresponding requests.
Logs Streaming enhancements
This release adds the ability to send Security Events and option to select between Request Logs and Security Events for streaming. This allows customers to send Security Events to the monitoring systems (such as SIEM systems) so that those events can be processed by those systems.
Note: The supported targets are the same as for Request Logs (AWS S3, Datadog, Splunk, Azure Blob Storage, Azure Event Hubs, and Generic HTTPS Endpoint). Security Events are sent in the same format (JSON) as Request Logs.
Removal of Skip IP Reputation Option from Service Policy Rule
The Skip IP Reputation
functionality is removed from Service policy custom rule in Action
section. To bypass IP Reputation feature for one or more IP addresses, the Trusted Client Rules feature should be used. The Trusted Client functionality is available in the Common Security Controls
section of HTTP Load Balancer.
Replace Query Parameters for Redirect Requests
The redirect route configuration is enhanced to replace query parameters of incoming requests with user specified values.
Note: Before this release, the redirect route configuration only supported removing or retaining the query parameters of incoming request.
Delay Deletion of Endpoints Discovered via vk8s Service Discovery.
When endpoints discovered via vk8s service discovery are getting deleted, they are marked with lower priority in data-path for short duration. Newly discovered endpoints are programmed with higher priority. The old endpoints programmed with lower priority will be removed after short duration. This will ensure that traffic will always go to new discovered endpoints when they are available and minimize traffic drops.
Improved Navigation for WAAP Features in HTTP Load Balancer
The security configuration section has been enhanced to highlight the different features of WAAP to simplify onboarding and enablement of security features for users. The WAF, Bot Protection, API Protection, Dos Protection, Client Side Defense, and Common Security Controls sections now simplify and highlight the various WAAP features.
Note: The enhancement is supported with API backward compatibility.
Enabling support for Containerised Data Importer for Kubevirt
DataVolumes are a way to automate importing virtual machine disks onto PVCs during the virtual machine's launch flow. Without using a DataVolume, users have to prepare a PVC with a disk image before assigning it to a VM or VMI manifest. With a DataVolume, both the PVC creation and import is automated on behalf of the user. A DataVolume is a custom resource provided by the Containerized Data Importer (CDI) project. KubeVirt integrates with CDI in order to provide users a workflow for dynamically creating PVCs and importing data into those PVCs.
Note: In order to take advantage of the DataVolume volume source on a VM or VMI, CDI must be installed.
Enhancements to Service Policy Blocking Page
Service policy blocking page shows the request ID. This provides the ability to search for the blocked request in the security events page and view the details and reason for blocking.
Note: The
Support ID
value displayed in the blocking page is the request ID.
New Capability to Enable App Firewall Policy per Route
A simple route matches on a path and/or HTTP method and forwards the matching requests to one or more origin pools. Users now have the ability to configure an App Firewall policy per simple route.
Ability to Identify the Real Client IP Address with Trusted Client IP Headers
Trusted Client IP Headers feature provides the ability to identify the real client IP address, that initiated the connection to the platform, as the source IP, from the configured http headers, when there are one or more proxies between the real client and the distributed cloud platform.
Security events and request logs will show this extracted IP address from the HTTP headers as the source IP, when this feature is enabled.
Support for Discovery of Services without Specifying Port
During service discovery, port of the service was required to be specified. This release introduces automatic port option where specifying port is not mandatory. When this automatic port option is specified for K8S service discovery, the service will be discovered in port 80 or 443 depending on whether TLS is configured in Origin Pool. For Consul discovery, all the services matching the name will be discovered. When there are multiple ports for Consul service, all of them will be discovered with one endpoint for each unique port.
Application Misconfiguration Detection (OWASP API Security 7 - Security Misconfiguration)
Users may misconfigure their application, which may result in application information leakage to an attacker. A new detection capability for information leakage has been added to identity this scenario. In case of information leakage detection, an API security event is triggered in the Security events page with Event Name
as App Security Misconfiguration
. This security event will be triggered when WAF is enabled on the HTTP Load Balancer. See App Firewall for more information on App Firewall.
Enable Client-Side Defense for F5 Distributed Cloud WAAP
F5 Distributed Cloud Mesh Connector now supports automatic JavaScript injection for Client-Side Defense to enable new add-on protection for WAAP users. Client-Side Defense users will have an enhanced self-serving onboarding alternative to leverage native JavaScript injection. Users can configure Client-Side Defense JavaScript injection through an HTTP Load Balancer and define the protected domains still through the Client-Side Defense Service card. Users will be able to view statistics and detection report in the Client-Side Defense Service dashboard. For more information, see Client-Side Defense.
TLS Support on TCP Load Balancer
This release introduces TLS support for TCP Load Balancer. To use this feature, user can pick TLS type or TLS with autocert option in newly added loadbalancer_type
property.
Detect Users Sending High Number of Requests to Non-Existing URLs (OWASP API Security 1- Broken Object Level Authorization)
Attackers can exploit API endpoints that are vulnerable to Broken Object Level Authorization (BOLA) by manipulating the ID of an object that is sent within the request. This may lead to unauthorized access to sensitive data. In many cases attacker sends requests to multiple non-existent URLs trying to discover unprotected resources. Malicious Users feature detection mechanism is enhanced to identify and flag such scenarios.
System for Cross-Domain Identity Management (SCIM)
SCIM is a standard protocol for automating the exchange of user identity information between identity domains and IT systems. This feature simplifies organisations in managing their employees On-boarding / Off-boarding from F5 Distributed Cloud Console and also managing the roles.
The SCIM feature simplifies user acccess management for organisations. This release adds the SCIM support for Azure AD.
Header Transformation Support on Origin Pool
A new feature that provides the ability to normalize the headers of the upstream requests, has been added. To use this feature, the user can enable header transformation options present under advanced options of Origin Pool configuration.
Console
Synthetic Monitoring Feature as Part of the New Observability Service
This release introduces the Synthetic Monitoring feature for Observability. For more information, see Synthetic Monitoring guide.
Enhancements for API Endpoints page.
The enhancements include refresh and auto-refresh capabilities, ability to filter items in the Table
tab, and a new tooltip for Security Events
column.
Note: The new tooltip in the security events column provides details for the values in that column.
Attack Activity Insights for HTTP Load Balancer Security Dashboard
The following capabilities are added to Security Dashboard:
- Ability to filter dashboard metrics per domain
- Global filter for the dashboard widgets
- Security Events by Type widget
- Top Attack Sources widget
- Top Attacked Domain/Path widget
- Top Attacked API Endpoints widget
- Top Attacks by Violations widget
- Top Attacks by Threat Campaigns widget
- Domains filter to view stats per domain
- Global filter for metrics in the dashboard
The Recent Security Events
widget is removed.
Improved Attacks Investigation with Forensics
Forensics provides an intuitive approach to investigate attacks and take relevant actions. More than 25 metrics are supported that provide the ability to slice and dice the security events, along with the ability to view the top values for each metrics along with the percentages.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
In the
Synthetic Monitoring
service, a 500 error is returned when a synthetic configuration has aNumber of Failed Locations
value greater than number of configuredExternal Sources
. The workaround is when creating/editing a synthetic monitor, ensure theNumber of Failed Locations
value is not greater than the number of provider regions under 'External Sources'. -
In the
Synthetic Monitoring
service, newly added regions may not display in theAvailability chart
temporarily. After adding another region(s) to an existing Synthetic Monitor, that region(s) may not be displayed on the Availability chart, depending on the selected time window (e.g. Last 1 hour, Last 24 hours, Last 7 days). Once the monitor has run for a period of time long enough for the selected time window step size, the region(s) will show. -
In the
Synthetic Monitoring
service,Global
andRegional
health event timestamps may differ slightly. As the Global monitor derives health from its Regional monitors, there may be a slight mismatch in health event timestamps. -
The
Synthetic Monitoring
service does not support aregion down threshold
greater than 5. If a monitor has more than 5 regions, entering a value greater than 5 in the region down threshold will cause alerting/global health to not work correctly for a given monitor. Synthetic Monitoring will support "region down thresholds" greater than 5 in an upcoming update. -
Azure Vnet Peering with auto routing should only be enabled for spoke vnets without custom route table attached to subnets. User can enable custom routing option for spoke vnets with existing route table attached to subnet on the spoke net.
-
CDN distribution domains are limited to 52 characters. Using a CDN distribution domains longer than 52 characters will result in a non-compliant domain name for the service that is not resolvable.
-
The Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove old resource, and it requires deleting resource using the
k delete apiservices v1alpha3.subresources.kubevirt.io
command. -
When enabling the AWS Direct Connect on existing AWS VPC Site and AWS TGW Site from the Console, only hosted VIF mode for VIF configuration is allowed in this release. Setting standard VIF mode for VIF configuration is not allowed. However, it can be configured through API or vesctl.
-
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
August 30, 2022
Last Updated: August 30, 2022
New Features
Site Management
Updated Workflow for Cloud Sites
Cloud Site terraform parameters should be applied after the cloud site software upgrade. A message box will pop up after the Cloud Site software upgrade is started, which will ask users to run apply after the site is updated successfully.
AWS Site - Show Imported Routes from VGW
In case of AWS Sites with direct connect enabled, if user selects show direct connect status
option, a new field is displayed showing routes imported by the Distributed Cloud dataplane from VGW.
Remove Option to Block Web UI port for Cloud Sites
While configuring "Blocked Services" in cloud sites, the option to chose "Web UI" was present. This option is incorrect as the cloud sites do not have "Web UI". This option is now removed. You can now choose only SSH or DNS.
Mesh
Bot Defense - Known Good Bot
F5 Distributed Cloud Bot Defense new dashboard is enhanced with known good bot classification.
Assign Custom/Private ASN for AWS Direct Connect Gateway
Custom ASN option enabled while configuring direct connect on AWS Sites.
BGP Passive Mdde Support
BGP can be configured to run in passive mode where it will never initiate a session to the peer. Instead, it will wait for connection to be initiated by peer. By default, BGP is configured in active mode until passive mode is set.
Increased Number of Static Routes per Virtual Network
Number of static routes that can be attached per virtual network is increased to 100.
Logs Streaming Enhancements
This release adds the ability to send logs to Splunk and Datadog receivers, as well as a generic HTTP(s) endpoint, allowing to cover a broader set of targets. It is now also possible to send logs to Azure Blob Storage and Azure Event Hubs. Customers can now select for which application namespaces they want the logs to be sent, and the Shared namespace is also supported.
Site Labels Usage as Labels in Network Policy
Site labels can be added by user in the Site which in turn can be used to define destination and source in network policy. The keys of these labels should be the known_label
key.
API Discovery - Generate OpenAPI 3
OpenAPI 3 is introduced to define discovered API endpoints instead of OpenAPI 2 (Swagger). The OpenAPI spec can be retrieved per discovered API endpoint or downloaded for all discovered API endpoints of a specific HTTP Load Balancer.
F5 Distributed Cloud Bot Defense Mesh Connector Web-scraping Protection for WAAP
Introduces F5 Distributed Cloud Bot Defense Mesh Connector web-scraping protection for WAAP. Users will be able to configure the protection natively on an HTTP Load Balancer through Bot Defense protected endpoint with either GET(XHR) or GET(Document) HTTP methods(not both). Users will be able to view traffic statistics and security report in the dashboard. For more information, see Bot Defense.
Support Non-SNI Clients when more than one HTTP or TCP LB is Advertised on a Single VIP Address.
When multiple HTTP and/or TCP Load balancers are adverstised on a single VIP address, the load balancer allowed requests only when their Server Name Indication(SNI) matches. This feature introduces concept of default load balancer per Advertise policy. If SNI is missing in the TLS client Hello, default HTTPS LB is selected and certificate of this HTTPS LB is used for TLS session.
Multiple Interfaces Network Support
Appstack/Pk8s site Kubevirt VMs now support multiple interfaces each from different subnets.
Azure VNET to VNET Peering (Hub & Spoke Model)
Azure Vnet Site can now support VNET to VNET peering in a hub and spoke VNET model. Any Azure VNET Site can now be made as Hub VNET and users can give list of azure spoke VNETs which need to be peered. These spoke VNET CIDR routes will be discovered by the Site, and it will route from hub VNET. All spoke VNET to VNET communication will happen via Azure Vnet Site.
Upload File Directly from API Definition
OpenAPI Specification files can be uploaded during HTTP Load Balancer Configuration flow. During HTTP Load Balancer configuration a user can define API Definition of the application by uploading OpenAPI Specification files. This API Definition sets the Inventory of paths and methods for the given HTTP Load Balancer. The Inventory and Discovered API can be monitored in API Endpoints Security dashboard. In addition, the Inventory can be used in suggestions to build API Protection and Rate Limit rules. The flow is improved by allowing to upload OpenAPI files directly during HTTP Load Balancer or API Definition configuration.
F5 Distributed Cloud Content Delivery Network (CDN)
This release introduces F5 Distributed Cloud CDN. A CDN is a geographically distributed set of highly efficient servers that cache and deliver static and dynamic content. Users can take advantage of Distributed Cloud CDN to improve application performance and end user experience when delivering apps on Distributed Cloud. Distributed Cloud CDN is available to all users. For more information, see Configure CDN.
Console
Bot Defense: Support Up to 30 days Time Range for Dashboard
Support of time range selection up to 30 days for F5 Distributed Cloud Bot Defense is enabled.
Connectivity Graph API/UI Support for 30 days of Observability
Following connectivity graph API's now support 30 days of observability data:
-
api/data/namespaces/<namespace name>/graph/connectivity/edge
-
api/data/namespaces/<namespace name>/graph/connectivity/node
-
api/data/namespaces/<namespace name>/graph/connectivity.
Following connectivity graph UI pages now support 30 days of observability data:
cloud-and-edge-sites/sites/site_connectivity/connectivity/system_metrics.
Note: It will take 30 days after the upgrade to populate the backend storage with the historical data required to support this functionality. If you make a query for a time range greater than 24 hours before 30 days data is fully populated, you will see that the data is unavailable for the time range before the upgrade. If you narrow down the range using brush over the time range where data is unavailable, you may suddenly see data. This inconsistency will be resolved once backend storage has complete data for 30 days.
Modification of Refresh Button with the Timestamp Value
This release will show Timestamp along with Refresh for Dashboard and other Data Visualisation page.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
Kubevirt VM upgrade requires to execute removal of old API resource. When new version of kubevirt v0.54 is deployed, it does not automatically remove the old API resource and requires deleting using the
kubectl delete apiservices v1alpha3.subresources.kubevirt.io
command. -
In case the Azure VNET Site deployment is failed with the
You have not accepted the legal terms on this subscription...
error message, perform the following even if the VM marketplace agreement terms were already accepted:- Select correct subscription; Enter the
az account set -s <subscription-id>
command. - In case of Ingress Gateway Site type, enter the
az vm image terms accept --publisher volterraedgeservices --offer volterra-node --plan volterra-node
command. - In case of Ingress/Egress Gateway site type, enter the
az vm image terms accept --publisher volterraedgeservices --offer entcloud_voltmesh_voltstack_node --plan freeplan_entcloud_voltmesh_voltstack_node_multinic
command. - In case of App Stack site type, enter the
az vm image terms accept --publisher volterraedgeservices --offer entcloud_voltmesh_voltstack_node --plan freeplan_entcloud_voltmesh_voltstack_node
command.
- Select correct subscription; Enter the
-
When enabling the AWS Direct Connect on existing AWS VPC Site and AWS TGW Site from the Console, only hosted VIF mode for VIF configuration is allowed in this release. Setting standard VIF mode for VIF configuration is not allowed. However, it can be configured through API or vesctl.
-
The global resource hosted VIF acts as a region specifically from AWS terraform provider point of view. Therefore, only attaching VIF from single region is supported for this release.
-
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
August 02, 2022
Last Updated: August 02, 2022
New Features
Site Management
Kubevirt Upgrade v0.54.0
The Kubevirt version v0.54.0 is released. The F5 Distributed Cloud Services automatically upgrades all sites to this version via the software upgrade.
Mesh
F5 Distributed Cloud Bot Defense - Mobile SDK and Base Configuration Management
This release introduces F5 Distributed Cloud Bot Defense self-serving mobile SDK and base configuration download capability for all Connectors as applicable (Mesh, iApp, and Custom Connectors). Users can download the stable and Long Term Support(LTS) versions of compatible Mobile SDK from the Standalone Bot Defense Service. In addition, users can manage base configuration and access mobile integration guide from the F5 Distributed Cloud Console. For more information, see Bot Defense.
DNSSEC for Primary DNS
This release adds support for DNSSEC in Primary DNS Zone.
Trusted Client Enhancement to Skip IP Reputation Detection for Specific IP
IP reputation database is frequently updated and reputation of any specific IP may change between updates. With this enhancement, you can set trusted client to skip IP reputation check if this IP is misclassified and blocked by IP reputation policy.
Trusted Client Enhancement to Skip Malicious User Detection for Specific User IDs
The enhancement to skip Malicious User Detection for Specific User IDs is introduced. With this, you can create Trusted Client to skip Malicious User detection check and mitigation if this user ID is mistakenly detected and blocked as Malicious User.
Trusted Client Enhancement to Skip API Protection Rules for Specific IP
Trusted Clients can be defined to skip different protection layers, including API Protection. For example, you can define API Protection allowing access to only specific APIs. With this enhancement, you can configure to bypass these rules for specific trusted clients.
App Stack
Support POD Priority Class for Managed K8s
Managed K8s configured with Allow K8s API Access to ClusterRoles
, ClusterRoleBindings
, MutatingWebhookConfiguration
, and ValidatingWebhookConfiguration
is enhanced to allow you to perform Create, Replace, Update, and Delete (CRUD) operations on POD Priority Classes denoted with the priorityclasses
construct. Users can manage their own classes. However, users cannot delete or modify existing list of priority classes deployed by the F5 Distributed Cloud Services.
The following is the existing list of priority classes:
Existing Priority Classes
NAME VALUE GLOBAL-DEFAULT AGE
ares-priority 900000 false 468d
ares1-priority 900000 false 495d
argo-priority 1000000 false 251d
bdbewaf-priority 1000000 false 83d
envoy-priority 1000000 false 251d
etcd-priority 1100000 false 495d
fluentbit-priority 600000 false 495d
frr-priority 1000000 false 251d
ganges-priority 900000 false 13d
gubernator-priority 900000 false 495d
ike-priority 1000000 false 251d
keepalived-priority 1000000 false 251d
obelix-priority 1000000 false 495d
openvpn-priority 1000000 false 251d
opera-priority 1000000 false 495d
phobos1-priority 900000 false 456d
piku-priority 900000 false 495d
pmtud-priority 1000000 false 251d
prometheus-priority 700000 false 495d
sredns-priority 1000000 false 468d
system-cluster-critical 2000000000 false 495d
system-node-critical 2000001000 false 495d
test-priority 1100000 false 16s
ver-priority 1000000 false 495d
voucher-priority 1100000 false 495d
webroot-priority 1000000 false 210d
Support User Application to Access Kube State Metric
This release introduced support for user's application to access kube-state-metric which allows user to scrape the metrics via Kubernetes service kube-state-metrics.kube-system.svc:65031
. It is accessible from any namespace/pod in the Site. This feature is enabled by adding ClusterWideApplication
called Prometheus
.
Note: It also opens the ports to be accessible from the outside of the Site. Therefore, block the ports if your Site is exposed to the Internet.
Console
UX Enhancements - New Configuration Forms
The F5 Distributed Cloud Console User Interface is enhanced with configuration forms with improved presentation and functionalities. This contains layout resizing with change in screen resolution (anything above 1280 px resolution is supported). It contains visual changes for grouping of form components, reference viewing, editing, and mixing of modes within same form. The enhancement also updated left side navigation to provide more information about all steps in reaching a nested level.
Not all objects are enhanced with the improved forms. Only the following objects are enhanced in this release:
Objects with Enhanced UI
DNS Zones
DNS Delegated Domain
Fast ACLs
Workload flavor
App Setting
App Type
K8s Clusters
USB Policy
API Definition
Certificate Revocation List
Rate Limiter
Log Receivers
Global Log Receiver
Alert Receivers
User Identifications
K8s Cluster Role Bindings
Applications
Malicious User Mitigation
Bot Defence - Applications
Mobile Base Configs
K8s Cluster Roles
K8s Pod Security Policies
Note: Configuration for enhanced objects does not load the improved form if it is created or updated from a parent object that is not enhanced with the new UI.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
When enabling the AWS Direct Connect on existing AWS VPC Site and AWS TGW Site from the Console, only hosted VIF mode for VIF configuration is allowed in this release. Setting standard VIF mode for VIF configuration is not allowed. However, it can be configured through API or vesctl.
-
The global resource hosted VIF acts as a region specifically from AWS terraform provider point of view. Therefore, only attaching VIF from single region is supported for this release.
-
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
July 05, 2022
Last Updated: July 05, 2022
New Features
Site Management
Support for Multi-Node Site Deployment in a Single AZ
Support for multi-node Site deployment in a single Availability Zone (AZ) is introduced.
Mesh
Direct Connect Support for AWS VPC Site and AWS TGW Sites
Using Direct Connect Support, users can connect the on-premises data centers to VPC in which F5 Distributed Cloud Sites are hosted. The platform automatically discovers the on-premises datacenter routes advertised by the on-premises router connected to AWS router via direct connect. These routes will be learnt on the inside network of the F5 Distributed Cloud Site. Two modes of direct connect private vif interface are supported.
Note: User must manage the direct connect connection.
F5 Distributed Cloud Bot Defense Mobile Protection
This release introduces F5 Distributed Cloud Bot Defense mobile application protection for Mesh Connector through an HTTP Load Balancer, iApp Connector(v3.0.3), and Native Connector for BIG-IP. Users can configure mobile protection natively on an HTTP Load Balancer though Bot Defense configuration and view the mobile traffic in the security dashboard. For BIG-IP users, mobile protection is configured through the iApp or Native Connectors and users can view the mobile traffic in the standalone Bot Defense service dashboard. For more information, see Bot Defense.
API Discovery Enhancement
HTTP Load Balancer API Endpoints dashboard is enhanced to present Inventory, Discovered, and Shadow API for better tracking of approved versus detected API. The Inventory API consists of operations defined in OpenAPI files, imported by a user for a given HTTP Load Balancer. Discovered set includes API with traffic detected during last few days. The Shadow API presents discovered which is not in inventory. User can configure API Protection or Rate Limiting rules for Shadow or other API Endpoints.
User Groups and User Group Roles
This release introduces support for configuring user groups and associated user group roles. For more information, see User Groups and User Group Roles.
Logs Streaming to AWS S3
This release introduces the capability to stream logs from a tenant to an AWS S3 bucket. Customers can have all the logs (sites, HTTP load balancers, etc.) sent to an AWS S3 bucket, allowing for an easy integration with SIEM solutions.
Note: Streaming of logs is only available for Organization plans.
Change to Logs Retention Period
The retention period for the logs is changed to 7 days. This change only applies to logs. The other events displayed in the dashboard (events, incidents, etc.) can be viewed up to 30 days back, with a 24 hours interval.
Support Site Mesh Group for AWS TGW Site
This release introduces support for Site Mesh Group connection type. Connection type can be privateIp
or publicIp
.
Service Discovery from Selective K8S/Consul Clusters in a Single Site.
When service discovery is configured on a given site, it discovers services from all the K8S/Consul clusters on the site by default. With the introduction of cluster-identifier configuration, users can assign an identifier for each K8S/Consul cluster. This allows users to control service discovery from one or more clusters thereby allowing service to be discovered from only subset of clusters on a given site.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
The Bot Defense does not function correctly when hash policy is enabled on load balancer (for example, source IP stickiness). As workaround, disable hash policy on the load balancer.
-
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
-
In case the Azure VNET Site deployment is failed with the
You have not accepted the legal terms on this subscription...
error message, perform the following even if the VM marketplace agreement terms were already accepted:- Select correct subscription; Enter the
az account set -s <subscription-id>
command. - In case of Ingress Gateway Site type, enter the
az vm image terms accept --publisher volterraedgeservices --offer volterra-node --plan volterra-node
command. - In case of Ingress/Egress Gateway site type, enter the
az vm image terms accept --publisher volterraedgeservices --offer entcloud_voltmesh_voltstack_node --plan freeplan_entcloud_voltmesh_voltstack_node_multinic
command. - In case of App Stack site type, enter the
az vm image terms accept --publisher volterraedgeservices --offer entcloud_voltmesh_voltstack_node --plan freeplan_entcloud_voltmesh_voltstack_node
command.
- Select correct subscription; Enter the
June 07, 2022
Last Updated: June 07, 2022
New Features
Site Management
Support for DC Cluster Group on Inside network
This release introduces support for DC cluster group on inside network. Sites can form a DC cluster group on the site local inside network.
Mesh
Disable Installation of Virtual Host if its Service Policy Set is Invalid
Virtual hosts can have references to one or more service-policy-set
objects. If any of these objects have invalid or incomplete configuration, the corresponding virtual host object will not get installed.
API Protection Rules in HTTP Load Balancer
API protection rules are introduced for HTTP load balancers. These rules can be defined in two categories. The first category includes fine-grained rules, per API path and methods. The second category includes rules per API groups or server URLs. If request matches any rule in the first category, second category rules are not evaluated. Rules can also include additional conditions. For example, specific clients can access certain API endpoint or API group.
Support Bot Defense Multiple Data Region for F5 Distributed Cloud Mesh Connector
This release introduces Bot Defense multiple data region support for F5 Distributed Cloud Mesh Connector through an HTTP Load Balancer. You will be able to configure Bot Defense to select the EU region for data residency. For more information, see Bot Defense.
Primary and Secondary Authoritative DNS support
This release introduces primary and secondary DNS zone support using F5 Distributed Cloud Services. You can now manage your DNS zones and leverage F5 Distributed Cloud Anti-DDoS and Anycast architecture to distribute DNS delivery globally.
Support for Explicit Knob to Disable/Enable Access for Services
Services such as SSH, DNS, and Web UI are allowed implicitly. User's ACL can not disable them. However, if user wants to disable any or all of them, configuration is introduced to support disabling services. These services can be disabled or enabled using the Blocked Services
configuration in the Fleet.
Console
Console Assistant for Enhanced UX
This release introduces Console Assistant (CoA) for UI. The CoA provides assistance to users in executing various features and services provided via the Console. The assistance is offered in the form of step-by-step guides for various scenarios, inline access to various resources, etc. See Console Assistant for more information.
Fixed Issues
Resolution for App Firewall Configuration Update Problem
The App Firewall occasionally missed configuration updates due to internal concurrency issue in one of the components. Fix for this issue is delivered in this release.
Resolution for Automatic Certificate Renewal Problem
The F5 Distributed Cloud provided certificates expire after a certain period of time and are automatically renewed before expiry. An issue caused certificates to not automatically renewed. Fix for that issue is delivered in this release.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
Unnecessary pod is created by NFV Service deletion on unrelated site due to bug in logic of destroy code. Upgrading the Site software to the latest version resolves this problem.
-
Before creating NFV service with pre-existing AWS TGW Site, it is required to execute
terraform apply
on that AWS TGW Site. -
The Bot Defense does not function correctly when hash policy is enabled on load balancer (for example, source IP stickiness). As workaround, disable hash policy on the load balancer.
-
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
May 10, 2022
Last Updated: May 18, 2022
New Features
Site Management
Removal of Assisted Deployment from Cloud Sites
The Assisted Deployment
option is removed from the user interface for all cloud site types AWS VPC Site
, Azure VNET Site
, GCP VPC Site
, and AWS TGW Site
. In case of trying to create a site with Assisted Deployment
mode via API, an error is returned.
Note: Existing sites can continue using the site in assisted mode.
Mesh
Support Longer Idle Timeout on HTTP Load Balancer
Support for longer idle timeout (up to 60 minutes) in HTTP load balancer for specific application is introduced for specific application use cases. The number of such load balancers is limited based on the http_loadbalancer.large_idle_timeout
value.
Public Server CA Rotation and Credentials Update
Public certificates are switched from current provider to a new provider. The existing vK8s & managed K8s kubeconfig
will stop working. Users with existing kubeconfig
are required to download a new copy for the credentials.
F5 Distributed Cloud(XC) Bot Defense with iApp Connector for BIG-IP
This release introduces the F5 Distributed Cloud(XC) Standalone Bot Defense service with iApp Connector(v3.0.2) for BIG-IP. For more information, see Bot Defense.
Site Mesh Group for Public Cloud Sites
Support for configuring site mesh group for public cloud sites is introduced. Supported site types are AWS VPC Site
, Azure VNET Site
, and GCP VPC Site
.
Support HTTP Header as Allow list(Trusted Client Rules)
This release introduces HTTP header as Allow list(Trusted Client Rules) to skip Bot Defense, WAF, or Both. For more information, see Bot Defense.
Support for Multiple Tunnels between Sites
Support for multiple tunnels between the F5 distributed cloud Sites is introduced. Up to three tunnels will be setup between the Sites, if the Sites have more than one master node. A single-node Site will be set up with a single tunnel.
Support for Site Mesh Group for K8s Site
Support for configuration of Site Mesh Group for K8s Site is introduced. Multiple tunnels (up to three) will be setup between the Sites, if the Sites have more than one master node. A single-node Site will be set up with a single tunnel.
Support for Site Mesh Group for K8s Site on OpenShift
Support for configuration of Site Mesh Group for K8s Site on OpenShift platform is introduced. Multiple tunnels (up to three) will be setup between the Sites, if the Sites have more than one master node. A single-node Site will be set up with a single tunnel.
New Identifiers for User Identification Rule
In addition to blocking users based on the IP Addresses, more granularity is enabled by introducing the following identifiers in the user identification rule:
- TLS fingerprint
- Client IP + TLS fingerprint
- Client IP + HTTP header value
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
The external service creation is not successful with pre-existing AWS TGW site. Before creating the external service with pre-existing AWS TGW site, terraform needs to be applied for that AWS TGW site so that the creation is successful.
-
The Bot Defense does not function correctly when hash policy is enabled on load balancer (for example, source IP stickiness). As workaround, disable hash policy on the load balancer.
-
In the
Client Classification
section of HTTP Load Balancer’sSecurity Monitoring
dashboard, the human requests count may show incorrect data. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
April 12, 2022
Last Updated: April 19, 2022
New Features
Site Management
Factory-Reset Using Hardware Push Button on ISV
Support for doing factory reset for ISV boards with revision R0D
and above is introduced.
Multi-Node Support for Kubernetes Site
This feature introduces multi ETCD and multi VER pods on multi-node K8s cluster and adds support for various debug tools in the Tools
section in site dashboard and metrics for the Kubernetes Site.
Ability to Run Virtual Machines on Managed K8s Sites
It is now supported to run VMs on managed K8s Sites (physical K8s/App Stack Sites). Use the virtctl
binary to interact with the VM to perform functions such as start, stop, pause, etc.
Mesh
Optional Matching of Server Name Indication (SNI)
F5 load balancer will not match the SNI if single HTTPS load balancer is advertised on a given VIP:Port
. It always offers certificate programmed on the HTTPS load balancer ignoring the SNI. If additional HTTPS load balancers are advertised on this VIP:Port
, then SNI match becomes mandatory and requests without server name extension are rejected.
Custom Response Code for WAF Blocking Page
This release introduces support to set a response code to be returned when an incoming request is blocked due to WAF security violation.
DC Cluster Group Support for Cloud Sites
This feature introduces support for connecting Cloud Sites using DC cluster group.
Note: This requires Layer 3 connectivity between the Cloud Sites.
Ability to Configure IP Reputation per HTTP Load Balancer
This feature adds support to enable IP reputation per HTTP load balancer.
Rotate Public LE Certficatess
F5 Distributed Cloud Services will switch public certificates from current provider to a new provider in the release of May 10, 2022. The existing vK8s & managed K8s kubeconfig
will stop working after the release of May 10, 2022. Users with existing kubeconfig
are required to download a new copy which will include both old and new CAs. This will ensure that the kubeconfig
continues to work when switching to new provider.
Support Mutating Webhook Configuration
This feature introduces support to enable mutating webhook configuration for the F5 Distributed Cloud Sites.
Allowed Request Rate per API Base URL, Path, and Method
The API rate limit rules are introduced in two categories. The first category includes the rules defined in API server and base URL level. In the second category, more granular rules such as per API path and methods can be configured. Each rule is composed of match conditions (domain, base path, API endpoint, and methods), allowed rate (requests per given duration), and user identifier. Requests matching the configured conditions are counted per defined user identifier.
Note: Only first match is counted per each category.
Simple Route Based on Host Header in the Incoming Traffic
Configuring header matcher to allow/deny traffic to an origin pool based on the incoming traffic header is introduced for HTTP load balancer. Multiple rules can be configured in this header matcher.
Handling of Accidental Deletion of Secret Policies
The following functionalities are introduced in case of deletion of secret policies:
- Secret policy soft-delete and recover facility added
- A custom list API is introduced for listing secret policies based on their state such as active or deleted
- UI is introduced with soft-delete in place of default crud delete
- UI is enabled with facility to recover the soft-deleted policy
Header Match for Routes in HTTP Load Balancer
This feature adds support to match on header in simple route, redirect route, and direct response route in case of HTTP load balancer.
Handling TLS Coalescing
Connection Coalescing also known as Connection Reuse is a mechanism to reuse same HTTP/2 connection for new requests. To support coalescing requests across load balancers, multiple load balancers configuration is merged to a single one when the load balancers have same certificate configured. If mTLS is configured, then this merging of load balancer configuration is not done. This means 421 (misdirected request)
is returned by the load balancer. The browsers are expected to initiate new connection on receipt of 421
. The other exceptions for not merging load balancer configurations are when they have different configurations for the following features.
- Path Normalization
- Server Header
Support for Deploying External NFV Service - F5 BIG-IP
This release introduces External Service Object which can be used to deploy F5 Big IP VE EC2 instance in the services VPC of an already deployed AWS TGW Site. The External Service object deploys F5 BIG-IP VE and sets up bootstrap configuration needed for the service instance to become functional. The AWS TGW Site acts as external network load balancer and distributes the traffic to F5 BIG-IP nodes. Traffic may be East-West (originating from apps running on spoke) or North-South (originating from Internet).
Note: This feature requires users to manage virtual server and security policy configuration on F5 BIG-IP.
App Stack
Advertise Local Prometheus in Managed Kubernetes
This feature allows accessing Prometheus API on local K8s API endpoint on route/prometheus. You can integrate with this endpoint to receive node and kube-state-metrics monitoring.
Upgrade to K8s Version
The Kubernetes main version 1.21.x support is introduced. It automatically upgrades all sites to this version via software upgrade.
Console
Network Policy Update
The sidebar entry for Network Policy under shared
and system
namespaces is renamed to vK8s Network Policy
and Firewall Policy
respectively.
Note: The
vK8s Network Policy
is supported only under application and shared namespaces. TheFirewall Policy
is supported only under thesystem
namespace.
Recently Visited Service Links
The Console is updated to show recently visited service links. Up to four recently visited services links are shown in the Select Service
drop-down to simplify navigation between services. Also, the All Services
links are removed.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
The Bot Defense does not function correctly when hash policy is enabled on load balancer (for example, source IP stickiness). As workaround, disable hash policy on the load balancer.
-
In the
Client Classification
section of HTTP Load Balancer’sSecurity Monitoring
dashboard, the human requests count may show incorrect data. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently, they are not enforced at the vK8s API server.
-
A recent software update to documentation may result in rendition/formatting issues in few documents.
February 22, 2022
New Features
Mesh
Enhancement for WAF Exclusion Rules
This release introduces enhancements in configuring exclusion rules for the WAF.
Certificate Revocation List Support
A CRL object can be created that configures HTTP server information reachable from site local network of a site. HTTP loadbalancer can refer to this CRL object whenever client certificate needs to be verified against a revocation list provided by the server. CRL file will be downloaded periodically and applied.
App Stack
Upgrade to k8s Version
This feature updated the K8s and vK8s main version support to 1.21. System automatically upgrades all sites to this version via the software upgrade.
Console
Managed K8s Overview Page
This release introduces a new page for the Managed K8s. The overview page is located at the Managed K8s
> Overview
option in the Multi-Cloud Network Connect
service in the Console.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
The Bot Defence does not function correctly when hash policy is enabled on load balancer (for example, source IP stickiness). As workaround, disable hash policy on the load balancer.
-
In the
Client Classification
section of HTTP Load Balancer’sSecurity Monitoring
dashboard, the human requests count may show incorrect data. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently they are not enforced at the vK8s API server.
February 03, 2022
New Features
Node/Site Management
Support Cloud Tags and Labels in Cloud View Sites
This feature allows to insert custom tags/labels in the Console for all auto-provisioned Sites. It also automatically add tags ves-io-creator-id
with value of creator (for example, user1@f5.com
) and ves-io-site-name
with value of name. In case of GCP, this feature stripes at-sign suffix and replace dots by underscores. For example, the creator value p.user1@f5.com
is converted to ves-io-creator-id: p_user1
.
Mesh
Ability to Correlate between WAF Custom Blocking Page and Security event
With this feature, user can add \{\{request_id\}\}
placeholder in the custom blocking page to include the request identifier from the WAF security event.
Client IP Reputation Capability Using Service Policy
This release introduces the service policy rule based on IP reputation. The rule can allow or deny traffic based on IP score and/or IP Threat categories.
Policy-Based Malicious User Mitigation
This release introduces Malicious User Mitigation settings for HTTP Load Balancers. User can customize the settings to have corresponding actions for threat-levels LOW
, MEDIUM
, and HIGH
. The supported actions are Javascript Challenge
, Captcha Challenge
, and Block Temporarily
.
Console
F5 Distributed Cloud Services Brand Update
This release introduces updates to product terminology in all resources of F5 Distributed Cloud Services, including updated UX for Console users. F5® Distributed Cloud Services denotes set of distributed cloud services/products such as the following.
- F5® Distributed Cloud Mesh (Mesh)
- F5® Distributed Cloud App Stack (App Stack)
- F5® Distributed Cloud Console (Console)
Note: This list does not show all names of services/products. See About F5 Distributed Cloud and Services for more information on products and services.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
In case of few IP addresses that might not be temporarily found in Webroot threat database, the IP reputation score might be inconsistent for the same IP address in different instances
-
Customizing of App Settings for functionalities such as malicious user mitigation is supported only for multip app load balancer and not for single app load balancer.
-
In the
Client Classification
section of HTTP Load Balancer’sSecurity Monitoring
dashboard, the human requests count may show incorrect data. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently they are not enforced at the vK8s API server.
January 13, 2022
New Features
Node/Site Management
OVF Property to Set Admin Password on Deployment
VMware OVF property to set admin password on deployment is introduced. It can be set from OVA form during provisioning and allows to fully automate deployment via terraform.
Mesh
Removal of CoreDNS Configmap on EKS
AWS introduced CoreDNS as an add-on for EKS clusters. When CoreDNS is an add-on, it resets any changes done to the Configmap of CoreDNS. This causes DNS Delegation to fail. Therefore, CoreDNS add-on is disabled while keeping CoreDNS service active.
Note: If CoreDNS add-on is deleted without the preserve flag, it will remove CoreDNS service itself. If a service discovery object is already present, then do one of the following: Execute the
eksctl delete addon --cluster <cluster-name> --name coredns --preserve --region <region-name>
command. Delete and re-create the discovery object. Restart the VER service from theTools
tab of the site monitoring page. Go toSites
->Site List
and click on your site to view the site monitoring page.
Remove Auto Generated Public CRUD for User Object
Some user management APIs exposed in earlier versions of software are now removed as they are no longer supported.
For example, instead of the https://www.volterra.io/docs/api/user#operation/ves.io.schema.user.API.Create
API, use the https://www.volterra.io/docs/api/user#operation/ves.io.schema.user.CustomAPI.Create
Custom API.
The following list of APIs are removed:
https://www.volterra.io/docs/api/user#operation/ves.io.schema.user.API.Create
https://www.volterra.io/docs/api/user#operation/ves.io.schema.user.API.Replace
https://www.volterra.io/docs/api/user#operation/ves.io.schema.user.API.List
https://www.volterra.io/docs/api/user#operation/ves.io.schema.user.API.Get
Note: Clients using the above APIs are required to use equivalent custom APIs for these operations. See API for more information.
Alert Policy Matching on any Alert Label
This feature allows users to configure alert policies with different alert labels as custom matchers and can configure receivers to send or block notifications based on matched alert labels.
Advanced AddOn Bot Defense in HTTP Load Balancers
This release introduces the Advanced AddOn Bot Defense feature for HTTP Load Balancers. For more information, see Bot Defense.
Define and Enforce API for the App Using its Swagger.
This feature introduces support for defining and enforcing API for application using its swagger files. After uploading application swagger files, you can create API definition which includes paths and methods from the swagger files. The API definition includes default API groups, all operations, and base URL. You can refer to the API definition and apply custom service policy in HTTP load balancer to allow/deny access to specific API groups. You can also tag operations in swagger by specific custom tag (x-volterra-api-group
). In this case, API definition builds multiple groups and allows higher granularity in policy rules.
Generic object store
This feature introduces ability to upload and download Open API Specification files. You can upload a swagger file via the Console in JSON or YAML format. The content is checked if it is a valid v2/v3 swagger. Each file version gets a unique URL in F5 Distributed Cloud Services and can be referred by other configuration objects such as API Definition. The user can list and download files in his tenant and required/accessible namespace.
App Stack
Permit K8s API Access to ClusterRole and ClusterRoleBinding
Managed K8s API server accesscontrol
is now enhanced to permit CRUD operations to manipulate ClusterRoles and ClusteRoleBindings. This can be enabled as advanced option on K8s Cluster configuration.
Console
Primary Navigation UX Enhancement
Console is enhanced with a new user experience. The primary navigation elements have been updated and the interface can be tailored to match any level of expertise. The new experience makes it easier for various teams to focus on tasks related to their respective roles. The following are the highlights of the new experience:
- Services - services are arranged into logical groups of tasks and functionalities that improve focus while making it easier to find and switch between services.
- Work domains - When you log in for the first time, you will be asked to choose your work domain. The user interface is tailored to this selection and the services related to that domain are filtered for easy access. You can also change your domain selection at any time.
- Home page - A new home page is created where you can access common, persona-related services accompanied by walkthroughs and solution videos.
For more information, see Getting Started.
Observability for Site Mesh Group and DC Cluster Group
This feature enables user to see enhanced sites topology connecting across different cloud providers (Site Mesh Group) and to check similar connection within a single data centre (DC Cluster Group).
Observability for L3-L4 DDoS
This feature enables user to have visibility into various L3-L4 monitoring Dashboards. The dashboards include the following:
- Top talkers
- Events list and details
- Alerts list and detail
- Mitigations list
- Details and Annotations
- Graphs by - Network, Application, Zone and Mitigation.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
In the
Client Classification
section of HTTP Load Balancer’sSecurity Monitoring
dashboard, the human requests count may show incorrect data. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently they are not enforced at the vK8s API server.
-
In case of new user signup using the Single sign-on (SSO), after first time successful login, you may get an error screen with
permission denied
or403
message. To resolve this, retry login after some time and a choice of plan selection is displayed. Upon completion of the signup flow, you will be able to login and start using the Console.
December 13, 2021
New Features
Node/Site Management
Enhanced Ability to Update HTTP Proxy After Registration
New CLI command configure-http-proxy
is introduced to update HTTP proxy and distribute the update to all K8s nodes at the same time instead of doing it one by one. The command sends the updated information to SaaS and reloads the software with new HTTP Proxy parameters.
Note: This action is the same as software upgrade. Therefore, rolling update for all software components is expected.
Create and Support Cluster on Multiple Certified Hardware
Creating and supporting cluster with multiple certified hardware options is introduced.
Mesh
Configuration knob for SNI Strict Check
The HTTP load balancer is updated with configuration option for enabling and disabling SNI strict checking.
Support Certificate Minting without Requiring Domain Delegation
This feature introduces automatic certificate management on load balancer without delegating a sub domain to F5 Distributed Cloud. When such a load balancer is created (with automatic certificate management enabled but domain not delegated to F5 Distributed Cloud), a CNAME record information is provided. This record is required to be created in the parent domain by the user. Certificate is minted once this CNAME record is added.
Support for Priority-Based Load Balancing Among Origin Pools
This feature introduces option to specify priority among origin pools. When the highest priority origin pool is not available either due to health check or due to discovery of retracting the endpoint, subsequent priority origin pool is used for handling requests. The priority-based switching of origin pool is pre-emptive. That is, when the highest priority origin pool is available, it is selected immediately for load balancing the requests.
Support configuring OCSP Stapling in HTTP Load Balancer
This feature enables user to configure OCSP Stapling when using HTTPS proxy with custom certificates. The option to disable OSCP Stapling, enable OCSP Stapling with system configuration, or enable OCSP Stapling with custom order of Hash Algorithms is introduced.
Support Configuring AI Options Directly from HTTP Load Balancer
This feature enables the user to configure, for an HTTP load balancer, AI options like API discovery, malicious user detection and ddos detection directly from the HTTP load balancer form.
Replacement of Rule-Based WAF to Signature-Based Advanced WAF
This feature replaces the rules-based Web Application Firewall (WAF) with a much advanced application firewall that supports classification and detection for attacks that are based on signatures, provides bot protection, detects security violations, and also offers controlling response traffic. The WAF also provides protection against threat campaigns, supports enabling/disabling false positive suppression, and allows to mask sensitive data in request logs.
For more information on the new WAF, see WAF and Configure App Firewall.
Path URL Normalization
HTTP load balancer is enhanced to normalize the Path URL according to RFC 3986. Additionally, the slashed is merged when path URL normalization is enabled.
Path normalization is supported with an option to enable it for HTTP load balancer of type HTTPS proxy. In case of HTTP proxy, it is enabled by default.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
-
In the
Client Classification
section of HTTP Load Balancer’sSecurity Monitoring
dashboard, the human requests count may show incorrect data. -
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently they are not enforced at the vK8s API server.
-
In case of new user signup using the Single sign-on (SSO), after first time successful login, you may get an error screen with
permission denied
or403
message. To resolve this, retry login after some time and a choice of plan selection is displayed. Upon completion of the signup flow, you will be able to login and start using the Console.
August 26, 2021
New Features
Mesh
SSL Renegotiation for Origin Pool
This features enables SSL renegotiation by default in case of origin pools.
App Stack
Enhancement to Cluster Role and Role Binding Manifest
This release introduces support for adding the :
character to the YAML format for the K8s cluster and cluster role binding manifests.
Changes to Default Behavior
None.
Caveats
The following caveats apply:
- AWS VPC site may operate as open recursive DNS resolver, conflicting with AWS Acceptable Use Policy (AUP).
- vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently they are not enforced at the vK8s API server.
August 5, 2021
New Features
Node/Site Management
Support Licensing VMware Site with NVIDIA vGPU
This features enables unlocking full GPU performance for a VMware site with licensing support. You can enable this by adding VMware site to a fleet that has vGPU enabled in its advanced configuration section. Also ensure that you add license server address and license type.
Mesh
Health Check Support for HTTPS-Enabled Origin Pool
This feature introduces support for health checks for origin pools enabled with HTTPS. The TLS configuration of the origin pool is used to make TLS connection with the upstream server for the periodic health check requests.
Monitoring for Site Networking
This release introduces Site Networking monitoring views. The dashboard view provides a summary view of site metrics dividied into sections and each section shows Top 10 Sites for that metric. The following metrics are presented:
- Data Sent
- Data Received
- Tunnels by Latency
- Tunnels by Drop Rate
- Tunnels by Throughput
The dashboard also provides table view for data for the following:
- Data when data plane reachability < 75%
- Data when control plane status is down
- Data when tunnel health < 70 score.
Note: The view also shows the tunnel alerts in the dashboard.
App Stack
Support Custom storage class
This feature introduces new storage device and storage class type called Custom, which allows to insert K8s storageclass manifest. This gives ability to enforce own storageclass names or integrations with 3rd party storages.
Console
Notification Preferences for Users
This features allows to set user notification preferences for the following notifications:
- Access Requests
- Product updates
- System Maintenance
You can select a notification to receive or deselect to disable receiving notifications for it. The notification preference setting is available in the General
-> Personal Management
-> My Account
page.
Changes to Default Behavior
None.
Caveats
vK8s Role and Role Binding currently does not support user level K8s RBAC and is applicable only for vK8s workloads. vK8s Role and Role Binding are propagated to the K8s cluster and enforced at the K8s cluster only. Currently they are not enforced at the vK8s API server.
July 15, 2021
New Features
Node/Site Management
Ability to Specify Total Worker Nodes Irrespective of Availability Zones
This feature adds support for specifying total number of work nodes, irrespective of the availability zones in case of Azure sites.
Addition of Alternate Region for Azure Sites
This feature introduces support to use the following alternate regions for the Azure VNET sites:
- northcentralus
- koreacentral
- centralindia
- southindia
- australiacentral2
- australiacentral
- southafricanorth
- norwayeast
- swedencentral
- switzerlandnorth
- uaenorth
- uaecentral
- switzerlandwest
- norwaywest
- germanynorth
- francesouth
- canadaeast
- koreasouth
VMware Site Deployment Enhancements
The default certified hardware for VMware site is changed to simplify user experience. The updated certified hardware is called as vmware-regular-nic-voltmesh
. This contains by default 2 virtual interfaces eth0 (site local outside) and eth1 (regular interface). The eth1 is optional interface, and can be configured only from Console through the network interface object.
Mesh
Storing All Request Logs
This feature enabled storing all request logs without any sampling on cold storage by default.
Handling of HTTP2 Coalescing
Client browsers may employ a performance optimisation known as connection coalescing. In case 2 virtual hosts with different domain names are hosted on the same IP address and present in the same TLS certificate, the HTTP/2 connection is reused between them when connection coalescing is used.
System detects such case and returns 421(Misdirected request)
error to the client browser. Client browser receiving a 421 (Misdirected Request)
response may retry the request over a different connection.
Note: Handling of
421
error is browser-specific. Not all client browsers handle this error code.
App Stack
Support NVIDIA Tesla T4 on VMware ESXi
This feature introduces support for NVIDIA Tesla T4 on App Stack on VMware ESXi on certified commodity hardware. By enabling fleet vGPU option, The Site switches runtime to NVIDIA and can deploy vGPU workload applications.
Upgrade to k8s Version
This feature updated the Kubernetes main version support to 1.19.11. System automatically upgrades all sites to this version via the software upgrade.
Console
Credential Create API Expiration Field Update
The expiration option for create and renew API for my credentials and service credentials is changed from expiration_timestamp
to expiration_days
. In case you are using API to download the credential, ensure that you use the updated expiration field. See Credentials guide for more information on using the credentials.
Note: This effects only users of the API and not the Console users.
Changes to Default Behavior
Expiration field for create and renew APIs of my credentials and service credentials is changed from expiration timestamp to number of days. Use the updated field expiration_days
for create and renew API requests.
Caveats
None
June 24, 2021
New Features
Node/Site Management
Support upgrading to a specific version
This feature allows specifying a particular software or operating system version during a site upgrade.
Note: You must obtain the correct version information before setting it for site upgrade.
Mesh
Deprecation of Virtual Host Object
The virtual host object creation is deprecated for most of the virtual host types except UDP and SMA proxy types.
Site Service Network Support
HTTP and TCP Load balancers can be configured to be advertised on vK8s network on F5 Distributed Cloud ADN. However, for TCP Load balancers, there is a limitation that no two TCP Load balancers can be advertised on the same port. Also, the HTTP and TCP Load balancers can only be advertised on the site local outside or site local inside networks of a customer site.
The site service network allows multiple TCP Load balancers to be advertised on the same port on vK8s network on the ADN. It also allows HTTP and TCP Load balancers to be advertised on vK8s network on customer site.
Console
Implicit Role for Tenant owner
An implicit system assigned role ves-io-tenant-owner-role
is added to all the tenant owners. All existing and new tenant owner users will be assigned with this role in the system namespace. This is internally assigned for tenant owners and cannot be assigned to normal users. A tenant admin can go to General
-> IAM
-> Users
and click ...
-> Upgrade to Tenant Owner
for a user to get this role added implicitly to the user.
Support SSO-based Signup for Free and Individual Accounts
This features allows users to signup for Free or Individual plan using Single Sign-On (SSO). As part of the initial support, sign in via Google is enabled. Users can use their existing Gmail/Google account credentials to authenticate when they choose SSO.
Changes to Default Behavior
None
Caveats
Routing to workload subnet in AWS site fails. The following is a workaround:
In case of AWS VPC Ingress-Egress Gateway and AWS TGW site, for the successful routing towards applications running in workload subnet, an inside static route to the workload subnet CIDR needs to be added on the respective site object.
June 3, 2021
New Features
Node/Site Management
Enable Site Local K8s API access for VoltStack Cluster Cloud Sites
This feature allows tenant to enable site local K8s API access for AWS, Azure, and GCP VoltStack cluster cloud view sites.
This provides same ability as VoltStack site to link K8s cluster on cloud site and access native K8s API server. See VoltStack Site for information on how to enable site local K8s API access for VoltStack site.
Mesh
Ability to Disable Advertisement of Services on Public Internet
This feature introduces the ability to disable advertisement on the public internet by default. This prevents unintended data leak to the public. Users are required to open a support ticket to use this feature.
Control Communication across Namespaces Using Implicit Namespace Label
This feature introduces implicit labels for namespaces. These labels can be used by administrators in service policies and network policies to control communication between namespaces.
Note: All objects in a namespace get the implicit label. This label cannot be modified by the user.
Inside and Outside VIP Enhancements for Multi-node Cloud Sites
This release introduces support for using AWS and Azure loadbalancer IP addresses as the Site-Local Outside (SLO) or Site-Local Inside (SLI) VIP addresses to reach the applications advertised using F5 Distributed Cloud load balancer on the multi-node sites.
In case of AWS sites, you are required to configure allowed VIP port configuration on multi-node AWS VPC site. This is to explicitly specify which ports are going to be used while configuring the F5 Distributed Cloud loadbalancer. After this is configured, you can use the AWS loadbalancer IP address as the SLO/SLI VIP. Also, using AWS loadbalancer frontend IP as the VIP to external K8s or Consul cluster is supported. This is when a discovery object with VIP publishing configuration is enabled on the AWS site.
In case of Azure sites, you can use Azure loadbalancer IP as the SLO/SLI VIP to reach applications advertised using F5 Distributed Cloud loadbalancer on Azure multi-node sites.
Changes to Default Behavior
None
Caveats
The UI does not support the option to update inside/outside VIP port configuration for AWS VPC Site. However, you can perform the updates using either of the following ways:
-
Using terraform run custom API.
-
Using the following vesctl ciommand
vesctl request rpc terraform_parameters.CustomActionAPI.Run --http-method POST --uri /public/namespaces/system/terraform/aws_vpc_site/<site-name>/run --json-data '{"namespace":"system","view_kind":"aws_vpc_site","view_name":"<site-name>","action":"APPLY"}'
May 13, 2021
New Features
Node/Site Management
Factory Reset Using Hardware Push Button on IGW
This feature adds support to do factory-reset on the IGW 5000 series using the hardware reset button. Press the button continuously for 5 seconds to trigger factory-reset.
App Stack
Nodes View for Managed K8s
This feature adds Nodes
tab to the monitoring of F5 Distributed Cloud's managed k8s cluster. This tab will give details about nodes in the cluster.
Managed K8s Monitoring Enhancements
This feature allows monitoring of the managed K8s cluster even when API access from Console is disallowed. This is done using metrics collected from the cluster and the monitoring dashboards appear different compared to when API access is allowed. The K8s monitoring is shown as Monitor K8s cluster
when API access from Console is allowed. It is shown as Monitor K8s cluster(with metrics)
when API access from Console is disallowed.
Global Kubeconfig for Managed K8s
This feature allows to download Kubeconfig for a the managed K8s cluster from Console gateway. Log into Console, navigate to Sites
-> Site List
in the system namespace, and click ...
-> Download Global Kubeconfig
for your App Stack site enabled with managed K8s.
Console
F5® Application Traffic Insight on F5® Distributed Cloud Console
F5 Application Traffic Insight (ATI) is a real-time, high-precision device identifier that utilizes advanced signal collection and machine learning algorithms to assign a unique identifier to each device visiting your site. This feature introduces ATI on Distributed Cloud Console and associated monitoring to view various dashboards of devices visiting your site. For more information, see ATI.
Improvements in Subscription Plan Transition Workflow
This feature improves the transition between teams plan and organization plan by enhancing error handling. This is applicable while upgrading from teams to organization plan and vice versa.
Changes to Default Behavior
None
Caveats
None
April 22, 2021
New Features
Mesh
F5 Distributed Cloud Services Direct Connect Enhancement
F5 Distributed Cloud Services support enabling direct connectivity to the backbone network. This feature enhances the direct connect functionality by adding support to advertise and discover services. Advertising of services is supported using HTTP and TCP load balancers.
Active Alert Policies in Namespace
This feature simplifies configuration of active alert polices in a namespace. A new API on the namespace is added that takes a list of alert policies and makes them active in that namespace. Corresponding UI enhancement is also added.
Managed Certificate option for HTTP Connect & DRP proxy
Users can use the TLS interception feature for HTTP Connect & DRP proxy without managing custom certificates with this feature. To use this feature, users need to simply select the Volterra Managed Signing Certificate
option in the downstream certificate configuration for TLS intercept configuration. After that they can use Download CA Certificate
menu item for the HTTP Connect & DRP proxy to download and use it from the browser and non-browser clients.
Simplified BGP Object Configuration
Configuring BGP object is simplified by removing invalid field combinations. The updated configuration form also allows the user to select interfaces per-peer instead of specifying a single list of interfaces for all peers.
Node/Site Management
Install CE with Specific Software Version
This feature adds support for specifying a specific software version and a specific operating system version when bringing up a Customer Edge (CE) site. For view based sites, the versions can be specified when creating the site or during registration approval. For other site types, the versions can be specified during registration approval.
Changes to Default Behavior
None
Caveats
None
April 01, 2021
New Features
Mesh
FQDN Support in Tunnel Configuration
This feature introduces using Fully Qualified Domain Name (FQDN) in case of establishing IPsec/SSL VPN connection to the Regional Edge Sites. A Site can be configured to use IPSec/SSL VPN with an option of going through a site proxy. This feature now allows server FQDN in site configuration in addition to IP address. This FQDN gets resolved to establish VPN connection.
In case of proxy configuration being used with OpenVPN tunnels, FQDN is sent in the HTTP Connect request to the configured proxy server. Proxy server is required to resolve the FQDN and relay the connection to the final destination server to establish OpenVPN tunnel.
Cluster Retraction
This feature allows configuring fallback in case of endpoints behind a cluster are not healthy and route points to multiple clusters as part of weighted cluster configuration. In such case, the traffic is distributed only among the remaining clusters that have one more healthy endpoints.
Load Balancing AWS Egress Traffic
This feature introduces load balancing of egress traffic to all the nodes in case of AWS Egress gateway site.
App Stack
Site Status in vK8s Pods View
vK8s monitoring is enhanced to show the site status in the Pods
view. Site status is shown in colored dots in the Node name
column. Healthy sites are shown in green color, unhealthy sites are shown in red color, and if health information is not available, grey color is shown for those sites.
Note: If node status is down, then pod status should be considered as unavailable even if it shows as available or running.
Changes to Default Behavior
None
Caveats
None
March 11, 2021
New Features
Node/Site Management
Monitoring GPU Status
Node monitoring is enhanced to display the GPU status in the node status dashboard view. GPU status shows information such as temperature, power, CPU utilization, throughput, etc. Find the GPU status by navigating to Sites
-> Site List
, click on your site to load its dashboard, select the Nodes
tab, and click on the node for which you want to monitor GPU status.
Mesh
F5 Distributed Cloud Direct Connect
This feature allows tenants to have a direct connecting link to the REs. This enables the CE to RE connectivity to be on the direct private link instead of being in public.
Allow VIP Usage for DNS Resolution
This feature allows VIP address to be used as DNS server to resolve domain names configured in the load balancers. DNS queries can be sent to the VIP addresses configured on CE. The system software on the CE runs DNS server on the VIP addresses and resolves queries for domain names configured in the load balancers. It will also forward other requests to external DNS servers.
Server Response Header Manipulation for HTTP Load Balancer
Support for manipulating server token in the HTTP load balancer response headers is introduced. User can now configure the response header to do the following:
- Set a default value
- Set a specific value
- Append a server name if no server header is not present
- Set to pass through if the server header is present
Malicious User Mitigation Enhancements
Malicious user detection and mitigation is enhanced in the HTTP load balancer monitoring. In the Security Monitoring
view, the Malicious Users
tab now allows admin to view and act upon malicious users identified by a user-identification object or the source IP address in case a user identification object is not defined. The following monitoring functionalities are added:
- Malicious user security events
- Activity timeline for the identified user
- Activity that contributed to the current suspicion score
- Time series variation for the suspicion score
- Options to block or whitelist users
F5 Distributed Cloud Services use advanced machine learning techniques and analyzes information to identify the malicious users. Analysis is performed on information such as WAF security events, forbidden access events, failed login attempts, and anomalous behavior.
Changes to Default Behavior
None
Caveats
None
February 18, 2021
New Features
Node/Site Management
Monitoring GPU Status
Node monitoring is enhanced to display the GPU status in the node status dashboard view. GPU status shows information such as temperature, power, CPU utilization, throughput, etc. Find the GPU status by navigating to Sites
-> Site List
, click on your site to load its dashboard, select the Nodes
tab, and click on the node for which you want to monitor GPU status.
TGW Service Policy for East-West Traffic
The AWS Transit Gateway (TGW)site allows for attaching multiple VPCs and forwarding of traffic between VPCs. This feature introduced support for service policy on the VPC-to-VPC traffic or east-west traffic and can be set in the security configuration section of TGW configuration wizard. User can enable the east-west service policy in the Manage East-West Service Policy
section and attach a service policy. The service policy can be created in system namespace in the Security
-> Firewall
-> Service Policies
page. It can also be created and attached from within the TGW configuration wizard.
Note: User can also enable east-west service policy with allowing all traffic to be sent via proxy.
Mesh
Policy Based Security Challenge
Support for enabling policy-based security challenges is introduced. User can now set policy based challenge in load balancer configuration and specify whether to always enable a challenge or disable it while also setting override rules for specific match conditions. Both javascript challenge and captcha challenge are supported. The matching parameters include IP, domain, path, peader, query parameters, etc. These are similar to the parameters in service policy rules.
Note: The security challenge can be enabled in the advanced configuration section of HTTP load balancer configuration. See Configure Javascript Challenge for more information.
Server Response Header Manipulation for HTTP Load Balancer
Support for manipulating server token in the HTTP load balancer response headers is introduced. User can now configure the response header to do the following:
- Set a default value
- Set a specific value
- Append a server name if no server header is not present
- Set to pass through if the server header is present
WAF Rule Exclusion for Security Events
Support to set WAF rules for exclusion in HTTP load balancer security events is introduced. User can now select security events and create an exception rule for them from the HTTP load balancer monitoring page in Console. Navigate to Virtual Hosts
-> HTTP Load Balancers
in your namespace and click on your load balancer. Select Security Events
tab and click ...
-> Create Exception Rule
for the security event entries for which you want to enable the WAF rule exception.
Note: Creating exception rule for an event will open HTTP load balancer configuration form with the WAF excluded rule added to the security configuration section. Click
Save and Exit
to update the configuration.
IP/User Blocking Rules for HTTP Load Balancer
Support for whitelisting or blocking specific clients for HTTP load balancer is introduced. The load balancer configuration is added with client blacklisting rules and trusted client rules sections. User can set to block or whitelist specific clients based on the IP addresses or AS numbers.
Route Options for TCP Load Balancer
TCP load balancer is enhanced to set load balancing schemes for the traffic to the origin servers. The schemes supported are round-robin, least active, random, and hash of source IP.
BGP Peering in Multiple Networks
This feature introduces ability to do BGP peering on multiple networks on a customer edge site. Networks could be site local, site local inside, or per site networks.
App Stack
Default Workload Flavor for vK8s
Support for setting default resource limits for vK8s containers is introduced using a default workload flavor object. User can now create a workload flavor object in the shared namespace at the Manage
-> Workload Flavors
page and attach it as a default limit in vK8s configuration. See Create Default Workload Flavor for more information.
Physical K8s Access for VoltStack Site
This feature gives ability to access customer edge (CE) K8s cluster through a kubeconfig file on the local network. Using this feature, user can deploy applications that can manage kubernetes workloads on the CE K8s cluster.
Changes to Default Behavior
None
Caveats
None
January 21, 2021
New Features
Node/Site Management
VoltStack Site Support
This release introduces support for creating Data Center (DC) or physical hardware edge sites using the VoltStack site object from Console.
Stream Logs to External Service
Support for stream request logs to syslog service is introduced. User can now create a log receiver object and attach to the fleet of sites. Log receiver object can be created in Console in the Manage
-> Site Management
-> Log Receivers
.
Note: The host IP of the external service must be reachable from the Site.
AWS View Site Enhancements
The following enhancements are added to AWS VPC site and AWS TGW site:
- Configuring workload subnets
- Configuring worker nodes - Supported only for AWS VPC site
- Site admin state field is added in the
Manage
->Site Management
->AWS VPC Sites
page and also in the JSON view for the AWS VPC site object.
AWS TGW Site Monitoring Enhancements
The Sites
-> Connectivity
page view for the AWS TGW site is enhanced with representing the site with transit gateway, tunnels, and attached VPCs. Also, the details view for the AWS TGW site is enhanced to show the information on tunnels to TGW. This information includes the data transfer, throughput, and BGP connection status.
USB Whitelisting with Fleet
Support for whitelisting USB devices from Fleet is introduced. Users can now create a USB policy to allow specific USB devives and apply the policy using Fleet.
Site Local UI Enhancement
Site local UI URL is enhanced to be more usable. User can now access site local UI using the https://volterra.local:65500
URL. For more information on using site local UI, see Site Local UI guide.
Mesh
Active Policies in Application Namespace
Support for adding active network and service policies in the application namespace is introduced. User can add active policies in the Security
-> vK8s Network Policy
-> Active Network Policies
and Security
-> Service Policy
-> Active Service Policies
pages.
App Stack
PVC Disk Usage
The virtual K8s (vK8s) dashboard and PVCs view are enhanced to display the disk usage of PVCs.
Changes to Default Behavior
None
Caveats
Deploying F5 Distributed Cloud Sites on the same broadcast domain/subnet with other F5 Distributed Cloud sites/devices enabled with VRRP is not supported. This will be supported in a future update.
December 17, 2020
New Features
Node/Site Management
Operating System Update
The OS is updated with CentOS release 7-9.2009 and kernel release 4.18.0-193.28.1.ves1.el7.x86_64
Proxy Support for VMware CE Site
Support for configuring HTTP Proxy for VMware CE site is introduced. During the initial configuration using CLI, user can set HTTP proxy. For more information on VMware site installation, see Create VMware Site. Download the latest image from the VMware Site Images page.
Site Monitoring Enhancements
- The tcpdump collection is improved for usability in the
Tools
tab of the site monitoring page. User can now start, fetch, and stop the tcpdump from single page for a selected target. - Traceroute utility is added to the
Tools
tab of the site monitoring page.
Note: Navigate to
Sites
->Site List
in theSystem
namespace and click on any site to display its monitoring view. The dashboard tab is loaded by default.
Mesh
Fast ACL Updates
The following updates are made to Fast ACLs:
- Fast ACL set table list is removed from the Console
- Fast ACLs for Internet VIPs object is introduced. The Fast ACL objects can be directly added to this.
DNS Management Enhancements for Load Balancers
In addition to supporting the HTTPS load balancer, the F5 Distributed Cloud DNS management is extended to TCP and HTTP load balancers. With this, users can now delegate domain to F5 Distributed Cloud and use the domains in load balancer of type TCP and HTTP.
AppType Creation for Application Namespace
App Type object support is enhanced so that it can be created from within the App Settings object in the application namespace.
Service Discovery for Kubernetes Headless Service
The service discovery is enhanced to discover and route to headless K8s services without depending on K8s DNS.
Note: This requires Layer 3 routing to be established between the CE site and the K8s pods.
App Stack
vK8s Resource Enhancements
The virtual K8s (vK8s) is enhanced to configure daemonsets, cronjobs, and service accounts.
vK8s Deployment Quota Increment
The deployments per vK8s is increased to 25.
Isolation for vK8s Services Across Namespaces
Support for restricting communication between vK8s services belonging to different namespaces is introduced. User can enable namespace isolation in the vK8s configuration and override this behavior for specific services by setting the ves.io/serviceisolation
annotation to false for that service.
Changes to Default Behavior
Cloud View Site Object Management Updates
The cloud sites created using the Manage
-> Site Management
page are updated to be edited and deleted only from the cloud site object menu and not from the Sites
-> Site List
menu. Navigate to Manage
-> Site Management
and click ...
for your cloud site object to edit or delete the object and this in turn applies the operation on the sites.
November 25, 2020
New Features
Node/Site Management
Enhanced HA on SLI
Node mastership is now based on all configured VIPs across Site Local Outside (SLO) and Site Local Inside (SLI) interfaces.
Local UI Enhancements
Introduced status and tooling enhancements to the local UI dashboard of F5 Distributed Cloud Site.
Mesh
Automatic API Schema Generation
Introduced per API endpoint Swagger API schema documentation generation. This can be found under App Namespace
-> Mesh
-> Service Mesh
-> API Endpoints
-> Endpoints Details
-> Swagger
.
Active Service Policies for HTTP Load Balancers
Introduced the ability to define active service policies for a specific HTTP Load Balancer. You can choose one of the following service policy options for the load balancer:
- Set a default service policy
- Apply active service policies
- Disable the active service policy
IP Prefix & Prefix List Options for Forward Proxy Policy
Introduced ability to match destinations based on IP prefix and IP prefix lists under the custom rule list of the forward proxy policy.
BGP ASN and GeoIP Support for Forward Proxy Policy
Introduced ability to create a forward proxy policy matching on a specific BGP AS, ASN list, and GeoIP labels.
Forward Proxy Support for Global Networks
Introduced support for configuring forward proxy in the network connector when connecting Site Local Inside (SLI) to Global Network Type VNs.
App Stack
Enhanced vK8s Workload Dashboard
Enhancements are added to the vK8s workload dashboard under App Namespace
-> Applications
-> Virtual K8s
-> Workloads
.
Container Registries
Introduced the ability for users to configure private registries for their vK8s workloads.
Console
Flow Table Under Site Management
Introduced the ability for the user to view existing flows per node.
Sidebar Navigation Enhancements
Several enhancements are added to the UX of the sidebar in Console.
Tooling
Beta Release of Public Terraform Provider
Introduced beta support for F5 Distributed Cloud's public terraform provider. See Terraform Provider for more information.
Changes to Default Behavior
Change to Packaging and Management Providers
In case of a CE Site behind a firewall that is performing URL filtering, ensure that you update it with the latest domains listed in the Network Cloud Reference page.
November 5, 2020
New Features
Node/Site Management
Upgrade Guided Sites (AWS VPC/TGW, Azure & GCP) directly from Site List
Introduced support for users to directly upgrade site deployments via Site Management
for AWS/Azure/GCP/TGW sites and also from the Site List
page for sites.
GCP and Azure support for VoltStack Cluster Deployment Option
Enhanced the Site Management
page for Azure VNET & GCP VPC to support a 3rd deployment option called VoltStack Cluster (One Interface)
.
Site Health Calculation Enhancements
Enhanced health score calculation to take Site Admin
state into account.
Mesh
TLS interception support for HTTP Connect & DRP
Introduced support for TLS interception when configuring an HTTP Connect or DRP (Dynamic Reverse Proxy) virtual host.
Descriptions for Policy Rules
Introduced logging of the description field for the configured policy in the hit logs. The policies include service policy, forward proxy policy (simple and custom rule set), network policy, and secret policy.
AWS TGW - East - West Forward Proxy Support
When provisioning an AWS TGW Site, East-West traffic now supports forward proxy policies by default.
App Stack
vK8s Workload & Jobs View Enhancements
Enhanced the vK8s workload & Pods table view to include deployment name, running pods, total pods, total sites, sites with error, sites without pods, virtual site, upgrade, and actions.
vK8s Virtual Site Descriptions
During vK8s virtual site selection, the selection table now shows descriptions for the virtual sites (system or user created).
Console
Site Security Dashboard
Introduced the beta version of the site security dashboard. This view provides tenant and site level firewall events and logs. This is available at Sites
-> Site Security
.
API Endpoint Enhancements & Fixes
Enhanced UX and navigation of endpoint details in the API Endpoint page.
Notification Dashboard Enhancements
Enhanced Alerts
and Audit Logs
pages under Notifications
section.
Revoking API Certificates and Kubeconfig
Support for revoking API certificates and Kubeconfigs is introduced. In case of API certificates and Kubeconfigs created prior to this release, you might receive the Client certificate is invalid or revoked
response for API requests. In such case, create new certificates and download for use.
F5 Distributed Cloud Services Hardware
ISV 8000 Series GA
The Industrial Server (ISV) 8000 is now Generally Available. The Industrial Server is a series of ruggedized edge computing devices providing hyper-converged compute, GPU, storage and networking. They are easy to deploy and operate systems capable of running learning, inference, containerized or legacy (VM) workloads—from manufacturing plants to retail stores and small branch offices. The Industrial Servers combine the capabilities of hyper-converged infrastructure (HCI) with a GPU for machine learning and robust connectivity (4G LTE/GPS/Wi-Fi/Bluetooth) in a single ruggedized device designed to meet the rigorous demands of edge and industrial environments. You can learn more about the Industrial Server from the data sheet here and the User Manual here.
Changes to Default Behavior
The System
-> Security
-> Advanced
page is deprecated.
October 14, 2020
New Features
Node/Site Management
Enhanced Remote Tooling (show service status)
The user can now query service specific status on a Per Node basis from Console. System -> Site -> Tools -> Show services status
Default Fleet
During CE setup the user can now configure a default ves.io/fleet type. This is helpful in scenarios where CEs required a basic working configuration on CE registration (i.e., Local breakout).
AWS TGW Site
Console now supports the deployment of Sites and management of AWS TGW's. System -> Site Management -> AWS TGW Site.
GCP VPC Site
Console now supports the deployment and management of Sites in GCP. System -> Site Management -> GCP VPC Site.
Site Wizard Improvements
The Site Wizard Page has been improved for better UX, readability and error/status reporting.
Mesh
DDoS forensics and analysis
DDoS forensics and analysis for Load Balancers and Site (Forward Proxy) Enhanced ability to perform forensics and analysis of configured HTTP & TCP Load Balancers and per Site Forward Proxy.
Enhanced Alerting of DoS/DDoS
Using Time Series Analysis (TSA) of the Request Rate, Response Throughput, Latency and Error Rate anomalous enhanced DoS/DDoS alerting has been enabled.
HTTP/HTTPS on additional ports
This release has added additional HTTP & HTTPS ports to be advertised on F5 Distributed Cloud's REs (Public Network). Supported HTTP ports are 80 8080 8880 2052 2082 2086 2095 25565. Supported HTTPS ports are 443 2053 2083 2087 2096 8443 25565.
Forward Proxy in Denied Rules Hit
Site Dashboard Denied Rules Tile now includes Forward Proxy. The site dashboard Denied Rules tile now includes Forward Proxy as an option, in addition to Service & Network Policy.
App Stack
VoltStack DC Cluster
Guided Configuration for F5 Distributed Cloud DC Cluster - This feature brings in vK8s application deployment workflow to ease deploying applications on F5 Distributed Cloud Services platform. The interface given caters to the developers, provides application level interface and hides some of the underlying infrastructure related tasks.
Storage
Storage Device Support - This feature brings support for Dell EMC Isilon F800 & HPE Nimbus Storage AF40, this is configured in the Fleet object under Storage Configuration.
Simplified Workload Deployments on vk8s
Simplified Workload Deployments on vk8s - This feature brings in vk8s application deployment workflow to ease deploying applications on F5 Distributed Cloud Services platform. The interface given caters to the developers, provides application level interface and hides some of the underlying infrastructure related tasks.
F5 Distributed Cloud Services Hardware
NVIDIA GPU support on ISV 8000 Series - Updated the ISV Certified Hardware Profiles to download to support NVIDIA GPUs.
Console
New User Type: Debug User
There is a new user type called "Debug User". This allows the tenant admin to provide the F5 Distributed Cloud Support team access to the tenant to enhance troubleshooting.
New Alert Receivers (SMS/Email)
Email and SMS are supported receivers under Alert Management.
Enhanced Connection Log Views
The connection log page has been enhanced to render the data in a more user friendly format.
Upcoming Changes to Default Behavior
In the planned November release, the System -> Security -> Advanced will be deprecated.
Caveats
In case of node hardware, the USB device whitelisting is enabled by default. Connecting a new device after registration of the node does not work.
Note: You can see the USB devices by navigating to your site dashboard via
Sites
->Site List
path. Open theNodes
tab and click on a node to open its dashboard view. ClickHardware Information
tab to see the USB devices list.
September 24, 2020
New Features
Node/Site Management
Per Node Tooling from Site Dashboard
The site dashboard in Console allows additional troubleshooting and status commands to be executed remotely.
Fleet Configuration Enhancements
Fleet Configuration and related objects (Network Interface, Virtual Networks, Network Connectors, Network Firewall, Network and Forward Policies) can be initially configured during Fleet creation. This is configured under System
-> Site Management
-> Fleets
.
For information on fleet configuration, see Create Fleet.
Mesh
Fast ACL Configuration Enhancements
Guided form is introduced to enable easier configuration of fast ACLs. See Fast ACLs for configuration instructions.
Hub Group Only Mesh
For smaller deployments, it is desired to configure site-to-site mesh groups without a hub & spoke model. This release introduces the ability to configure a mesh with a hub group only.
HTTP Connect & Dynamic Reverse Proxy Wizard
Guided forms are introduced to enable easier configuration of HTTP Connect & Dynamic Reverse Proxy under the <Namespace>
-> Manage
-> Load Balancer
.
App Stack
vK8s Dashboard
The vK8s dashboard is updated for a better UX experience and end-to-end view of pods deployments, statistics, and health.
F5 Distributed Cloud Services Hardware
IGW 5000 Series
General Availability (GA) Support for the Industrial Gateway 5008 & 5508 series is introduced.
Console
Site List & Connectivity Enhancements
Updates are made to the default System
-> Sites
-> Site List
page to provide clear views of per site data. Connectivity topologies are now arranged based on site longitude/latitude and no longer based on alphabetical order.
App Traffic Enhancements in App Namespaces
Optimizations are delivered to the app traffic graphs views under <Namespace>
-> Sites
-> App Traffic
.
General Tab Updates
Updates are introduced to the General
tab and layout for simplified UX for Billing
, Support
, IAM
, and Personal Management
.
Tenant Settings
A new section called Tenant Settings
is added. The tenant settings section provides an overview of tenant information such as tenant ID, domain and company name. System wide IAM credentials can be configured here.
Billing Enhancements
Updates are introduced to billing reports, usage details, and billing settings. These include options to request changes to existing plans and viewing existing tenant wide quotas.
Support
Updates to the escalation processes are added to team and organizational plans.
Changes to Default Behavior
The default time interval for App Firewall Dashboard is changed to 12hrs from 5 minutes.
Caveats
- Performing reboot of active master node of a multi-node site from the Console requires you to wait till the reboot is completed before attempting the reboot of other nodes.
August 13, 2020
New Features
Node/Site Management Site Deployment Wizards
In this release, we've introduced a simplified Site Deployment Wizard. Initial Cloud Providers include AWS and Azure.
Site Local UI and F5 Distributed Cloud CLI Enhancements
Introduction of Site Local UI Dashboard at https://<volterranode-ip>:65500
. Various debugging enhancements to F5 Distributed Cloud Admin CLI are added.
F5 Distributed Cloud CLI for Cloud Instances
Cloud instances for Node now support the F5 Distributed Cloud CLI for enhancement debugging. Users can access it using the ssh key used when used in the deployment of the Cloud instance.
Enhanced Site Monitoring
This feature enhanced existing site monitoring pages in the Site Dashboard. Enhancements included per node health, metrics (CPU/Memory), DHCP Server (Client Leases, Hostnames, IPs, etc.), Per Interface metrics, etc.
Multi-Node Master Node Replacement Support
Support for replacing a master node in a multi-node cluster configuration. Details can be found here.
Mesh - Virtual Hosts - Load Balancers
Default Pages Error Pages for JS Challenge, Captcha and Errors
Added default pages for all VIPs configured using an HTTP Load Balancer or advanced Virtual Host configurations.
Mesh - Delegated Domains
Delegated Domain - Enhancements
We now support native integration with LetsEncrypt for those customers who don't want to BYOC and want a secure app experience, this is available as part of the Virtual Host -> HTTP Load Balancer configuration. Provided enhancements in the Domains Verification setup and post-verification displays.
Delegated Domain - DNSSEC
We now support DNSSEC for Delegated Domains. More information here.
App Stack - vK8s
vK8s Auditability
This enables the ability to get audit logs for Create/Update operations on k8s objects (for e.g deployment, service, etc.) in vk8s.
Console
UI/UX Enhancements
Console sidebar and overall navigation has been augmented to enhance the UX and to simply NetOps, DevOps, Secops and Developer workflows.
2FA Authentication
This feature allows the ability for customers to enable 2FA Authentication for freemium tenants and tenants who use F5 Distributed Cloud Services for Authentication. This does not apply to tenants that use SSO Authentication.
Okta SSO support
This release introduces tenant SSO support for Okta.
July 23, 2020
New Features
vK8s PVC Storage on Regional Edges
In case of Regional Edge sites the F5 Distributed Cloud Services ADN now support Persistent Volume Claims (PVC) for vK8s pods.
Ability to Select a List of Sites for vK8s Objects
This feature provides the ability to select a list of sites (using the ves-io/sites: site1,site2
annotation) for vK8s objects. This is an enhancement to the current ability to select a list of virtual sites(using the ves.io/virtual-sites: vsite1,vsite2
annotation).
See vK8s Resource Management for more details.
Audit Logs for Operations on K8s Objects in vK8s
This feature enables audit logs for the Create/Update operations on K8s objects (such as deployment, service, etc.) in vK8s.
Ability to Test Alert Notifications
This feature enables user to test alert notifications to an alert receiver. Once an alert receiver is created, a verify API on the alert receiver will generate a test alert to that receiver.
API User/Client Rate Limiting
This feature introduces the support for rate limiting the number of API requests per user over a time period. Rate limiting per user is based on the user identification configured on the rate limiter object. For more information, see Configure Rate Limiting.
Support TLS Fingerprinting in Service Policy Rules
This feature introduces the support for configuring a service policy rule to match TLS fingerprint and action. Actions are deny and rate-limit. For more information, see Configure TLS Fingerprinting.
Two Factor Authentication (2FA) VoltConsle Support
This feature introduces support for enabling 2FA for all plans for customers who use F5 Distributed Cloud Services for authentication. This does not apply to tenants that use SSO for authentication.
API Tokens for F5 Distributed Cloud Services APIs
This feature introduces support for API tokens to be used with APIs. This is in addition to the already supported API certificates. For more information, see Obtain Credentials.
Delegate Domains to F5 Distributed Cloud Services
This feature introduces support for delegation of domains to F5\ Distributed Cloud Services for DNS management. When a domain is delegated to F5 Distributed Cloud Services, all subsequent HTTP load balancer names created will result in the proper DNS RR records to be created. For more information, see Delegate Domains.
HTTPS Load Balancer Automatic SSL certificate Creation for Delegated Domains
This feature introduces support to enable automatic TLS certificate minting and verifying for a HTTPS load balancer provided a DNS domain is delegated to F5®Distributed Cloud Services. For more information, see Create HTTP Load Balancer.
Support for GCP
This feature introduces support for site deployment in GCP using the Node GCP images.
CentOS Support for VMWare Images
Node CentOS support is introduced on VMware ESXi hypervisors.
June 9, 2020
New Features
Verify Domain Ownership in the Bring Your Own Certificate (BYOC)
F5 Distributed Cloud Services will confirm domain ownership by verifying the domain in the virtual-host field matches that in the TLS certificates. If there is no match, the configuration is rejected.
Enable Wizard Forms for Alert Notifications
This feature presents simplified configuration views for alert notifications.
F5 Distributed Cloud Site on MiniKube, EKS, and AKS
This feature introduces the ability to deploy a Node on MiniKube, EKS, and AKS for site creation and use in Console tenant.
vK8s: K8s Pod Delete
This feature introduces support for pod deletion in vK8s and is supported using kubectl.
Support API Token
In addition to certificates, this introduces support for API tokens for 3rd party/external API to access Console services.
Caveats & Changes to Default Behavior
Network policies to implicitly deny traffic is now the system default behavior the moment network policy is configured. Prior to R1.2, the behavior was an implicit allow. In case you have an existing network policy set with no explicit rule to allow the ingress or egress traffic, the traffic will be dropped.
On this page:
- Objective
- October 07, 2024
- August 13, 2024
- May 28, 2024
- March 26, 2024
- January 16, 2024
- December 12, 2023
- November 07, 2023
- September 12, 2023
- August 08, 2023
- July 11, 2023
- June 06, 2023
- May 09, 2023
- April 11, 2023
- March 14, 2023
- February 14, 2023
- January 17, 2023
- December 06, 2022
- November 01, 2022
- September 27, 2022
- August 30, 2022
- August 02, 2022
- July 05, 2022
- June 07, 2022
- May 10, 2022
- April 12, 2022
- February 22, 2022
- February 03, 2022
- January 13, 2022
- December 13, 2021
- August 26, 2021
- August 5, 2021
- July 15, 2021
- June 24, 2021
- June 3, 2021
- May 13, 2021
- April 22, 2021
- April 01, 2021
- March 11, 2021
- February 18, 2021
- January 21, 2021
- December 17, 2020
- November 25, 2020
- November 5, 2020
- October 14, 2020
- September 24, 2020
- August 13, 2020
- July 23, 2020
- June 9, 2020