Node Long Term Software (LTS) Releases
Objective
This document lists out release notes for the F5 Distributed Cloud Customer Edge (CE) Long Term Software (LTS). The release notes include information on various New Features or Functionalities, Fixed Issues and Caveats delivered.
LTS Overview
The F5 Distributed Cloud Services Customer Edge (CE) Site Node Software is either based on a standard release crt-<version-date> or a long-term support (LTS) release lts-<version-date>. The standard releases (crt-xxxx) are regular software releases with a six-month support window offering bug fixes as well as new features better suited for customers who want to consume latest features and functionality. The LTS releases (lts-xxxx) are intended to offer a stable software that is supported for an extended duration for a total of 24 months with only critical bug fixes and security vulnerabilities delivered in maintenance releases.
Note: For changelogs on standard (crt-xxxx) releases, see Node CRT Software Changelogs.
Release Cycle
The LTS releases are for supporting customers looking to run CE software version for extended duration compared to the standard six month window with regular releases.
The following details describe the typical characteristics of LTS CE Software:
- LTS version will be in the
lts-<version-date>-<build-number>format. An example islts-20240528-0001indicating that this is the first revision of LTS base image released on May 28, 2024. - New LTS versions are released every six months, and these contain new features along with relevant bug fixes.
- A given LTS software version is supported for 24 months after it is released. Periodically maintenance versions of the LTS Software will be made available that include critical bug fixes and security vulnerabilities fixes. No new features are delivered as part of the LTS maintenance releases.
Note: LTS CE software is available for customers as an add-on SKU. Check with your F5 account team to learn more about running LTS software on your CEs.
The following is a pictorial representation of how CRT and LTS releases are rolled out within a given timeline:
Figure: LTS and CRT Versions
Note: The release months shown in the above image are for representational purpose only and do not indicate the actual releases. For actual release information, see the individual release notes.
Upgrade Policy
In case a user desires features/enhancements, the user can upgrade to the next LTS version released. User can also switch between LTS and CRT versions.
The following upgrades are supported:
| Type | Details | Example |
|---|---|---|
| LTS to LTS | For an LTS version N, upgrades to version N+1 and N+3 (next 2 incremental releases) are supported. | The LTS release of May, 2024 can be upgraded to the LTS release of November, 2024 and LTS release of November, 2025. |
| LTS to CRT | For an LTS version N, upgrades to CRT versions up to six months from the date of releases of LTS version N are supported. | The LTS release of May, 2024 can be upgraded to any CRT release till November, 2024. |
Note: It is not recommended to downgrade to a previous LTS version or switch from an LTS version to an older CRT version.
LTS Software Release Versions
lts-20250620-0001
LTS Node software is updated as part of June 20, 2025 SaaS Release. The recommended OS version for the lts-20250620-0001 is RHEL OS 9.2025.17.
Upgrading to lts-20250620-0001
When users switch from previous LTS version to new LTS version lts-20250620-0001, they need to first perform an OS upgrade to 9.2025.17. Following that, they should proceed with the software upgrade to lts-20250620-0001.
New Features
For comprehensive details on New Features, Fixed Issues, and Caveats for lts-20250620-0001, please refer to the SaaS Changelogs dated from May 28, 2024 till June 05, 2025.
Additionally, following features will be delivered for lts-20250620-0001 software release:
-
Kubernetes Version 1.32 Upgrade - Kubernetes version will be upgraded to v1.32 during this release. To upgrade to Kubernetes 1.32, the site needs to be upgraded to OS version 9.2025.17 and SW version lts-20250620-0001.
-
HPE CSI Version 2.5.2 Upgrade - The Alletra CSI driver has been upgraded to support Kubernetes v1.32. If the version of CSI in Alletra is still out of date, this may cause problems not envisaged in Kubernetes v1.32. Applied HPE CSI Driver for Kubernetes 2.5.2 to lts-20250620-0001.
Important: In the LTS version lts-20250620-0001 Fluent Bit will be replaced by Vector on the Customer Edge (CE).
Fixed Issues
The following fixes will be delivered for LTS Software Version lts-20250620-0001 in addition to the fixes provided in SaaS Release. This table organizes the issues, symptoms, conditions, and fixes described in the source document for clarity and easy reference.
| Description | Symptom | Condition | Fix |
|---|---|---|---|
| Fix issue with the Debug-info file, as its size increases each time it is acquired due to the log collection process in VPM. | The debug-info itself is getting larger each time it is acquired. | VPM keeps storing logs even after users acquire it while collecting debug-info. | The fix is to add clean option in VPM that cleans up unneeded logs every time VPM finishes collecting debug-info. |
Fix issue where the command execcli get-auxilary-root-access-to-node gets stuck after displaying F5 Terms and Conditions, and does not accept any keyboard inputs via SSH. | The command execcli get-auxilary-root-access-to-node gets stuck after displaying F5 Terms and Conditions. Keyboard inputs via SSH are not accepted or rejected and user cannot create xcuser (root shell access on CE). Similar issues occur with other execcli commands like execcli ping 8.8.8.8 and execcli journalctl -fu vpm. Ctrl+C does not stop these commands, and SiteCLI or execcli ignores inputs from stdin, except when using the pager (less) or editor (vim). | The affected OS versions are 9.2024.40, 9.2024.22 and the affected software versions are crt-20241206-3066 and all newer builds after lts-20240528-0002. Not affected OS versions are 9.2024.40 with software version lts-20240528-0002. | Fix the keyboard inputs via SSH issue, allow user to type "agree" after the command execcli get-auxilary-root-access-to-node. Allow terminal to accept terminate signal to stop continuous command such as ping, journalctl. |
Fix issue related to internal stig-hardening service running on a CE introduced an issue where the xcuser account, created by the execcli get-auxiliary-root-access-to-node command, becomes inaccessible for SSH login after the service starts running. | The internal stig-hardening service was designed to modify the SSH daemon configuration (sshd_config) to enable UsePAM: yes for enhanced security. However, due to a bug, this configuration change did not take effect as intended, resulting in the inability to log in via SSH. | The issue occurs because the internal stig-hardening service overrides the /etc/ssh/sshd_config, which inadvertently causes the login issue. | This issue has been resolved by adjusting the internal stig-hardening service to correctly enforce the UsePAM: yes setting within /etc/ssh/sshd_config on the CE. This ensures that SSH functionality is restored while maintaining the desired security configuration. The fix is included in version 9.2025.8 or later. |
| Storage Class and secret settings are not reflected in the environment of crt-20241206-3066. When a Storage Class is created from the Global Controller, a Storage Class and corresponding Secret are created. This is then reflected in the Customer Edge. | When a Storage Class is created from the Global Controller, the Storage Class is created and reflected at the Customer Edge, but the corresponding secret is not created. This will affect the use of PersistentVolume (PV) and PersistentVolumeClaim (PVC). | Creating a Storage Class from Global Controller in crt-20241206-3066. | Modified VPM image for appstacksite version, fixed entrypoint.sh script to resolve wingmanctl executable issue. |
| Users can process the kubectl command without receiving the “error: You must be logged in to the server (Unauthorized)” message. | When using kubectl commands, users receive "error: You must be logged in to the server (Unauthorized)" errors. The kube-apiserver logs show authentication failures with errors like "x509: certificate signed by unknown authority". | Access the Kubernetes API from outside the CE node. | Added offline client CA certificate to kube-apiserver trusted CA bundle. Specifically, the Passport Offline Client Root CA and Passport Offline Client Issuing CA certificates must be added to /etc/kubernetes/pki/apiserver_ca.crt. This allows kube-apiserver to authenticate clients using offline certificates when the site is in offline mode. |
| Fix the certificate validity period which is only 240/210 days when Customer Edges (CEs) are disconnected from Global Controllers (GC) or Regional Edge (RE). | Expected validity period is 820 days as per lts-20240528-0002. | Certificate validity period is only 240/210 days when CEs are disconnected from GC or RE. This discrepancy could potentially cause communication issues between pods if offline leaf certificates expire during the LTS support period. | Occurs in "Offline Survivability" mode when CEs are disconnected from GC or RE and affects deployment using LTS software. When pods are restarted in offline environment, "Local Passport XXXX Issuing CA" certificates appear. Leaf certificates issued by these local CAs have shorter validity periods (210 days) compared to online certificates (820 days). |
| A fix has been applied to prevent the Active Alert for VER detected link down when the connection to the RE and the state of the Node or Pod are normal. | Active Alert of VER detected link down is output even though the connection to the RE and the state of the Node and Pod are normal. | Using a firmware of the Intel NIC E810. | Due to a problem with the firmware of the Intel NIC E810, a publicly available workaround was applied. |
| A process has been added to prevent old offline sites from remaining in the DC Cluster Group (DCG), and a fix has been made to avoid restarting the Phobos container. | If a DC Cluster Group (DCG) is configured that contains older offline sites, a liveness probe failure is detected and the Phobos container restarts. | DCG configuration with old offline sites. | Added an extra cautious fix to decouple the error logs from the liveness probe task. |
| TCP LB CPS (Connections Per Second) performance has been improved. | TCP LB CPS performance is much lower than HTTP LB. TCP LB could not bind sockets for traffic to the origins because there was a shortage of ephemeral ports on TCP LB and all TCP connections stayed in TIME-WAIT state between TCP LB and Origin. | Using the TCP LB as it could not bind sockets for traffic to the origins due to a shortage of ephemeral ports. | Enabled on "VegaIP_BIND_ADDRESS_NO_PORT" and net.ipv4.tcp_tw_reuse=1 on Envoy. |
| Ares BGP Failure Alert Description has been corrected. | Although the Ares BGP Failure Alet alert is output, it is difficult to know where and how to look because the target is not listed. | View Ares BGP Failure Alert. | Modified the Document Template in the Description to allow the user to see information on the peer destination. |
| Fix the issue that occurs when a user configures multiple static routes on storage interfaces. | When user configures multiple static routes on storage interfaces, only the last one is actually configured on a CE. | Configure multiple static routes on storage interfaces. | Fixed static route configuration to apply all the storage interface static routes in UI. |
Fix the issue where the kube-multus-ds Pod memory usage was excessively high. | High memory usage of kube-multus-ds Pod with CE raised to lts-20240528-0001. | High memory usage of kube-multus-ds Pod on lts-20240528-0002. | Memory Limit or Request was raised from 50M to 128M. |
| Fix the packet splitting process when going through UDP virtual hosts, so that BAD LENGTH is not generated. | When going through a UDP virtual host, packets may not be split into appropriately sized packets, resulting in bad lengths and communication failures. | UDP communication in which fragmentation occurs on lts-20240528-0002. | In some cases, fragmentation was not properly processed. Therefore, a correction was made for this problem case. |
| The issue of the output file size becoming excessively large after multiple Debug Info acquisitions has been fixed. | Each time debugging information is retrieved, the file size becomes bloated. | Execute a command to get Debug Info on lts-20240528-0001 or lts-20240528-0002. | Added the ability to delete unnecessary past information in the process of creating Debug Info. |
After rebooting the OS, you can still log in with xcuser with a changed password. | Run "execcli get-auxilary-root-access-to-node --password" in SiteCLI to create a user account xcuser for bash login. After that, change the password to any desired password using the "passwd" command and reboot the OS, and you will not be able to log in with xcuser. | After rebooting the OS, login with the bash login user xcuser with a changed password. | The login user xcuser for bash is excluded from VPM password-related processing as a special exception. |
Caveats
- There is a known issue in lts-20250620-0001 where the Metric Server is not working and needs to be disabled.
Node Software Releases
Refer to the Node Software Changelogs for more details on Node Software Releases dated from May 28, 2024 till June 05, 2025 (Software Version: crt-20240528-2793 to crt-20250526-3343).
Node OS Releases
Refer to the Node Operating System Changelogs for more details on Node OS Releases (9.2025.17).
lts-20240528
lts-20240528-0001
LTS software is updated as part of May 28, 2024 SaaS release. The recommended OS version for the lts-20240528-0001 version is RHEL OS 9.2024.11.
lts-20240528-0002
Fixed Issues
The following fixes were delivered. This table organizes the issues, symptoms, conditions, and fixes described in the source document for clarity and easy reference.
| Description | Symptom | Condition | Fix |
|---|---|---|---|
| Fix issue where site is stuck in upgrading when new CA certificate is issued and software upgrade is triggered at the same time. | Site is stuck in upgrading state after triggering software upgrade to latest software version between 28th to 29th May 2024 GMT. | Triggered software upgrade to latest software version between 28th to 29th May 2024 GMT. | Resolve the CA certificate renewal and upgrade dependency between components during software upgrade. Only issue new CA certificate to site that upgrades to software version containing the fix. |
| Fix issue where debuginfo in a site cannot be downloaded 10days after the site creation. | Site deleted /tmp/debuginfo mount along with files by systemd-tmpfiles-clean.service unless you settle an extra-configuration for cleaning. | This issue is observed 10 days later after Site is created. | Add extra-configuration to keep /tmp/debuginfo directory. |
| Update execcli commands to improve user experience. | automic and kubectl commands do not work properly and traceroute command should be tracepath on RHEL. | When using RHEL operating system, automic and kubelet commands are not supported in execcli and traceroute command is renamed to tracepath. | Update execcli command tree in SiteCLI to reflect corresponding findings. |
Fix issue where configure-http-proxy in VPM CLI does not work on RHEL CE. | When configure-http-proxy is called on RHEL CE, it immediately returns err rather than showing edit screen. | When configure-http-proxy CLI is used on RHEL CE. | Fixed configure-http-proxy command to show edit screen correctly. |
| Fix issue where enabling xcuser misconfigures the sshd config. | Creating xcuser leads to creation of user, but the user will not be able to SSH into the machine. | When get-auxilary-root-access-to-node command is used on RHEL CE. | Fixed get-auxilary-root-access-to-node command to edit the sshd config appropriately. |
| Improve the query success rate of name resolution for k8s service. | Name resolution for k8s service fails irregularly on CE. | The issue is observed irregularly on a CE where a pod attempts to resolve a k8s service. | Improve name resolution to retry queries upon failure. |
lts-20240326-0001
LTS Node software is updated as part of March 26, 2024 SaaS Release.