​

Node Long Term Software (LTS) Releases

Published April 5, 2023 | Last modified December 8, 2025

Objective

This document lists out release notes for the F5 Distributed Cloud Customer Edge (CE) Long Term Software (LTS). The release notes include information on various New Features or Functionalities, Fixed Issues and Caveats delivered.

LTS Overview

The F5 Distributed Cloud Services Customer Edge (CE) Site Node Software is either based on a standard release crt-<version-date> or a long-term support (LTS) release lts-<version-date>. The standard releases (crt-xxxx) are regular software releases with a six-month support window offering bug fixes as well as new features better suited for customers who want to consume latest features and functionality. The LTS releases (lts-xxxx) are intended to offer a stable software that is supported for an extended duration for a total of 24 months with only critical bug fixes and security vulnerabilities delivered in maintenance releases.

Note: For changelogs on standard (crt-xxxx) releases, see Node Operating and System Changelogs

Release Cycle

The LTS releases are for supporting customers looking to run CE software version for extended duration compared to the standard six month window with regular releases.

The following details describe the typical characteristics of LTS CE Software:

  • LTS version will be in the lts-<version-date>-<build-number> format. An example is lts-20240528-0001 indicating that this is the first revision of LTS base image released on May 28, 2024.
  • New LTS versions are released every six months, and these contain new features along with relevant bug fixes.
  • A given LTS software version is supported for 24 months after it is released. Periodically maintenance versions of the LTS Software will be made available that include critical bug fixes and security vulnerabilities fixes. No new features are delivered as part of the LTS maintenance releases.

Note: LTS CE software is available for customers as an add-on SKU. Check with your F5 account team to learn more about running LTS software on your CEs.

The following is a pictorial representation of how CRT and LTS releases are rolled out within a given timeline:

Figure: LTS and CRT Versions

Figure: LTS and CRT Versions

Note: The release months shown in the above image are for representational purpose only and do not indicate the actual releases. For actual release information, see the individual release notes.

Upgrade Policy

In case a user desires features/enhancements, the user can upgrade to the next LTS version released. User can also switch between LTS and CRT versions.

The following upgrades are supported:

TypeDetailsExample
LTS to LTSFor an LTS version N, upgrades to version N+1 and N+3 (next 2 incremental releases) are supported.The LTS release of May, 2024 can be upgraded to the LTS release of November, 2024 and LTS release of November, 2025.
LTS to CRTFor an LTS version N, upgrades to CRT versions up to six months from the date of releases of LTS version N are supported.The LTS release of May, 2024 can be upgraded to any CRT release till November, 2024.

Note: It is not recommended to downgrade to a previous LTS version or switch from an LTS version to an older CRT version.

LTS Software Release Versions

lts-20250620-0001

LTS Node software is updated as part of July 22, 2025 SaaS Release. The recommended OS version for the lts-20250620-0001 is RHEL OS 9.2025.39.

Upgrading to lts-20250620-0001

When users switch from the previous LTS version to the new LTS version lts-20250620-0001, they should first perform a software upgrade to lts-20250620-0001. After that, they should proceed with the OS upgrade to 9.2025.39.

New Features

For comprehensive details on New Features, Fixed Issues, and Caveats for lts-20250620-0001, please refer to the SaaS Changelogs dated from May 28, 2024 till June 05, 2025.

Additionally, following features will be delivered for lts-20250620-0001 software release:

  • Kubernetes Version 1.32 Upgrade - Kubernetes version will be upgraded to v1.32 during this release. To upgrade to Kubernetes 1.32, the site needs to be upgraded to OS version 9.2025.39 and SW version lts-20250620-0001.

  • HPE CSI Version 2.5.2 Upgrade - The Alletra CSI driver has been upgraded to support Kubernetes v1.32. If the version of CSI in Alletra is still out of date, this may cause problems not envisaged in Kubernetes v1.32. Applied HPE CSI Driver for Kubernetes 2.5.2 to lts-20250620-0001.

Important: In the LTS version lts-20250620-0001 Fluent Bit will be replaced by Vector on the Customer Edge (CE).

Fixed Issues

The following fixes will be delivered for LTS Software Version lts-20250620-0001 in addition to the fixes provided in SaaS Release. This table organizes the issues, symptoms, conditions, and fixes described in the source document for clarity and easy reference.

DescriptionSymptomConditionFix
Fix issue with the Debug-info file, as its size increases each time it is acquired due to the log collection process in VPM.The debug-info itself is getting larger each time it is acquired.VPM keeps storing logs even after users acquire it while collecting debug-info.The fix is to add clean option in VPM that cleans up unneeded logs every time VPM finishes collecting debug-info.
Fix issue where the command execcli get-auxilary-root-access-to-node gets stuck after displaying F5 Terms and Conditions, and does not accept any keyboard inputs via SSH.The command execcli get-auxilary-root-access-to-node gets stuck after displaying F5 Terms and Conditions.
Keyboard inputs via SSH are not accepted or rejected and user cannot create xcuser (root shell access on CE).
Similar issues occur with other execcli commands like execcli ping 8.8.8.8 and execcli journalctl -fu vpm.
Ctrl+C does not stop these commands, and SiteCLI or execcli ignores inputs from stdin, except when using the pager (less) or editor (vim).		The affected OS versions are 9.2024.40, 9.2024.22 and the affected software versions are crt-20241206-3066 and all newer builds after lts-20240528-0002.
Not affected OS versions are 9.2024.40 with software version lts-20240528-0002.		Fix the keyboard inputs via SSH issue, allow user to type "agree" after the command execcli get-auxilary-root-access-to-node. Allow terminal to accept terminate signal to stop continuous command such as ping, journalctl.
Fix issue related to internal stig-hardening service running on a CE introduced an issue where the xcuser account, created by the execcli get-auxiliary-root-access-to-node command, becomes inaccessible for SSH login after the service starts running.The internal stig-hardening service was designed to modify the SSH daemon configuration (sshd_config) to enable UsePAM: yes for enhanced security. However, due to a bug, this configuration change did not take effect as intended, resulting in the inability to log in via SSH.The issue occurs because the internal stig-hardening service overrides the /etc/ssh/sshd_config, which inadvertently causes the login issue.This issue has been resolved by adjusting the internal stig-hardening service to correctly enforce the UsePAM: yes setting within /etc/ssh/sshd_config on the CE. This ensures that SSH functionality is restored while maintaining the desired security configuration. The fix is included in version 9.2025.8 or later.
Storage Class and secret settings are not reflected in the environment of crt-20241206-3066. When a Storage Class is created from the Global Controller, a Storage Class and corresponding Secret are created. This is then reflected in the Customer Edge.When a Storage Class is created from the Global Controller, the Storage Class is created and reflected at the Customer Edge, but the corresponding secret is not created. This will affect the use of PersistentVolume (PV) and PersistentVolumeClaim (PVC).Creating a Storage Class from Global Controller in crt-20241206-3066.Modified VPM image for appstacksite version, fixed entrypoint.sh script to resolve wingmanctl executable issue.
Users can process the kubectl command without receiving the “error: You must be logged in to the server (Unauthorized)” message.When using kubectl commands, users receive "error: You must be logged in to the server (Unauthorized)" errors. The kube-apiserver logs show authentication failures with errors like "x509: certificate signed by unknown authority".Access the Kubernetes API from outside the CE node.Added offline client CA certificate to kube-apiserver trusted CA bundle. Specifically, the Passport Offline Client Root CA and Passport Offline Client Issuing CA certificates must be added to /etc/kubernetes/pki/apiserver_ca.crt. This allows kube-apiserver to authenticate clients using offline certificates when the site is in offline mode.
Fix the certificate validity period which is only 240/210 days when Customer Edges (CEs) are disconnected from Global Controllers (GC) or Regional Edge (RE).Expected validity period is 820 days as per lts-20240528-0002.Certificate validity period is only 240/210 days when CEs are disconnected from GC or RE. This discrepancy could potentially cause communication issues between pods if offline leaf certificates expire during the LTS support period.Occurs in "Offline Survivability" mode when CEs are disconnected from GC or RE and affects deployment using LTS software.
When pods are restarted in offline environment, "Local Passport XXXX Issuing CA" certificates appear.
Leaf certificates issued by these local CAs have shorter validity periods (210 days) compared to online certificates (820 days).
A fix has been applied to prevent the Active Alert for VER detected link down when the connection to the RE and the state of the Node or Pod are normal.Active Alert of VER detected link down is output even though the connection to the RE and the state of the Node and Pod are normal.Using a firmware of the Intel NIC E810.Due to a problem with the firmware of the Intel NIC E810, a publicly available workaround was applied.
A process has been added to prevent old offline sites from remaining in the DC Cluster Group (DCG), and a fix has been made to avoid restarting the Phobos container.If a DC Cluster Group (DCG) is configured that contains older offline sites, a liveness probe failure is detected and the Phobos container restarts.DCG configuration with old offline sites.Added an extra cautious fix to decouple the error logs from the liveness probe task.
TCP LB CPS (Connections Per Second) performance has been improved.TCP LB CPS performance is much lower than HTTP LB. TCP LB could not bind sockets for traffic to the origins because there was a shortage of ephemeral ports on TCP LB and all TCP connections stayed in TIME-WAIT state between TCP LB and Origin.Using the TCP LB as it could not bind sockets for traffic to the origins due to a shortage of ephemeral ports.Enabled on "VegaIP_BIND_ADDRESS_NO_PORT" and net.ipv4.tcp_tw_reuse=1 on Envoy.
Ares BGP Failure Alert Description has been corrected.Although the Ares BGP Failure Alet alert is output, it is difficult to know where and how to look because the target is not listed.View Ares BGP Failure Alert.Modified the Document Template in the Description to allow the user to see information on the peer destination.
Fix the issue that occurs when a user configures multiple static routes on storage interfaces.When user configures multiple static routes on storage interfaces, only the last one is actually configured on a CE.Configure multiple static routes on storage interfaces.Fixed static route configuration to apply all the storage interface static routes in UI.
Fix the issue where the kube-multus-ds Pod memory usage was excessively high.High memory usage of kube-multus-ds Pod with CE raised to lts-20240528-0001.High memory usage of kube-multus-ds Pod on lts-20240528-0002.Memory Limit or Request was raised from 50M to 128M.
Fix the packet splitting process when going through UDP virtual hosts, so that BAD LENGTH is not generated.When going through a UDP virtual host, packets may not be split into appropriately sized packets, resulting in bad lengths and communication failures.UDP communication in which fragmentation occurs on lts-20240528-0002.In some cases, fragmentation was not properly processed. Therefore, a correction was made for this problem case.
The issue of the output file size becoming excessively large after multiple Debug Info acquisitions has been fixed.Each time debugging information is retrieved, the file size becomes bloated.Execute a command to get Debug Info on lts-20240528-0001 or lts-20240528-0002.Added the ability to delete unnecessary past information in the process of creating Debug Info.
After rebooting the OS, you can still log in with xcuser with a changed password.Run "execcli get-auxilary-root-access-to-node --password" in SiteCLI to create a user account xcuser for bash login. After that, change the password to any desired password using the "passwd" command and reboot the OS, and you will not be able to log in with xcuser.After rebooting the OS, login with the bash login user xcuser with a changed password.The login user xcuser for bash is excluded from VPM password-related processing as a special exception.
Resolved an issue that caused the Control Plane to send IPv6 DHCP dial attempts to non-IPv6 enabled sites.We use third-party open-source service Dnsmasq to provide DHCP functionality for CE devices that require IP address assignment.Due to a bug, the control plane application attempted to create a UDP IPv6 socket to the dnsmasq-v6 endpoint even when IPv6 was disabled on the site. These repeated dial attempts caused the control plane's memory usage to grow steadily over time.The issue has been fixed by preventing IPv6 socket dials when IPv6 is not enabled on the site.

Caveats

  • There is a known issue in lts-20250620-0001 where the Metric Server is not working and needs to be disabled.

  • OS versions 9.2025.30 and above introduces a limitation where downgrading to a version below 9.2025.30 is not supported. This is due to kernel compatibility issues, that can corrupt the initramfs and grub configurations after a rollback, which can lead to system boot failures. When this issue occurs, the root filesystem (/) is mounted as read-only and rpm-ostree becomes non-functional. As a result, the VM is no longer operational and cannot be recovered through normal means in a cloud environment. While recovery might be possible by manually selecting an older kernel from the GRUB menu on physical systems, this option is not available on cloud platforms like AWS. In such cases, the VM becomes unrecoverable and must be rebuilt or restored from a backup.

lts-20250620-0003

LTS Node software is updated as part of August 26, 2025 Release. The recommended OS version for the lts-20250620-0003 is RHEL OS 9.2025.39.

Fixed Issues

The following fixes will be delivered for LTS Software Version lts-20250620-0003. This table organizes the issues, symptoms, conditions, and fixes described in the source document for clarity and easy reference.

DescriptionSymptomConditionFix
The metrics-server does not spin up.The metrics-server does not spin up during software upgrade. As a result, the software upgrade operation will fail.Customer Edge (CE) sites whose "metrics server" option is enabled.The cause of problems is a side effect of enhancement in kubelet authorization settings. And new settings require addition permissions to spin up metrics-server. To resolve this, F5 added the necessary permissions, allowing the metrics server to function properly again.
A recent Node Operating System update, removed the hostname command, causing the debug script to fail.The debug info archives become flat and disorganized, losing their node-based hierarchical structure and making it difficult for operators to identify the source of the data.The Node Operating System (OS) update starting with version 9.2024.40, removed the hostname command, causing the /opt/bin/collect-debuginfo script that collects debug information to fail. The /opt/bin/collect-debuginfo script relies on the "hostname -s" command to determine the node name for creating directory structures in debug info archives. The absence of this command causes the script to fail, breaking the node-based directory structure and making it difficult for operators to identify the source of the debug data.The proposed below solution adds a fallback mechanism for collecting information:
  1. Try the original method first: _NODE_NAME=$(hostname -s)
    • On older OS versions (before 9.2024.40), this works normally.
    • On newer OS versions, this fails and returns empty.
  2. Fallback to environment variable: if [ -z "${_NODE_NAME}" ]; then _NODE_NAME="${HOSTNAME}"; fi
    • If _NODE_NAME is empty (hostname command failed), use the $HOSTNAME environment variable.
    • $HOSTNAME is a standard shell environment variable that contains the system hostname.
  3. Use the determined name:
    • Replace all instances of $(hostname -s) with ${_NODE_NAME}
This approach ensures the script works on both older OS versions (with hostname command) and newer OS versions (9.2024.40+) and preserves the critical directory structure that operators rely on for troubleshooting.
The kube-controller-manager pods repeatedly crash after the CE site upgrade.A change in Kubernetes version 1.32 kube-proxy causes repeated crashes of kube-controller-manager pods during upgrades due to excessive memory consumption, which can exceed 20GB and sometimes lead to upgrade failures.
  • CE site upgrades.
  • The Kubernetes version on LTS or CRT being upgraded to is 1.32.
The fix involves reverting a particular patch that was applied in Kubernetes 1.32.
Upgrade to OS version 9.2025.39 on CE nodes causes Multus pods to stop sending GARP packets.GARP packets are not being sent from Multus pods after upgrading Customer Edge (CE) nodes from OS version 9.2025.17 to 9.2025.39, causing network connectivity issues.

Multus pods cannot announce their presence on the network.

External switches maintain stale ARP entries.

Network connectivity fails until ARP tables are manually cleared and affects customer's upgrade and testing schedules.		The issue is caused by the kernel upgrade (kernel-5.14.0-570.23.1.el9_6.x86_64) in OS version 9.2025.39.

When L2 pods are created using Multus CNI with bridge networking, they should automatically send GARP packets with the arp_notify option enabled.

On 9.2025.17, GARP packets successfully traverse through the xc-br0 bridge and reach xc-br0-end.100 interface.

On 9.2025.39, GARP packets are generated at the pod level but get dropped before reaching the bridge interface, preventing ARP table updates on external switches.		Updated the CNI bridge plugin in the VPM (Volterra Pod Manager) image to properly handle GARP packet forwarding with the new kernel.

Node Operating and System Changelogs

Refer to the Node Operating and System Changelogs for more details on Node Software Releases dated from May 28, 2024 till June 05, 2025 (Software Version: crt-20240528-2793 to crt-20250526-3343) and Node OS Releases (9.2025.39).

lts-20240528

lts-20240528-0001

LTS software is updated as part of May 28, 2024 SaaS release. The recommended OS version for the lts-20240528-0001 version is RHEL OS 9.2024.11.

lts-20240528-0002

Fixed Issues

The following fixes were delivered. This table organizes the issues, symptoms, conditions, and fixes described in the source document for clarity and easy reference.

DescriptionSymptomConditionFix
Fix issue where site is stuck in upgrading when new CA certificate is issued and software upgrade is triggered at the same time.Site is stuck in upgrading state after triggering software upgrade to latest software version between 28th to 29th May 2024 GMT.Triggered software upgrade to latest software version between 28th to 29th May 2024 GMT.Resolve the CA certificate renewal and upgrade dependency between components during software upgrade. Only issue new CA certificate to site that upgrades to software version containing the fix.
Fix issue where debuginfo in a site cannot be downloaded 10days after the site creation.Site deleted /tmp/debuginfo mount along with files by systemd-tmpfiles-clean.service unless you settle an extra-configuration for cleaning.This issue is observed 10 days later after Site is created.Add extra-configuration to keep /tmp/debuginfo directory.
Update execcli commands to improve user experience.automic and kubectl commands do not work properly and traceroute command should be tracepath on RHEL.When using RHEL operating system, automic and kubelet commands are not supported in execcli and traceroute command is renamed to tracepath.Update execcli command tree in SiteCLI to reflect corresponding findings.
Fix issue where configure-http-proxy in VPM CLI does not work on RHEL CE.When configure-http-proxy is called on RHEL CE, it immediately returns err rather than showing edit screen.When configure-http-proxy CLI is used on RHEL CE.Fixed configure-http-proxy command to show edit screen correctly.
Fix issue where enabling xcuser misconfigures the sshd config.Creating xcuser leads to creation of user, but the user will not be able to SSH into the machine.When get-auxilary-root-access-to-node command is used on RHEL CE.Fixed get-auxilary-root-access-to-node command to edit the sshd config appropriately.
Improve the query success rate of name resolution for k8s service.Name resolution for k8s service fails irregularly on CE.The issue is observed irregularly on a CE where a pod attempts to resolve a k8s service.Improve name resolution to retry queries upon failure.

lts-20240326-0001

LTS Node software is updated as part of March 26, 2024 SaaS Release.

On this page: