Early Access Node SiteCLI Reference

Objective

Specific Customer Edge (CE) serviceability tools, like debug log bundle and SiteCLI Commands, will be available as Early Access within the site's Tools tab (Multi-Cloud Network Connect > Overview > Infrastructure > Sites) with the September 24, 2024, release.

---

Prerequisites

• Ensure that the CE Site is running the September 24, 2024, release or higher.

• Ensure that the user is part of the following API groups:

User Role	Requirement
F5xc-site-management-standard-user	Read-only SiteCLI commands.
F5xc-site-management-monitor-user	Debug log bundle
F5xc-site-management-standard-admin	Read/write SiteCLI commands.

Debug log bundle

The Debug log bundle command includes all the diagnostic information to help F5 Technical Support investigate the root cause of an issue. Before this early access, this debug log bundle was only available to collect on a node-by-node basis. Customers had to identify the node in question and then navigate to the UI or SSH to the node to collect this information. With the Debug log bundle early access, customers can now collect a support bundle collection across all the nodes, or select nodes in a site directly from the Console without the need to identify the node IP addresses and connect to them on a node-by-node basis.

SiteCLI Commands

SiteCLI Commands allow you to debug and troubleshoot network issues, view forward state of the CE nodes, view BGP route information exchanged with external BGP endpoints, and run familiar Linux utility tools (like ip and netstat) directly from the Site Console to improve CE Site serviceability.

This functionality is available on a per-node basis as mode serviceability commands. You must log in to each node individually to run the commands. With this feature, a select set of commands are now directly available from Distributed Cloud Services Console, simplifying the CE serviceability needs.

As part of the Early Access, a select few commands are made available for you with a plan to expose a broader set of commands for General Availability (GA) in the future.

Here is a list of commands that are now available as part of the Early Access that you can run directly from the Distributed Cloud Services Console by selecting a node from a site:

Network Troubleshooting Commands

• Tunnel status commands: These commands can be executed to collect detailed status of the IPsec tunnels established between a Customer Edge and Regional Edges:

  • ipsec-status
  • ipsec-statusall

• Network connectivity commands: These IP address utility commands can be executed to check L2/L3 network state of the CE interfaces:

  • chronyc-sources: Displays the status of all NTP time sources.
  • netstat: Information about network connections.
  • ip command options include:
    • <link>: Information about network interfaces.
    • addr: Information about network interfaces.
    • route: Node route information.
    • neighbor: IP neighbor information.
    • ip-link-show: Information about all the virtual interfaces of the node forwarding state.

• CE site reachability to external F5 cloud endpoints: These commands can be used from the node (host) or from a system container, like Vega, by running the commands. This is especially useful as you check reachability to published public domains needed for healthy CE Site operation:

  • curl-host -v <host-name> e.g. gcr.io ; cloud.f5.com
  • curl-vega -v <host-name> e.g. gcr.io ; cloud.f5.com

• Network connections from the CE Site nodes can be verified by running the familiar netstat utility against any of the nodes in a site.

• Forward state commands: The CE Site node's forwarding state, such as active flows (flow-l), route (rt), next hop (nh) information, and drops statistics (dropstats) can provide a plethora of information to help troubleshoot connectivity issues in a self-service manner.

See below for details of each of the commands in sequence and how to interpret the output to help aid with the troubleshooting as well feed the next command inputs based on the previous command output.

• flow-l: Dumps the entire flow table for all flows seen by the node across all interfaces and route tables.

Output Interpretation

Above is a snipped output of the flow-l command output showing a flow with a source IP port of 10.1.20.10:3503 with destination IP port of 10.1.20.4:18093. Additionally, Proto(V) indicates that this is a TCP flow (protocol 6), and it belongs to VRF 9. Armed with VRF information you can get additional information about the next hop for a given route using rt commands.

• flow-l-match: Dumps the exact match flow entry from the CE node forwarding state.

Output Interpretation

For a given IP and port combination, as an input from the user, the matching flow information is provided as a response and the corresponding information, such as VRF, flow index, and more.

• rt: Route commands:
  • < --dump> <VRF>: Dumps the entire route table for a given VRF for IPv4 (inet) family.
  • <--dump> <VRF> <--family inet6>: Dumps the entire route table for a given VRF for IPv6 (inet6) family.
  • <--get> <route> <--family inet> <--vrf> <vrf-id>

Output Interpretation

The output above is a snipped output of rt command output showing a flow with a source IP port of 10.1.20.10:3503 with destination IP port of 10.1.20.4:18093. Additionally, Proto(V) indicates that this is a TCP flow (protocol 6), and it belongs to VRF 9. Armed with VRF information you can get additional information about the next hop for a given route.

• nh: Next Hop (NH) commands:
  • < --list>: Lists all the next hops programmed on the nodes.
  • <--get> <nh-id>

Output Interpretation

The command output above indicates that it is a "resolve" type next hop, and the outgoing interface ID is indicated where the traffic will egress.

• vif commands:
  • <--list>: List all virtual interfaces (VIF) on the node part of the SDN routing stack.

Output Interpretation

The command above lists out all the virtual interfaces on the node part of the SDN routing stack. Each of the interfaces has information around L2 address information and statistics. As seen in the sample output above, there is information about the parent interface, TX and RX statistics, and VRF this VIF belongs to.

• dropstats-non-zero: List of all non-zero counter stats key of the forwarding state.

Output Interpretation

The dropstats non-zero counter statistics showing dropped packet statistics and the counter against which the packets were dropped.

• dropstats: List of all counter statistics key of the forwarding state.

Output Interpretation

Output represents packet forwarding state packet drop statistics against each of the categories. The output is similar to dropstats-non-zero, except that it presents all counters both zero and non-zero.

• BGP Commands: CE Site nodes establish BGP sessions toward external BGP speakers for L3 reachability toward external destinations, origins, and more. The following commands can be run to check BGP state information:

• show-ip-bgp

• show-ip-bgp-neighbors

• show-ip-bgp-advertised-route

• show-ip-bgp-summary

System Troubleshooting Commands

F5 Distributed Cloud Services Customer Edge (CE) is a Kubernetes-based integrated software stack, which is managed centrally via the SaaS Console and can be instantiated in any of the customer environments – public clouds, on-premises Data Centers, Remote Branches, or even at the edge. Customer Edge software running on the CE nodes can run self-diagnosis on the overall system health, reachability to endpoints for health operation, and more. System Troubleshooting commands will allow customers to review system’s self-diagnosis information using ‘health’ and ‘diagnosis’ commands.

System Health and Self-Diagnosis Commands:

Customer Edge software running on the CE nodes can be invoked to run self-diagnosis of the system. This can be done by running health or diagnosis SiteCLI commands.

• health: No command options available.

Output Interpretation

The health command provides an overall summary of the Customer Edge site registration state with F5 SaaS platform, an inventory of discovered resource information, like CPU, memory, network, and more.

• diagnosis: No command options available.

Output Interpretation

The diagnosis command collects detailed health checks on the overall system state, health state of all the services in the system, reachability and domain resolution checks for public endpoints that are needed for healthy operation of customer edges.

System Services Health Commands:

F5 Distributed Cloud Services Customer Edge (CE) is a Kubernetes-based integrated software services stack. The following commands can be run to fetch detailed information about the container images for the container runtime, specific resource (e.g. pod, container, or image), list of all containers managed by container runtime or container logs.

These are advanced pieces of information that F5 support teams may ask customers to collect in certain scenarios.

• crictl-images: No command options available.
• docker-images: No command options available.

Output Interpretation

Lists the images available in the container runtime on the CE node.

• crictl-inspect: The option <container-id> provides information about a specific container ID.
• docker-inspect: The option <container-id> provides information about a specific container ID.

Output Interpretation

The command output provides detailed information about a specific resource (for example, a pod, container, or image) managed by the container runtime.

• crictl-logs: The option <container-id> provides information about a specific container ID for which logs need to be retrieved.
• docker-logs: The option <container-id> provides information about a specific container ID for which logs need to be retrieved.

Output Interpretation

The command output retrieves logs from a specific container managed by the container runtime.

• crictl-ps: No command options available.
• crictl-ps-a: No command options available.
• docker-ps: No command options available.
• docker-ps-a: No command options available.

Output Interpretation

The ps outputs all running containers, while the ps-a lists all containers running and stopped or exited containers.

• journalctl: The command option includes:
  • <-u> <service-name> <-n> <number-of-lines>

Output Interpretation

The journalctl command provides log information for critical node system services, like VPM, to find information about system health status.

Known Caveats

Debug log bundle collection is not supported for CE sites running as pos(s) on k8s sites.

---