Monitor HTTP Load Balancer

Objective

This document provides instructions on how to monitor your HTTP load balancer. F5® Distributed Cloud Services provide for load balancing and proxy capabilities, letting you control the flow of application and API traffic between services, to the internet, and from clients on the internet. To know more about how load balancing and service mesh, see Load Balancing and Service Mesh.

Using the instructions provided in this document, you can check various views that present HTTP load balancer monitoring information such as statistics, events, etc.


Prerequisites

Note: If you do not have an account, see Create an Account.

  • One or more applications deployed on F5 Distributed Cloud sites or network.

Note: If you do not have applications deployed, see vK8s Deployment.

  • An HTTP load balancer with one or more security features enabled.

Note: If you do not have an HTTP load balancer, see Create HTTP Load Balancer. See App Security and Advanced Security for information on enabling various security features.


Monitor Load Balancer

The Distributed Cloud Platform offers 2 types of monitoring for load balancer—performance monitoring and security monitoring. Performance monitoring offers operational information such as metrics, events, alerts, etc. Security monitoring offers security related information such as suspicious users, security events, API discovery, etc.

Go through the steps in the following chapters to learn detailed information on both types of monitoring.

Find a Load Balancer

Step 1: Select the namespace where the load balancer is configured.
  • Select the `Load Balancers" service.
  • Select your namespace from the drop-down list of namespaces.

image2
Figure: Navigate namespace

Step 2: Navigate to the load balancer monitoring.
  • Select Virtual Hosts > HTTP Load Balancers on the configuration menu to display a list of load balancers. Each load balancer is shown in a box with some high-level information about the load balancer.

image1
Figure: Load Balancer Monitoring

  • Under the name of the load balancer you want to monitor, select either Performance Monitoring or Security Monitoring to see the respective dashboard. You can also select the name of the load balancer, which will show the Performance Monitoring dashboard.

Explore Performance Monitoring

Select Performance Monitoring for your load balancer in the load balancer monitoring page. The Dashboard tab is displayed by default.

In the various monitoring tabs, one or more common options are available to use. The following list describes commonly available options:

  • Time interval selector to apply from a list of intervals. You can also set a custom 24-hour interval.
  • Refresh option refreshes the information displayed on the page.
  • Filter option to apply filters to the displayed information.
  • Search option to search for specific information.

The following entries describe the various tab views available for performance monitoring:

Dashboard

The dashboard tab offers a snapshot view for entire performance monitoring information. The performance monitoring dashboard shows overview sections such as health, alerts, metrics, clients, devices, policy, security, etc. Some of the sections contain links to more details, for instance, selecting the section name Active Alerts will display the alert tabs where you can see much more information about these and other alerts.

lb dash
Figure: Load Balancer Generic Monitoring Dashboard

The following list provides overview on the dashboard and the various sections it offers:

  • Metrics include requests, throughputs, and latency. However, you can filter the Top Clients view to display error rate also.
  • Client information includes details such as top clients, TLS fingerprints, client location, etc.
  • Device information includes device type and browser type.
  • Security information includes details such as top ASN, TSL/SSL statistics, URLs visited, service policy, etc. Also, HTTP error code trend is presented.
Metrics

Select the Metrics tab to load the load balancer application metrics view:

The metrics present the trend of the following metrics in graph view over a default or configured time interval:

  • Health score in terms of percentage.
  • Request Rate, Error Rate, and Drop Rate.
  • Latency.
  • App Latency.
  • Client and Server Round-Trip Time (RTT).
  • Connection Duration.
  • Upstream and Downstream Throughput.

lb metrics
Figure: Load Balancer Metrics

Note: The metrics are grouped into fields such as Rate, Throughput, etc. A field may have one or more metrics.

Select a metric from the available fields on the right-hand side to display its trend. Hover your mouse pointer over a graph bar to view information specific to the time interval of that bar. You can also select the bar to switch to Requests tab.

A red bar indicates a time period where the request rate is outside the confidence window, which is shown in light blue as part of the graph background.

You can select any two metrics under a field such as Rate to display the combined graph for them. To do this, do the following:

  • Each metric has 2 small graph bar buttons to its left arranged in a vertical stack. Select a metric under one field.
  • Select the lower graph button for the other metric of the same field to display combined graph.

lb metrics combined
Figure: Combined Trend for 2 Metrics

Note: Select Last 1 hour dropdown on the upper right end of the dashboard and select a time interval to inspect your site dashboard for that interval. The default for this is 1 hour and maximum allowed interval is 24 hours. You can customize the interval by selecting the Custom option and choosing date range. This can also be set graphically by adjusting the controls beneath the main graph.

Traffic

Select the Traffic tab to view the monitoring page for traffic from requestor to origin server. The following information is displayed:

  • The view shows a graphical representation where the traffic trend is presented between requestor and origin server. The representation shows sections for the trend of traffic from requesting site to load balancer and then from load balancer to origin server.

  • Hover mouse pointer over any border bar to view details of the entity represented by that bar. For example, select the bar representing origin servers to view detailed information on the applications at those origin servers.

lb traffic
Figure: Load Balancer Traffic View

  • Hover mouse pointer over any section to view details such as source, target, and request rate.
  • Select the Request Rate filter above the graphical representation and select Response Throughput to change the details to show response throughput instead of requests.
  • Select the Group by Service filter and select an option to change the origin server details. For example, if you select Group by Site, the bar representing origin server shows the site of origin server upon hovering mouse pointer over it.
Origin Servers

Select the Origin Servers tab to view the monitoring information for origin servers. In this view, you can see the list of origin servers for your load balancer and metrics associated with the origin server.

lb originserver
Figure: Load Balancer Origin Servers

Select > for an origin server entry to view its data in JSON format.

Alerts

Select the Alerts tab to load the alerts view. The active alerts are displayed by default.

You can filter the display for alerts of a specific severity using the severity selection options. All severity types are selected by default. Select a severity selection option to hide the alerts for that severity. You can again select it to display alerts for that severity.

Note: Severity selection options are color-coded and located beneath the Add filter option.

Use the toggle selection and select All Alerts to view alerts. The All Alerts view shows graph for alerts over a specific period. The list of alerts are displayed beneath the graph.

Hover mouse pointer over a graph bar to view the alerts information specific to the time interval in which the bar is generated. Selecting the bar updates the graph and the list beneath the graph for the interval in which the bar is generated.

lb alerts
Figure: Load Balancer Alerts

Note: You can also set a time interval in the All Alerts view to display alerts over a specific period of time.

Select > for any alert entry to display its details in JSON format.

Requests

Select the Requests tab to load the view for the trend of sampled HTTP requests.

The requests are displayed in a graphical trend as well as in a list for the default or specific time interval. Select > for any listed request to display detailed information in JSON format. Use the Hide Chart option on the top right side of the page to hide the graph and display only list entries.

Note: The system performs rate adaptive sampling to guarantee that a fair number of logs are stored even when the traffic loads are high.

lb reqs
Figure: Load Balancer Requests View

You can apply filters to display the trend for specific HTTP codes. For example, de-select all and select only 2xx to display the requests for HTTP code 2XX.

You can apply filters to the display using the Forensics option at the right of the graph to show the Forensics side panel. Select a filter and select Apply to filter the display accordingly. You can also include more filter options by editing the default options and adding more from the displayed list.

Note: You can apply filters using the Add Filter option located above the requests graph.

Errors

Select the Errors tab to load the view for the trend of client or origin server errors.

The errors are displayed in a graphical trend for the default or specific time interval. You can adjust the time interval either using the drop-down selector located on the top right side of the page or using the controls beneath the graph.


Explore Security Monitoring

Switch to security monitoring view. This can be done in any of the following 2 ways:

  • Select the Performance Monitoring drop-down option in the performance monitoring view and select Security Monitoring option.

  • Navigate to Virtual Hosts > HTTP Load Balancers page. Select the Security Monitoring option for your load balancer.

The following entries describe the various tab views available for security monitoring:

Dashboard

The dashboard tab displayed by default and offers a snapshot view for entire security monitoring information. Dashboards shows various security details such as security events, WAF events, service policy events, attack events, DDoS, Bots, etc.

sec mon db
Figure: Security Monitoring View

The following list provides overview on the dashboard and the various sections it offers:

  • Security Events by Type section shows the trend of security events over time. Select any event type above the graph to display or hide that type of event. Hover over the graph to see information for that time period. Select the section title, Security Events by Type to switch to the Security Events tab for a more detailed view of these security events.

  • Security Events by Country section shows the security events arranged in a map view. Use the Security Events drop-down filter to change the section to show DDoS events. Select a location with hits to switch to the Security Events or DDoS tab accordingly showing the events from that country.

  • Top Attack Sources section shows a list of the sources with the most security events. Use the top-right drop-down menu to select top attack sources based on IP address, ASN (Autonomous System Number), or TLS Fingerprint. Select an item in the left column (IP address, ASN, or TLS Fingerprint) to those events in the Security Events tab.

  • Top Attacked section shows where the most security events have occurred. Use the drop-down menu in the upper-right corner switch between top domains or top paths being attacked. Select a domain or path from the list to view the security events for that particular target.

  • Top Attacked API Endpoints section shows a list of the domains and paths that being attacked the most, as well as the HTTP method associated with the security event. Select a domain, method, or path to see the specific events associated with your selection in the Security Events tab.

  • Bot Classification section how many security events are caused by bots classified as suspicious. Select Suspicious Bots to see the events in the Security Events page.

  • Bot Defense: Top 3 Automations (Last 24 hr) shows the top 3 Bot automation types. Bot Defense must be configured in each HTTP Load Balancer in order to see Bot automation types in the section. Select the section title to get a complete view in the Bot Defense tab.

  • Top Attacks by Signature section shows which attacks as defined by their signature are being used against your app. Attack signatures are rules or patterns that identify attack sequences or classes of attacks on a web application and its components. You can look up signatures in the F5 Attack Signature database by the signature ID or other details.

  • Top Attacks by Attack Types section shows the primary methods of attack that are being utilized. Attack types the rules or patterns that identify attacks or classes of attacks on a web application and its components, such as Cross-site Scripting or Server Side Code Injection.

  • Top Attacks by Violations section shows the top ways your security policy is being violated. Violations occur when some aspect of a request or response does not comply with the security policy.

  • Top Attacks by Threat Campaigns provide another view of the types of attacks being seen. Threat Campaigns are signatures of pervasive attacks that are often coordinated by organized crime and nation states. Threat campaign signatures are based on current “in-the-wild” attacks that exploit the latest vulnerabilities and/or new ways to exploit old vulnerabilities.

  • Malicious Users shows the list of users flagged as malicious, and shows information such as their user id, suspicion score, etc.

  • DDoS Security Events shows the events flagged as DDoS events and suspicious clients. Select an entry in the time column to see more details in the DDos tab.

  • Top Policy Policies Hit shows the service policies ranked by policy violations for this load balancer.

API Endpoints

Select the API Endpoints tab to view the discovered API endpoints and information on the various metrics associated with each API endpoint. API discovery is based on requests to specific endpoints.

There are two ways to view the API endpoints data, selectable using the Graph and Table buttons at the top left of the API Endpoints tab. There are also three colored check box filters that allow you to show or hide certain types of endpoints.

  • Inventory - Known endpoints
  • Discovered - Endpoints discovered recently (in the last few hours)
  • Shadow APIs - Third-party APIs and services

You can also select the Download Swagger option in the upper right to download the API in a machine-readable swagger file.

Note: the downloaded swagger file will only show information discovered from requests. For instance, if an endpoint supports two HTTP methods, but only GET is seen, then only the GET method will be documented.

Graph view

The graph view shows your API by segments and leaves in a railroad style diagram.

lb api endpoints graph
Figure: API Endpoints Graph View

  • Select a segment number at the top of the graph to toggle the display of all successive segments for all segments in that column.

  • Select an individual segment to toggle the display of all successive segments for that branch.

  • Select an HTTP request type or leaf to see endpoint details. The three tabs give overview information, personally identifiable information (PII) and discovered specs, and the Open API spec in JSON format.

    lb api endpoint details
    Figure: API Endpoint Details View

  • Select a non-leaf element to hide all path information beyond that point.

  • Hover over a vertical or horizontal colored bar (including black) to see summary information for that portion of the path.

  • Select the Fit button in the lower right corner of the graph to toggle the zoom factor of the graph contents between fitting in the current view or possibly requiring scrolling to see al contents.

Table view

The table view lists your API's paths (including the endpoint) in a table with summary information the columns.

lb api endpoints table
Figure: API Endpoints Table View

  • Select an entry in the Path column to see endpoint details. The three tabs give overview information, personally identifiable information (PII) and discovered specs, and the Open API spec in JSON format.

  • Select ... > Show Security Events to see any flagged requests for that endpoint in the Security Events tab.

  • Select ... > Edit Protection Rule to make changes to the protection rules for this endpoint. This selection will open the HTTP load balancer setup in the appropriate location allowing you to make changes to the endpoint protection rules.

  • Select ... > Edit Rate Limit to update the rate limiting for your app. This selection will open the HTTP load balancer setup in the security configuration section allowing you to adjust the rate limiting for your origin servers.

Malicious Users

Select the Malicious Users tab to view trend and list of events flagged as malicious user activity.

The malicious users view shows a bar chart representing the trend of malicious user activity over a default or selected time period. The view shows a graphical representation as well as a tiled list of users flagged as malicious (to the left side of the graph). Upon selection of a user entry from the left-side list, the graph on the right-side reflects that user's activity.

lb mal users
Malicious User Monitoring and Mitigation

The view also displays a timeline section beneath the graph where suspicion scores for a user over the selected time period is displayed. The scores are categorized in terms of the severity of the events.

Malicious user mitigation is supported using the Block User option located on the top of the view. You can also use the Add to Allow List option to remove the user from the malicious users list.

Security Events

Select the Security Events tab to load security events view. This shows various types of security events over the time period set above the view. This page also displays filters for various types of events that are represented in different colored check boxes corresponding to the different colored bars in the chart. Beneath the graph, the security events page displays the events in a list showing summary information.

lb security events
Figure: Security Events Page

Perform the following to inspect various security events.

  • Select the time interval drop-down list on the top right side of the page to select another time interval or specify a custom interval.

  • Select the Add Filter option and select a key-value pair to apply specific filters. You can select available key-value pairs. You can also choose a custom entry. Type a key, select Select Custom Key, type a value, and select Select Custom Value to apply a custom filter.

  • Check or uncheck a colored check box to select or deselect those events from being displayed in both the graph and the event list below.

  • Hover over a bar in the chart to get summary information for that time period.

  • Select a bar to zoom into that time period.

  • Select > at the left of a list item to display information of that event in a fully expanded view. Select the JSON tab to obtain the information in JSON format.

  • Select ... at the right of a list item to add the client (source IP) to either the Blocked Clients list or Trusted Clients list.

You can apply filters to the display using the Forensics option at the right of the graph to show the Forensics side panel. Select a filter and then select Apply to filter the display accordingly. You can also include more filter options by editing the default options and adding more from the displayed list.

DDoS

Select the DDoS tab to monitor the DDoS information for this load balancer. The DDoS view shows the information on DDoS events occurring over default or selected time interval. The view shows a geographical map showing the event location(s). Hover the mouse pointer over the location to view attack score and location information.

lb ddos map
DDoS Events Map

Select the Timeline option at the bottom of the map to display trend for request rate, error rate, and throughput. This indicates which metric is associated with the DDoS event.

Select the DDoS Events drop down located at the top of the page to display the trend of DDoS events with list of events beneath the graph. Hover the mouse pointer over a graph bar to view the start time, end time, and number of events represented by that bar.

lb ddos events
DDoS Events Graph

Select > for an entry to view detailed information in JSON format. The information includes IP addresses of users flagged as suspicious users.

Select Analytics on the top of the page to view DDoS statistics for top IP addresses, regions, ASNs, and TLS fingerprints. Select the downward arrow for any field such as Top IPs to view the member list of that field. You can select members of any field and select Apply to filter the display for the selected members.

lb ddos analytics
DDoS Analytics View

After selecting members, select Add Rule to create and apply a DDoS mitigation rule to the load balancer. This opens the load balancer configuration rule with the selected members.

lb ddos rules
DDoS Mitigation Rules

For example, you can select an IP address and select Apply to filter the display for that IP address. Then selecting Add Rule opens the load balancer edit view with the DDoS rule populated with IP address as the source and blocking that IP address as the mitigation action. select Apply and Save and Exit to apply the rule to load balancer.

Note: Select View Rules to open the load balancer DDoS rules configuration page and view the existing rules.

Alerts

Select the Alerts tab to load the alerts view. The active alerts are displayed by default.

You can filter the display for alerts of a specific severity using the severity selection options. All severity types are selected by default. Select a severity selection checkbox to show/hide the alerts for that severity.

Note: Severity selection checkboxes are color-coded and located beneath the Add filter option.

Use the toggle selection and select All Alerts to view alerts. The All Alerts view shows graph for alerts over a specific period. The list of alerts is displayed beneath the graph.

Hover the mouse pointer over a graph bar to view the alerts information specific to the time interval for which the bar is generated. Selecting the bar updates the graph and the list beneath the graph for the interval for which the bar is generated.

lb alerts
Figure: Load Balancer Alerts

Note: You can also set a time interval in the All Alerts view to display alerts over a specific period of time.

Select > for any alert entry to display its details in JSON format.

Requests

Select the Requests tab to load the view for the trend of sampled HTTP requests.

lb requests
Figure: Load Balancer Requests

The requests are displayed in a graphical bar chart as well as in a list for the default or specific time interval. Select > for any listed request to display detailed information in either a human readable format or JSON format. Use the Hide Chart option on the top right side of the page to hide the graph and display only list entries.

Note: The system performs rate adaptive sampling to guarantee that a fair number of logs are stored even when the traffic loads are high.

You can apply filters to display the trend for specific HTTP codes. For example, de-select all and select only 2xx to display the requests for HTTP code 2XX.

You can apply filters to the display using the Forensics option at the right of the graph to show the Forensics side panel. Select a filter and select Apply to filter the display accordingly. You can also include more filter options by editing the default options and adding more from the displayed list.

Note: You can apply filters using the Add Filter option located above the requests graph.

Bot Defense

The Bot Defense tab provides information on Bot activity, including traffic types, the top automation type, Bot vs. human traffic, top malicious Bots, and top endpoints attacked.

lb bot defense
Figure: Bot Defense View

  • In the Top Automation Type section, hover over the value bar to see the percentage associated with that automation type.

  • Select Area or Bar in the Traffic Overview section to change the graph from an area chart to a bar chart.

  • Hover over the graph in the Traffic Overview section to get specific Humans/Malicious Bots values for that time point.

  • The Top Malicious Bots section shows a table of the top malicious Bots sorted first by malicious requests and then by the left column. Use the drop-down menu to select which information is in the left column, which also changes the following columns. Select a link element in the table to the security events associated with that item during that time period.

  • In the Top Endpoints Attacked section, hover over a table entry ending in ... to see the full entry.


Concepts


API References