Bot Defense

Objective

This guide provides instructions on how to configure and use Bot Defense on F5® Distributed Cloud Console. For more information on Bot Defense, see About Bot Defense.


Prerequisites

Note: If you do not have an account, see Create a Distributed Cloud Console Account.

  • An Organization plan.

Note: If you do not have an Organization plan, upgrade your plan.


Enabling Bot Defense

When both prerequisites above are met, you can enable Bot Defense in one of two ways:

Option 1 - WAAP Add-on Service

Step 1: On the Distributed Cloud Console Home page, click Billing. The Billing Plan page appears.

HomeBilling
Figure: Click Billing

Step 2: Scroll to the right until you see the Organization plan.

OrgPlan
Figure: Scroll to the right until you see Organization Plan.

Step 3: Under the Organization plan, click the Bot Defense link. The Bot Defense service landing page appears.

billingBotDefense
Figure: Click Bot Defense

BotDefenseAbout
Figure: Bot Defense Landing Page

Option 2 - Standalone Service

When you have an account and Organization plan, you can enable the Bot Defense add-on service by following the steps below.
  1. On the Distributed Cloud Console Home page, click the Bot Defense card. If Bot Defense is not enabled, the Bot Defense landing page appears.

Note: If Bot Defense is already enabled, the Monitor page appears.

  1. Click the Request Service button to enable Bot Defense.

HomeBotDefense
Figure: Click Bot Defense

BotDefenseAbout
Figure: Bot Defense Landing Page

Note: To manage protected applications, a ves-io-power-developer-role or higher is required in system and application namespaces. To access the Bot Defense Monitor page, a ves-io-monitor-role or higher is required.


Configuring Bot Defense to Protect XC Mesh Applications Through an HTTP Load Balancer (WAAP Add-on Service)

Step 1: In Distributed Cloud Console, navigate to the Load Balancers page.

VoltConsoleHomeNav
Figure: Click Load Balancers

Step 2: Click the … in the Actions column next to your load balancer and select Manage Configuration.

ManageConfigurationNav
Figure: Click '...' > Manage Configuration

Step 3: Click Edit Configuration.

EditConfigurationNav
Figure: Click Edit Configuration

Step 4: In the left-side navigation menu, click Security Configuration.

EditConfigurationSecurity
Figure: Click Security Configuration

Step 5: In the Bot Defense Config section, click the drop-down menu and select Specify Bot Defense Configuration.

SpecifyBotDefenseConfigNav
Figure: Select Specify Bot Defense Configuration

Note: For load balancers with types “HTTPS with Custom Certificate” and “HTTPS with Automatic Certificate” set the Path normalize field to Enable path normalization. This is the default setting for new load balancer objects.

pathNormalize
Figure: Path normalize

Note: Make sure the "Disable" checkbox is unchecked under Service Policy.

servicePolicy
Figure: Uncheck the Disable checkbox under Service Policy.

Adding Endpoints for Bot Defense Protection

In the Bot Defense Config section, follow these steps:

Step 1: Under Bot Defense Regional Endpoint, select the region where the endpoint resides.

RegionalEndpoint
Figure: Select Regional Endpoint

Step 1a (optional):

  • Click the Show Advanced Fields switch in the upper right corner. The Timeout field appears.
  • In the Timeout field, enter the number of milliseconds Bot Defense should wait for the inference check to complete before timing out. The default value is 1000.

timeout
Figure: Timeout field

Step 2: In the Bot Defense Policy section, click Configure.

BotDefensePolicyConfigure
Figure: Click Configure

Step 3: In the Protected App Endpoints section, click Configure.

ProtectedAppEndpointsConfigure
Figure: Click Configure

Step 4: Click the Add Item button to add an application endpoint.

Note: You can create multiple endpoints.

AddEndpoint1
Figure: Click Add Item

Step 5: Enter the following for each endpoint:

addEndpoints
Figure: Add Endpoints

  • Name: The name of the message. Must follow DNS-1035 format.
  • Description: A human-readable description of the endpoint.
  • HTTP Methods: Which HTTP methods are monitored on this endpoint.

Note: Commonly used methods include POST/PUT/GET(XHR). GET requests are protected only if they are sent by XHTTPRequest from the page that has Bot Defense JavaScript injected, not from direct navigation via address bar or link. The ANY method should be used carefully and only when intended.

  • Protocol: Which protocols are protected.

domainMatcher
Figure: Domain Matcher

  • Domain Matcher: Since HTTP load balancers can serve multiple domains, you can specify domains here. Enter an exact value, a suffix value, or a regex value.

path
Figure: Path

  • Path: Specify protected paths here. Enter a prefix, exact path, or regex value.

botTrafficMitigation
Figure: Bot Traffic Mitigation

  • Bot Traffic Mitigation: Specify what action to take when a bot is detected.
  • Block: The endpoint returns a status code and message. You can select the code and edit the message here.
  • Redirect: The endpoint forwards the browser to a URI, specified here.
  • Flag: Select No headers to create a log record only, or select Append Headers to create custom headers for Inference and Automation Type.

Injecting the Bot Defense JS into Your Web Pages

After you have added the domains in which to apply Bot Defense protection, you need to inject the Bot Defense JavaScript (JS) into the web pages.

To inject the Bot Defense JS, follow these steps:

In the JavaScript Insertion section:

Step 1: Under JavaScript Download Path, enter the path where the HTTP load balancer can find the JavaScript to serve to the client browser.

ProtectedAppEndpointsJavascriptPath
Figure: JavaScript Path

Step 2: Under JavaScript Insertion Settings, specify if the HTTP load balancer should insert JavaScript into all pages, or if some pages should be excluded.
If you select Insert JavaScript in All Pages:
  • Choose where the JavaScript will be inserted:
    • After <head> tag
    • After </title> tag
    • Before <script> tag

InsertJsInAllPagesJsLocation
Figure: JavaScript location

If you select Insert JavaScript in All Pages with the Exceptions:
  1. Choose where the JavaScript will be inserted:
* After \<head> tag
* After \</title> tab
* Before \<script> tag
  1. Click the Add Item button to add an excluded page. The JavaScript Insertion Exclusion Rule page appears.

InsertJsInAllPageswithExceptionsExcludePages
Figure: Exclude pages

  1. In the JavaScript Insertion Exclusion Rule page, enter the following for each excluded page:
* Name: The name of the message. Must follow DNS-1035 format.
* Description: A human-readable description of the endpoint.
* Domain Matcher: Since HTTP load balancers can serve multiple domains, you can specify domains here. Enter an exact value, a suffix value, or a regex value.      * Path: Specify protected paths here. Enter a prefix, exact path, or regex value.
  1. Click the Add Item button.

Note: The options below are only visible when the Show Advanced Fields switch is set to On.

If you select Custom JavaScript Insertion Rules:
  1. Under JavaScript Insertions, click Configure. The JavaScript Insertions page appears.

customJIRC
Figure: Configure
javascriptInsertionsAddItem
Figure: Add Item

  1. Click the Add Item button and enter the following for each endpoint:
* Name: The name of the message. Must follow DNS-1035 format.
* Description: A human-readable description of the endpoint.
* Domain Matcher: Since HTTP load balancers can serve multiple domains, you can specify domains here. Enter an exact value, a suffix value, or a regex value.
* Path: Specify protected paths here. Enter a prefix, exact path, or regex value.
* JavaScript location:
  * After \<head> tag
  * After \</title> tab
  * Before \<script> tag
  1. When you are finished adding endpoints, click the Back button. The JavaScript Insertions screen appears.
  2. Click the Back button.

customJIREP
Figure: Add Item

  1. Under Exclude Paths, click the Add Item button and enter the following for any paths you want to exclude from inserting JavaScript:
  • Name: The name of the message. Must follow DNS-1035 format.
  • Description: A human-readable description of the endpoint.
  • Domain Matcher: Since HTTP load balancers can serve multiple domains, you can specify domains here. Enter an exact value, a suffix value, or a regex value.
  • Path: Specify protected paths here. Enter a prefix, exact path, or regex value.
If you select Disable JavaScript Insertion, no further action is necessary. JavaScript will not be inserted to any page.

Best Practices For Configuring Bot Defense

  • When configuring endpoints, use wildcards to terminate the match.

    Example: /login*

    This will prevent an attacker from adding a slash to the end of the request (e.g. /login/) when both variants reach the same application endpoint.

  • The mitigating response should try to mimic what the application responds with when the request is invalid, for example, when invalid credentials are used.

  • If your site is served from an apex domain, configure a redirect from the apex domain to the FQDN domain (e.g customer.com redirects to www.customer.com). This is required as your domain will need to point to a Distributed Cloud-provided CNAME. Pointing a CNAME from an apex domain is not recommended as that prevents you from having SOA and all other DNS records at apex.

  • Do not use IP-based session persistence to load-balance internally. F5 Distributed Cloud has a range of egress IPs and if you have IP-based load balancing at the origin, this may break your session persistence. Use cookie-persistence, if possible.

  • Lock down your infrastructure to Distributed Cloud egress IPs only. This will prevent attackers from bypassing Distributed Cloud altogether and hitting your origin directly.

  • Validate that your internal tools (Logging tools, Fraud tools, etc.) are seeing the traffic accurately when going through Bot Defense.

  • If you have a CDN such as CloudFront implemented before F5 Distributed Cloud ( Client -> F5 Distributed Cloud -> Origin) please ensure :

  • The Bot Defense JS file is excluded from the cached entities

  • The CDN is passing the origin User-Agent rather than injecting CDN User-Agent such as "Cloud Front"

  • Identify all URL paths that need to be protected by Bot Defense.

  • Use wildcards cautiously. Avoid enabling Bot Defense protection for more URLs than required by adding wildcards.

Verifying Bot Defense

Confirming JavaScript Injection is Working

To confirm Bot Defense is adding JavaScript to an entry page, inspect the page in your browser and look for Bot Defense-specific headers using the procedure below.
  • Get the following information from your Load Balancer configuration: You can find the entry page's host and path in your HTTP Load Balancer configuration by navigating to Home > Load Balancers > Manage Configuration > Edit Configuration > Bot Defense Config > Bot Defense Policy > Protected App Endpoints.

    • The entry page path.

    Example: www.yourwebsite.com/endpoint

    • The JavaScript download path

      Example: /yourscript.js

  • Go to the entry page path in your browser.

Example: www.yourwebsite.com/endpoint

  1. Right-click (Windows) or Control-click (iOS) near the outer edge of the page and select Inspect. Your browser’s inspector opens.
  2. In the inspector, click the Network tab.
  3. Refresh the browser. A list of requests appears in the inspector.
  4. Look for three scripts in the list of requests named with your JavaScript download path followed by ?matcher, ?cache, and ?async query parameters.

Examples:
yourscript.js?matcher
yourscript.js?cache
yourscript.js?async

If you see three scripts with your JS download path followed by ?matcher, ?cache, and ?async, then Bot Defense is injecting JS into the entry page.

False Positive Analysis For Web Traffic

This section provides guidance for how to confirm that Bot Defense is working as expected on your website after it has been enabled for real users. Specifically, you will need to determine if any legitimate traffic is marked as automation.

Although this documentation provides some guidance, this is a creative process. Explore your traffic reports and examine anything unexpected. Resolve all issues before you move to blocking mode.

Identify False Positives

  1. Review human and bot traffic on the Bot Defense dashboard or the Bot Traffic Overview Dashboard.
  2. Is any traffic marked as non-human? If yes:
  • What is the automation type of the non-human traffic?
  • Does the traffic marked as malicious have a diurnal pattern which increases during the day and drops at night? This might indicate human traffic.
  • Look at the distribution of IPs and the countries they are from. Does this distribution look like it's coming in from your normal user base?
  • Look at the User Agent field. Are any suspicious user agents present? You can also identify wanted automation (Test Tools, SEO bots etc.) through this technique.

Enabling Mitigation (Blocking Mode) for Bot Defense

Typically, users configure Bot Defense in Flag mode to start, which logs but does not block any traffic. When users feel ready, they enable blocking mode (Block or Redirect) to start actually blocking malicious traffic.

This article assumes that you have completed the False Positive analysis for your traffic.

To start mitigating attack traffic:

  1. Navigate to Home > Load Balancers > Manage Configuration > Edit Configuration > Bot Defense Config > Bot Defense Policy > Protected App Endpoints > App Endpoint Type > Edit Endpoint
  2. Under Select Bot Mitigation Action, change the Action for the protected endpoints from Flag to Block or Redirect.
  3. If you selected Block, select a Status code and enter a message body. If you selected Redirect, enter a redirect URI.

Viewing Traffic Data for XC Mesh Applications Through the Load Balancer Dashboard

When Bot Defense is enabled and configured, the HTTP load balancer's security screen has two additional tabs: Bot Defense and Bot Traffic Overview. You can use these tabs to access their respective dashboards to view data about your traffic.

To access the Bot Defense and Bot Traffic Overview dashboards:

Step 1: In Distributed Cloud Console, navigate to Web App & API Protection > Apps & APIs > Security.

Note: You can also access the Bot Defense Dashboards via the HTTP Load Balancer's “Security Monitoring” page.

VoltConsoleHome2
Figure: Click Web App & API Protection

HttpLoadBalancers1
Figure: Click Security

Step 2: Click a load balancer in the Load Balancers list. The Security dashboard appears.

clickLoadBalancer
Figure: Select a load balancer

securityDashboard
Figure: Security dashboard

One card in the Security dashboard shows the Bot Defense top three automation types.

Step 3: In the Security dashboard, click the Bot Defense or Bot Traffic Overview tab.

BotTrafficOverview
Figure: Bot traffic overview

The Load Balancer Bot Defense Dashboard

The Bot Defense dashboard provides a snapshot of human and malicious bot activity in your web traffic for a specified time period.

The dashboard presents key information like which bots are making the most malicious requests, which endpoints are attacked most, and which automation types are being used most. You can customize the time period, filter results, and make other adjustments using the dashboard features described below.

BotDefenseDashboard
Figure: Bot Defense dashboard

Time Window selector: Choose a time range to analyze from this drop-down menu.

Hide Filter/Show Filter button: Click this to toggle the Add filter link and any active filters. You can filter results by IP address, AS Organization, and User Agent.

Traffic Types: Shows the total number of transactions in the selected time window, how many were malicious bots, and how many were humans.

Top Automation Types: Shows the most common automation type for the selected time window.

Traffic Overview: This graph shows the transactions per minute of human and malicious bot traffic for the selected time period. You can hover over the graph to see specific values.

Top Malicious Bots: This table shows the five bots that have made the most malicious requests in the selected time period. The table includes the source IP, ASN, user agent, country where the bot is based, and number of malicious requests in the selected time period. You can view each bot's Source IP, AS Organization, and User Agent.

Top Endpoints Attacked: This table shows the five endpoints that are being attacked most frequently by malicious bots. The table includes the host name, endpoint path, and number of malicious requests in the selected time period.

The Load Balancer Bot Traffic Overview Dashboard

The Bot Traffic Overview dashboard provides detailed insight into traffic on the HTTP load balancer.

BotTrafficOverview1
Figure: Bot traffic overview

In addition to the chart showing transactions per minute for a specified time window, you can also view details about every HTTP request sent through Bot Defense. Each HTTP request includes the following information:

  • Time
  • Country
  • IP Address
  • ASN
  • AS Organization
  • User Agent
  • Host
  • Path
  • Method
  • Inference

You can rearrange and sort the columns by clicking them. You can show/hide columns by clicking the gear icon in the upper right corner of the list.


Configuring Bot Defense to Protect Other Applications Through the Template Connector

Adding Protected Applications to Bot Defense

To add applications to Bot Defense, follow the steps below.

Step 1: On the Distributed Cloud Console Home page, click the Bot Defense card. The Applications page appears.

HomeBotDefense
Figure: Click Bot Defense

Step 2: On the Applications page, if you haven't added any applications yet, click the blue Add Protected Application button. Otherwise, click the Add Application button.

AddProtectedApplication1
Figure: Add Protected Application Button

AddApplication
Figure: Add Application Button

The Protected Application page appears.

ProtectedApplication
Figure: Protected Application Page

Step 3: On the Protected Application page, enter the following:
  • Name: Enter the application's name.
  • Labels: Select or add a label for this application.
  • Description: Describe the application.

PaMetadata
Figure: Metadata

  • Application Region: Select the region where the application's origin server resides.

PaRegion
Figure: Region

  • Connector Type: Select the type of connector the application uses.

PaConnectorType
Figure: Connector

Step 4: Click the Save and Exit button.

PaSave
Figure: Connector

Managing Protected Applications

To manage a protected application, follow the steps below.

Step 1: On the Distributed Cloud Console Home page, click the Bot Defense card.

HomeBotDefense
Figure: Click Bot Defense

Step 2: On the Applications page, in the Actions column, click the three dots next to your application. The Actions menu appears.

PaActions
Figure: Click the three dots

ApplicationsActions
Figure: Actions Menu

Step 3: From the Actions menu, select one of the following:
  • Manage Configuration: Opens the Protected Application page. Click the Edit Configuration button to make changes.
  • Copy App ID: Copy the application's unique ID to your clipboard.
  • Copy Tenant ID: Copy the tenant's unique ID to your clipboard.
  • Copy API Key: Copy the application's API key to your clipboard.
  • Delete: Delete the application.
  • Download Template: Download an iApp template for use with BIG-IP.

Note: Bot Defense supports iApp template versions 3.0.2 and above.

Viewing Traffic Data for Other Applications Through The Bot Defense Service Dashboard

When Bot Defense is enabled and configured, you can monitor traffic and activity with the Dashboard and Traffic Overview tabs on the Bot Defense page. Use these tabs to view data about your traffic.

To access the Bot Defense Dashboard and Traffic Overview screens:

Step 1: On the Distributed Cloud Console Home page, click the Bot Defense card. The Bot Defense page appears.

HomeBotDefense
Figure: Click Bot Defense

Step 2: Click Monitor.

MonitorDashboard
Figure: Click Monitor

Step 3: Click Dashboard or Traffic Overview.

MonitorTabs
Figure: Dashboard or Traffic Overview

The Bot Defense Dashboard

The Bot Defense dashboard provides a snapshot of human and malicious bot activity in your web traffic for a specified time period.

The dashboard presents key information like which bots are making the most malicious requests, which endpoints are attacked most, and which automation types are being used most. You can customize the time period, filter results, and make other adjustments using the dashboard features described below.

MonitorDashboard1
Figure: Bot Defense dashboard

Region: Select the geographical region to monitor (US, EU, or Asia).

Time Window selector: Choose a time range to analyze from this drop-down menu.

Hide Filter/Show Filter button: Click this to toggle the Add filter link and any active filters. You can filter results by IP address, AS Organization, and User Agent.

Traffic Types: Shows the total number of transactions in the selected time window, how many were malicious bots, and how many were humans.

Top Automation Types: Shows the most common automation type for the selected time window.

Traffic Overview: This graph shows the transactions per minute of human and malicious bot traffic for the selected time period. You can hover over the graph to see specific values.

Top Malicious Bots: This table shows the five bots that have made the most malicious requests in the selected time period. The table includes the source IP, ASN, user agent, country where the bot is based, and number of malicious requests in the selected time period. You can view each bot's Source IP, AS Organization, and User Agent.

Top Endpoints Attacked: This table shows the five endpoints that are being attacked most frequently by malicious bots. The table includes the host name, endpoint path, and number of malicious requests in the selected time period.

The Traffic Overview Page

The Bot Traffic Overview page provides detailed insight into traffic on the HTTP load balancer.

BotTrafficOverview2
Figure: Bot traffic overview

In addition to the chart showing transactions per minute for a specified time window, you can also view details about every HTTP request sent through Bot Defense. Each HTTP request includes the following information:

  • Time
  • Country
  • IP Address
  • ASN
  • AS Organization
  • User Agent
  • Host
  • Path
  • Method
  • Inference

You can rearrange and sort the columns by clicking them. You can show/hide columns by clicking the gear icon in the upper right corner of the list.


References