Configure Bot Defense Standard with F5 Distributed Cloud Web App and API Protection

This guide provides instructions on how to quickly configure a basic Bot Defense Standard test system using F5 Distributed Cloud Web App & API Protection and how to protect a test application from automated traffic (bad bots).

Note: To configure your production system with Web App & API Protection, see Configure Bot Defense on an HTTP Load Balancer for additional configuration information.

Note: If you use F5 BIG-IP or host your application on a content delivery network (CDN), see Configure Bot Defense (Connectors) in the Bot Defense Standard documentation for configuration instructions.

Step 1: Plan Your Bot Defense Deployment

Complete the following tasks to plan your Bot Defense Standard deployment.

Meet the Prerequisites

To enable and configure Bot Defense, you must have the following accounts and permissions:

• A valid Distributed Cloud Console account. For information, see Getting Started with Distributed Cloud Console.
• An Organization plan. To see plan information, from the Distributed Cloud Console Home page, select Billing.
• The f5xc-bot-defense-user-role (or ves-io-power-developer-role or greater).

Choose an Application to Protect

Choose an application and endpoint that you want to protect with Bot Defense. For example, for your initial system testing, you can protect a login endpoint that users typically reach when they first visit your website.

You must have the following information:

• The application domain name.
• The login HTTP URL path.
• Which HTTP methods you want to protect. Bot defense can protect GET, POST, and PUT methods.

Next Steps

Enable Bot Defense in the Distributed Cloud Console and then choose the load balancer where you want to enable Bot Defense. For information, see Step 2: Configure Bot Defense.

Note: For information about how to create an HTTP load balancer, see Step 2 of the Web App Security & Performance Quick Start.

Step 2: Configure Bot Defense

To configure Bot Defense, complete the following tasks.

Enable Bot Defense

To use Bot Defense, you must enable it in the Distributed Cloud Console.

1. On the Distributed Cloud Console Home page, select Billing.

2. Select Manage > Billing Plan and scroll to the Organization Plan.

3. Under the Organization Plan, select Bot Defense. If Bot Defense is not already enabled, the Bot Defense landing page appears.

   Note: If Bot Defense is already enabled, the Bot Defense Monitor page appears.

4. From the Bot Defense landing page, select Request Service.

After you request Bot Defense, it can take up to 24 hours for F5 to complete the request.

Enable Bot Defense on an HTTP Load Balancer

1. From the Distributed Cloud Console dashboard, select Web App & API Protection.
2. From the navigation menu, confirm that you are in the correct namespace.
3. Select Manage > Load Balancers > HTTP Load Balancers.
4. From the Actions column, select the Actions (…) menu next to the load balancer where you want to configure Bot Defense and then select Manage Configuration.
5. Select Edit Configuration and then select Bot Protection.
6. In the Bot Protection section, from the Bot Defense drop-down menu, select Enable Bot Defense Standard.

Configure Protected App Endpoints

To protect web-based endpoints, such as a web page, in the Bot Protection section, perform the following steps:

1. From the Bot Defense Region drop-down menu, select the region where the origin server for the application is located.

2. In the Bot Defense Policy section, select Configure.

3. In the Protected App Endpoints section, select Configure and then select Add item.

4. Enter the following information for each endpoint:

   • Name: Enter the name of the endpoint. The name must follow DNS-1035 format.

   • HTTP Methods: Select the HTTP methods you want to protect on this endpoint. You can select multiple methods.

     • ANY: Includes all methods except GET(Document). To select all methods including GET(Document), you must select each method from the list.
     • GET(XHR/Fetch): Use when the protected application makes an XHTTPRequest or Fetch API call to get the content of the page. Note that GET requests are protected only if they are sent by XHTTPRequest from a page with Bot Defense JavaScript injected, not from direct navigation using the address bar or link.
     • POST
     • PUT
     • GET(Document): Use to protect pages on a web site that can be accessible by GET requests without visiting the main page. When you configure an endpoint using GET(Document), Bot Defense displays an interstitial page that is transparent to the user but that allows it to collect telemetry data about the requests. Note that you cannot use GET(Document) with mobile endpoints.

   • Endpoint Label: Select an endpoint label to allow more granular attack intent identification and reporting when Bot Defense detects automation.

     For a full list of available endpoint labels, see Endpoint Labels in the Bot Defense Standard documentation.

   • Domain Matcher: Specify the domains you want to protect. Enter an exact value, a suffix value, or a regex value. Note that Bot Defense only supports JS regex and does not support inline modifiers.

   • Path: Enter the path to the endpoint you want to protect. For example, enter /login. Enter a prefix, exact path, or regex value. Note that Bot Defense only supports JS regex and does not support inline modifiers.

   • Bot Traffic Mitigation: Select Continue. This setting puts Bot Defense in monitoring mode and allows requests to continue to the origin. You can select Block or Redirect mitigation actions after you confirm that Bot Defense is correctly identifying automated traffic.

   • Good Bot Detection Settings: Specify whether the mitigation actions you selected above apply to both malicious automation and good bots. You can choose to flag good bots but allow them to continue to origin, or you can apply the mitigation actions to all automation. By default, mitigation actions are applied to all automation.

   Important: Good bots, also called benign bots, are bots that perform specific, helpful tasks, such as search engine bots, social media crawlers and aggregator bots. Many of these bots can be useful to your business and therefore you might decide to allow them to continue to origin.

5. Select Apply.

6. Select Add Item to add another endpoint or select Apply when you finish adding endpoints.

   

   Figure: Configure Protected App Endpoints

Next Steps

Configure how Bot Defense injects JavaScript into your application pages.

Configure JavaScript Injection

To protect web-based endpoints, Bot Defense injects JavaScript tags. The Bot Defense JavaScript collects telemetry data about requests to your endpoint. You must configure how the Bot Defense JavaScript tags are injected into the pages that you want to protect.

In the JavaScript Insertion section:

1. In the JavaScript Download Path field, enter a path to display to the browser and in the page source code in place of the actual download path for the Bot Defense JavaScript. This prevents malicious actors from determining what system you are using to protect your application.

   F5 recommends that you choose a URL or path that is similar to your existing JavaScript files, but that does not include “F5,” “Bot Defense” or other indications that it is used for security purposes. Enter a simple path that starts with / or a complete URL such as https://example.com/customer1.js. The URL or path you choose cannot conflict with your existing JavaScript files.

2. From the JavaScript Insertion Settings drop-down menu, select Insert JavaScript in All Pages.

3. From the JavaScript Location drop-down list, select one of the following locations to insert the JavaScript in your application pages:

   • After <head> tag
   • After </title> tag
   • Before <script> tag

   F5 recommends that you select After <head> so that the Bot Defense JavaScript is executed early. This allows time for the Bot Defense JavaScript to be fetched and executed while the rest of page is rendered. If you select After </title>, be sure your application pages have the <title> tag.

   

   Figure: Configure JavaScript Injection

4. Select Apply and then select Save HTTP Load Balancer to save your configuration.

Next Steps

When you finish configuring Bot Defense, test your deployment to make sure it is working as intended. See Step 3: Test Your Deployment.

Step 3: Test Your Deployment

Perform the following tests to help confirm that Bot Defense is identifying web traffic correctly.

Check JavaScript Injection

Open the application page where you injected the F5 Client JavaScript tags. Use the developer tools in your browser or view the page source to inspect the page and confirm that Bot Defense has inserted JavaScript tags with the following query string parameters:

• ?matcher
• ?cache
• ?async

For additional information about confirming proper JavaScript tag injection, see Test Your Bot Defense Configuration in the Bot Defense Standard documentation.

Check for False Positives

Review the Bot Defense dashboard to see the types of traffic identified by Bot Defense and confirm that no legitimate traffic is being incorrectly identified as automation (false positives). From the Bot Defense Home page, select Overview > Monitor.

• In the Protected Apps widget, check that your new application is listed and that the amount of traffic is appropriate.

• In the Traffic by Type widget, check that the level of human, benign bot, bad bot and other activity is appropriate.

  

  Figure: Bot Defense Dashboard

• From the time-period drop-down list, select Last 24 hours. In the Traffic Visualized section, check whether traffic marked as malicious increases during the day and decreases at night. If so, this might indicate human traffic.

  From the Bot Defense Home page, select Report > Traffic Analyzer.

• Look at the distribution of IP addresses and the countries of origin. Decide if this distribution looks like it comes from your normal user base.

• Look at the User Agent column and decide if there are suspicious user agents present. You can also use this technique to identify wanted automation (benign bots), such as test tools or SEO bots.

• Select Add Filter. From the drop-down list, select Bot Reason, select In, select Token Missing and select Apply. Review the traffic and determine if it looks legitimate.

For additional information about how to check for false positive results, see Analyze Bot Defense Results in the Bot Defense Standard documentation.

Next Steps

When you are ready to block or redirect automated traffic, see Step 4: Configure Mitigation Actions.

Step 4: Configure Mitigation Actions

When you are sure that Bot Defense is correctly identifying automated traffic, you can configure mitigation actions so that Bot Defense blocks or redirects automated traffic.

1. In Web App & API Protection, select Manage > Load Balancers > HTTP Load Balancers.
2. From the Actions column, select the Actions (…) menu next to the load balancer where Bot Defense is enabled and then select Manage Configuration.
3. Select Edit Configuration and then select Bot Protection.
4. In the Bot Defense Policy section, select Edit Configuration.
5. To edit your endpoint, in the Protected App Endpoints section, select Edit Configuration.
6. From the Actions column, select the Actions (…) menu next to the endpoint where you want to enable a mitigation action and select Edit.
7. From the Bot Traffic Mitigation section, select one of the following options:
   • Redirect. Enter the URL for the destination where you want to send the redirected bot.
   • Block. Enter the Response Code and Body of the response message that Bot Defense displays to Bots that are blocked.
8. Select Apply and then select Apply again.
9. Select Apply and then select Save HTTP Load Balancer.

Next Steps

Now that you have protected one endpoint, checked to make sure the system is functioning properly and configured a mitigation for the endpoint, try protecting additional endpoints.

For additional information about advanced configuration steps, see Configure Bot Defense on an HTTP Load Balancer in the Bot Defense Standard documentation.