Configure Bot Defense Standard with F5 Distributed Cloud Web App and API Protection

This guide provides instructions on how to quickly configure a basic Bot Defense Standard test system using the Web App and API Protection service and how to protect a test application from automated traffic (bad bots).

Note: To configure your production system with the Web App and API Protection service, see Configure Bot Defense on an HTTP Load Balancer for additional configuration information.

Note: If you use F5 BIG-IP or host your application on a content delivery network (CDN), see Configure Bot Defense (Connectors) in the Bot Defense Standard documentation for configuration instructions.

Step 1: Plan Your Bot Defense Deployment

Complete the following tasks to plan your Bot Defense deployment.

Meet the Prerequisites

To enable and configure Bot Defense, you must have the following accounts and permissions:

• A valid Distributed Cloud Console account. For information, see Create a Distributed Cloud Console Account.
• An Organization plan. To see plan information, from the Distributed Cloud Console Home page, click Billing.
• The f5xc-bot-defense-user-role (or ves-io-power-developer-role or greater).

Choose an Application to Protect

Choose an application and endpoint that you want to protect with Bot Defense. For example, for your initial system testing, you can protect a login endpoint that users typically reach when they first visit your website.

You must have the following information:

• The application domain name.
• The login HTTP URL path.
• Which HTTP methods you want to protect. Bot defense can protect GET, POST, and PUT methods.

Next Steps

Enable Bot Defense in the Distributed Cloud Console and then choose the load balancer where you want to enable Bot Defense. For information, see Step 2: Configure Bot Defense.

Note: For information about how to create an HTTP load balancer, see Step 2 of the Web App Security & Performance Quick Start.

Step 2: Configure Bot Defense

To configure Bot Defense, complete the following tasks.

Enable Bot Defense

To use Bot Defense, you must enable it in the Distributed Cloud Console.

1. On the Distributed Cloud Console Home page, click Billing.

2. Click Manage > Billing Plan and scroll to the Organization Plan.

3. Under the Organization Plan, click Bot Defense. If Bot Defense is not already enabled, the Bot Defense landing page appears.

   Note: If Bot Defense is already enabled, the Bot Defense Monitor page appears.

4. From the Bot Defense landing page, click Request Service.

After you request Bot Defense, it can take up to 24 hours for F5 to complete the request.

Enable Bot Defense on an HTTP Load Balancer

1. From the Distributed Cloud Console dashboard, click Web App & API Protection.
2. From the navigation menu, confirm that you are in the correct namespace.
3. Click Manage > Load Balancers > HTTP Load Balancers.
4. From the Actions column, click the Actions (…) menu next to the load balancer where you want to configure Bot Defense and then click Manage Configuration.
5. Click Edit Configuration and then click Bot Protection.
6. In the Bot Protection section, from the Bot Defense drop-down menu, select Enable Bot Defense Standard.

Configure Protected App Endpoints

To protect web-based endpoints, such as a web page, in the Bot Protection section, perform the following steps:

1. From the Bot Defense Region drop-down menu, select the region where the origin server for the application is located.

2. In the Bot Defense Policy section, click Configure.

3. In the Protected App Endpoints section, click Configure and then click Add item.

4. Enter the following information for each endpoint:

   • Name: Enter the name of the endpoint. The name must follow DNS-1035 format.

   • HTTP Methods: Select the HTTP methods you want to protect on this endpoint. You can select multiple methods.

     • ANY: Includes all methods except GET(Document). To select all methods including GET(Document), you must select each method from the list.
     • GET(XHR/Fetch): Use when the protected application makes an XHTTPRequest or Fetch API call to get the content of the page. Note that GET requests are protected only if they are sent by XHTTPRequest from a page with Bot Defense JavaScript injected, not from direct navigation using the address bar or link.
     • POST
     • PUT
     • GET(Document): Use to protect pages on a web site that can be accessible by GET requests without visiting the main page. When you configure an endpoint using GET(Document), Bot Defense displays an interstitial page that is transparent to the user but that allows it to collect telemetry data about the requests. Note that you cannot use GET(Document) with mobile endpoints.

   • Endpoint Label: Select an endpoint label to allow more granular attack intent identification and reporting when Bot Defense detects automation.

     For a full list of available endpoint labels, see Endpoint Labels in the Bot Defense Standard documentation.

   • Domain Matcher: Specify the domains you want to protect. Enter an exact value, a suffix value, or a regex value. Note that Bot Defense only supports JS regex and does not support inline modifiers.

   • Path: Enter the path to the endpoint you want to protect. For example, enter /login. Enter a prefix, exact path, or regex value. Note that Bot Defense only supports JS regex and does not support inline modifiers.

   • Bot Traffic Mitigation: Select Continue. This setting puts Bot Defense in monitoring mode and allows requests to continue to the origin. You can select Block or Redirect mitigation actions after you confirm that Bot Defense is correctly identifying automated traffic.

   • Good Bot Detection Settings: Specify whether the mitigation actions you selected above apply to both malicious automation and good bots. You can choose to flag good bots but allow them to continue to origin, or you can apply the mitigation actions to all automation. By default, mitigation actions are applied to all automation.

   Important: Good bots, also called benign bots, are bots that perform specific, helpful tasks, such as search engine bots, social media crawlers and aggregator bots. Many of these bots can be useful to your business and therefore you might decide to allow them to continue to origin.

5. Click Apply.

6. Click Add Item to add another endpoint or click Apply when you finish adding endpoints.

   

   Figure: Configure Protected App Endpoints

Next Steps

Configure how Bot Defense injects JavaScript into your application pages.

Configure JavaScript Injection

To protect web-based endpoints, Bot Defense injects JavaScript tags. The Bot Defense JavaScript collects telemetry data about requests to your endpoint. You must configure how the Bot Defense JavaScript tags are injected into the pages that you want to protect.

In the JavaScript Insertion section:

1. In the JavaScript Download Path field, enter a path to display to the browser and in the page source code in place of the actual download path for the Bot Defense JavaScript. This prevents malicious actors from determining what system you are using to protect your application.

   F5 recommends that you choose a URL or path that is similar to your existing JavaScript files, but that does not include “F5,” “Bot Defense” or other indications that it is used for security purposes. Enter a simple path that starts with / or a complete URL such as https://example.com/customer1.js. The URL or path you choose cannot conflict with your existing JavaScript files.

2. From the JavaScript Insertion Settings drop-down menu, select Insert JavaScript in All Pages.

3. From the JavaScript Location drop-down list, select one of the following locations to insert the JavaScript in your application pages:

   • After <head> tag
   • After </title> tag
   • Before <script> tag

   F5 recommends that you select After <head> so that the Bot Defense JavaScript is executed early. This allows time for the Bot Defense JavaScript to be fetched and executed while the rest of page is rendered. If you select After </title>, be sure your application pages have the <title> tag.

   

   Figure: Configure JavaScript Injection

4. Click Apply and then click Save HTTP Load Balancer to save your configuration.

Next Steps

When you finish configuring Bot Defense, test your deployment to make sure it is working as intended. See Step 3: Test Your Deployment.

Step 3: Test Your Deployment

Perform the following tests to help confirm that Bot Defense is identifying web traffic correctly.

Check JavaScript Injection

Open the application page where you injected the F5 Client JavaScript tags. Use the developer tools in your browser or view the page source to inspect the page and confirm that Bot Defense has inserted JavaScript tags with the following query string parameters:

• ?matcher
• ?cache
• ?async

For additional information about confirming proper JavaScript tag injection, see Test Your Bot Defense Configuration in the Bot Defense Standard documentation.

Check for False Positives

Review the Bot Defense dashboard to see the types of traffic identified by Bot Defense and confirm that no legitimate traffic is being incorrectly identified as automation (false positives). From the Bot Defense Home page, click Overview > Monitor.

• In the Protected Apps widget, check that your new application is listed and that the amount of traffic is appropriate.

• In the Traffic by Type widget, check that the level of human, benign bot, bad bot and other activity is appropriate.

  

  Figure: Bot Defense Dashboard

• From the time-period drop-down list, select Last 24 hours. In the Traffic Visualized section, check whether traffic marked as malicious increases during the day and decreases at night. If so, this might indicate human traffic.

  From the Bot Defense Home page, click Report > Traffic Analyzer.

• Look at the distribution of IP addresses and the countries of origin. Decide if this distribution looks like it comes from your normal user base.

• Look at the User Agent column and decide if there are suspicious user agents present. You can also use this technique to identify wanted automation (benign bots), such as test tools or SEO bots.

• Click Add Filter. From the drop-down list, select Bot Reason, select In, select Token Missing and click Apply. Review the traffic and determine if it looks legitimate.

For additional information about how to check for false positive results, see Analyze Bot Defense Results in the Bot Defense Standard documentation.

Next Steps

When you are ready to block or redirect automated traffic, see Step 4: Configure Mitigation Actions.

Step 4: Configure Mitigation Actions

When you are sure that Bot Defense is correctly identifying automated traffic, you can configure mitigation actions so that Bot Defense blocks or redirects automated traffic.

1. In BIG-IP, click iApps > Application Services > Applications.
2. From the list of applications, click the application you want to configure.
3. Select the Reconfigure tab.
4. In the section, F5 XC Bot Defense Protected Endpoints Configuration, from the Action drop-down list, select one of the following options:
   • Redirect. The Bot is sent to a different page. You provide the URL where the Bot is redirected in the Define Mitigation Actions section.
   • Block. The Bot is presented with a message that it has been blocked. You provide the Response Code and Body of the response message in the Define Mitigation Actions section.
5. When you finish making configuration changes, double-check your configuration settings and click Finished.

Next Steps

Now that you have protected one endpoint, checked to make sure the system is functioning properly and configured a mitigation for the endpoint, try protecting additional endpoints.

For additional information about advanced configuration steps, see Configure Bot Defense on an HTTP Load Balancer in the Bot Defense Standard documentation.