​

Protect Web Endpoints

Important: Bot Defense Advanced Self-Service Policy Management is a beta feature.

Bot Defense uses JavaScript to collect telemetry from web applications. This telemetry is then attached in the form of HTTP headers or included in the POST body to the protected requests.

The Bot Defense service then examines the telemetry collected from requests before they are permitted to reach your application. Based on the Bot endpoint policy that you configure, Bot Defense can prevent automated traffic from reaching your application by blocking or redirecting the traffic.

Identify Web-Based Endpoints to Protect

When you configure Bot Defense, you must identify all the endpoints (URL paths?) that you want to protect. F5 recommends that you protect all entry pages. An entry page is a page that users typically reach when they first visit your website. For example:

  • A login page where a username and password are required to access protected content.
  • A home page or landing page that users reach before visiting other pages on your site.

For information about how to identify the endpoints in your application that you must protect, see Perform a Site Analysis.

Web Endpoint Best Practices

F5 recommends that you follow these best practices when you configure protection for web endpoints:

  • If your site is served from an apex domain, configure a redirect from the apex domain to the FQDN domain (for example, example.com redirects to www.example.com). This is required because your domain must point to a Distributed Cloud-provided CNAME. Pointing a CNAME from an apex domain is not recommended as that prevents you from having SOA and all other DNS records at apex.
  • Do not use IP-based session persistence to load-balance internally. F5 Distributed Cloud has a range of egress IP addresses and if you have IP-based load balancing at the origin, this may break your session persistence. Use cookie-persistence, if possible.
  • Lock down your infrastructure to Distributed Cloud egress IP addresses only. This prevents attackers from bypassing Distributed Cloud altogether and hitting your origin directly.
  • Make sure your internal tools (such as logging or fraud tools) see the traffic accurately when going through Bot Defense.

Web Scraping

Configure web scraping protection for documents and endpoints on pages where JavaScript cannot be injected, or that are accessible by GET requests without visiting the main page, such as through a link on social media or from an email or saved bookmark.

For example, when you search for an airline flight through an online booking site and click a link that initiates a query on the airline site, you often bypass the entry page for the airline site.

This direct access to resources prevents Bot Defense from running the protection JavaScript . When you configure an endpoint for web scraping, Bot Defense displays an interstitial page that is transparent to the user but that allows Bot Defense to collect telemetry data about the requests.

HTTP Methods

For each endpoint that you protect, you must select which HTTP methods to monitor. You can select multiple methods.

  • ANY: Includes GET(XHR/Fetch), POST and PUT. Does not include GET(Document). To select all methods including GET(Document), instead of ANY, you must select each method from the list .
  • GET(XHR/Fetch): Use when the protected application makes an XHTTPRequest or Fetch API call to get the content of the page. GET requests are protected only if they are sent by XHTTPRequest from a page with Bot Defense JavaScript injected, not from direct navigation using the address bar or link.
  • POST: The most commonly attacked method. F5 recommends that you protect all POST requests.
  • PUT
  • GET(Document): Use to protect endpoints on a web page that can be accessible by GET requests without visiting the main page. When you configure an endpoint using GET(Document), Bot Defense displays an interstitial page that is transparent to the user but that allows it to collect telemetry data about the requests. Note that you cannot use GET(Document) with mobile endpoints.

Note: If you select both GET(XHR/Fetch) and GET(Document), only GET(XHR/Fetch) requests are monitored. If you need to monitor both methods, add the endpoint separately for each method .

Important: If you choose to protect all pages and endpoints (/*), F5 recommends that you select specific HTTP methods to monitor and that you do not select the ANY option. Protecting all HTTP methods for all pages can adversely impact performance.

On this page: