​

Protect Web Endpoints

Published April 5, 2023 | Last modified January 3, 2025

Important: Bot Defense Advanced Self-Service Policy Management is an Early Access feature.

Bot Defense uses JavaScript to collect telemetry from web applications. This telemetry is then attached in the form of HTTP headers or included in the POST body to the protected requests.

The Bot Defense service then examines the telemetry collected from requests before they are permitted to reach your application. Based on the Bot endpoint policy that you configure, Bot Defense can apply different mitigation actions that prevent automated traffic from reaching your application by blocking or redirecting the traffic.

Identify Web-Based Endpoints to Protect

When you configure Bot Defense, you must identify all the endpoints that you want to protect. F5 recommends that you protect all entry pages. An entry page is a page that users typically reach when they first visit your website. For example:

  • A login page where a username and password are required to access protected content.
  • A home page or landing page that users reach before visiting other pages on your site.

For information about how to identify the endpoints in your application that you must protect, see Perform a Site Analysis.

Web Endpoint Best Practices

F5 recommends that you follow these best practices when you configure protection for web endpoints:

  • If your site is served from an apex domain, configure a redirect from the apex domain to the FQDN domain (for example, example.com redirects to www.example.com). This is required because your domain must point to a Distributed Cloud-provided CNAME. Pointing a CNAME from an apex domain is not recommended as that prevents you from having SOA and all other DNS records at apex.
  • Do not use IP-based session persistence to load-balance internally. F5 Distributed Cloud has a range of egress IP addresses and if you have IP-based load balancing at the origin, this may break your session persistence. Use cookie-persistence, if possible.
  • Lock down your infrastructure to Distributed Cloud egress IP addresses only. This prevents attackers from bypassing Distributed Cloud altogether and hitting your origin directly.
  • Make sure your internal tools (such as logging or fraud tools) see the traffic accurately when going through Bot Defense.

Web Scraping

Configure web scraping protection for documents and endpoints on pages where JavaScript cannot be injected, or that are accessible by GET requests without visiting the main page, such as through a link on social media or from an email or saved bookmark.

For example, when you search for an airline flight through an online booking site and click a link that initiates a query on the airline site, you often bypass the entry page for the airline site.

This direct access to resources prevents Bot Defense from running the protection JavaScript . When you configure an endpoint for web scraping, Bot Defense displays an interstitial page that is transparent to the user but that allows Bot Defense to collect telemetry data about the requests.

HTTP Methods

For each endpoint that you protect, you must select which HTTP methods you want Bot Defense to protect. You can select multiple methods.

  • GET(XHR/Fetch): Use when the protected application makes an XHTTPRequest or Fetch API call to get the content of the page. GET requests are protected only if they are sent by XHTTPRequest from a page with Bot Defense JavaScript injected, not from direct navigation using the address bar or link.
  • POST: The most commonly attacked method. F5 recommends that you protect all POST requests. Bot Defense analyzes POST requests to determine if they are part of a bot attack such as credential stuffing or form submission abuse.
  • PUT: Use to protect PUT requests and prevent data uploads or modifications from bots.
  • PATCH: Use to protect PATCH requests and prevent bots from altering resource data.
  • DELETE: Use to protect DELETE requests and prevent bots from deleting content or resources.
  • GET(Document): Use to protect endpoints on a web page that can be accessible by GET requests without visiting the main page. When you configure an endpoint using GET(Document), Bot Defense displays an interstitial page that is transparent to the user but that allows it to collect telemetry data about the requests. Note that you cannot use GET(Document) with mobile endpoints.
  • HEAD: Use to protect HEAD requests and to prevent bots from repeatedly pinging a server to check metadata.
  • OPTIONS: Use to protect OPTION requests and to prevent bots from checking to find out what CORS options are supported by a server.
  • TRACE: Use to protect TRACE requests to and prevent bots from repeatedly pinging a server with diagnostic requests.

Note: If you select both GET(XHR/Fetch) and GET(Document), only GET(XHR/Fetch) requests are protected. To protect both methods for the same endpoint, add the endpoint separately for each method.

Mitigation Actions

For each endpoint that you configure, you can specify which action you want Bot Defense to take when a bot is detected. F5 recommends that you configure mitigation actions to mimic the normal behavior of the application so that an attacker does not know that the request was intercepted.

  • Continue: Allows requests to continue to the origin. A log record is created. You can optionally add headers to the request.
  • Block: The endpoint returns a status code and message that you configure.
  • Redirect: The endpoint forwards the browser to the URL that you specify. You can only select Redirect for web endpoints.
  • Transform: Allows the request to continue to the origin. Headers are added to the request for inference and automation type. Use transform when you want to allow the request to continue and use a system other than Bot Defense to mitigate.
On this page: