API Endpoint - Discovery & Control

Objective

This document provides instructions on how to enable markup and analysis of API endpoints for your application. F5® Distributed Cloud Services discover API endpoints for your application services and performs behavioural analysis on the various logs collected from the endpoints using advanced machine learning. Thus, the Probability Distribution Function (PDF) for metrics related to each endpoint is generated. The analysis is performed periodically, and the PDFs are updated accordingly. To know more about behaviour analysis, see Behavioural Firewall.

Using the instructions provided in this document, you can enable the API endpoint markup for your application, inspect the discovered API endpoints, and monitor the related PDFs in Distributed Cloud service mesh.


API EndPoint Learning

The API EP is a tuple of URL and method for the API. There could be absolute paths or there could be variables in the path such as namespace/$namespace where $namespace indicates the namespace name. These types of URLs are called collapsible URLs, and Distributed Cloud learns such kind of URLs and presents them in the collapsed format.

Enabling the API Endpoint (EP) markup and analysis results in the following benefits:

  • Dynamic discovery of all the API endpoints of your application.
  • Monitoring the performance and trends for your APIs.
  • Determining which APIs are supposed to be between a set of services and enhancing security to allow only those APIs.
  • Obtaining insights such as which API is most hit and the associated request size.

The PDFs are obtained for the following metrics:

  • Request size and response size
  • Latency with data and without data
  • Request rate
  • Error rate
  • Response throughput

API Schema Learning

Distributed Cloud also learns schema structure of API by analyzing sampled request data examples for each API. Reverse-engineering the schema is supported for JSON and x-url-form-encoded formats. The following are learnt for each field:

  • Regex that represents this field with high probability.
  • Personal Identification Information (PII) data for the field if applicable and shows PII examples.

The system also provides option to download the swagger specification for the learnt API schemas at the following levels:

  • HTTP Load Balancer
  • AppType (in the service mesh page)
  • Per API

Prerequisites

Note: If you do not have an account, see Create an Account.

  • One or more applications deployed on Distributed Cloud site and services configured.

Note: See App Management for more information. See Site Management for site creation instructions.


Types of APIs

Inventory APIs: The API Endpoints defined as per the user's API Definition are known as API Inventory.

Discovered APIs: The API Endpoints are discovered by Console using AI/ML and are based on traffic to them. These may contain the Inventory APIs and more.

Shadow APIs: The APIs that are discovered but not in inventory.


Configuration

The following image describes the configuration work flow for enabling API EP markup for your application:

seq apiep
Figure: Work-flow for Enabling API EP Markup

Configuration Sequence

The following table presents the sequence of activities in enabling the API discovery:

Activity Description
Create App Type Create app type and configure the API EP markup functionality.
Assign App Type Label to Load Balancers Set the app type label to the load balancers and services for which API EP markup needs to be performed.
Monitor API EPs and PDFs Monitor the service mesh or load balancer for API endpoints and the associated PDFs.

Create App Type

To enable API EP markup for your application services, it is required to first enable the associated machine learning model for those services using the app type object.

The app type object is created in the shared namespace. The load balancers of that app type in different namespaces need to be assigned with the label of the app type object.

Perform the following to create app type and enable generating the anomaly model.

Step 1: Navigate to App Types configuration page.
  • Select the Shared Configuration service.

  • Navigate to Security > AI & ML > App Types.

nav atype 2
Figure: Navigate to App Type Configuration

Step 2: Configure app type object settings.

Select + Add App Type > enter the configuration in the app type object creation form using the following guidelines:

apptype cnf 2
Figure: App Type Feature Configuration

  • Enter a name for the app type. This is the value for the app type label to be assigned to the load balancers for which the API discovery needs to be enabled.

  • Select + Add item in Application Type Features section.

  • Select API Discovery in AI/ML Feature Type drop-down menu.

  • Select Add item again, and select Per API Request Analysis for the AI/ML Feature Type field.

  • Optionally, select Enable learning from redirect traffic in the Business Logic Markup Setting section. This enables the AI engine to learn the endpoints from redirected traffic.

  • Select Save and Exit button to complete creating the app type object.


Assign App Type Label to Load Balancers

After creating the app type, it is required to assign the app type label to the load balancers for which you want to enable API EP markup.

Note: Enabling API EP markup for all load balancers in a namespace requires you to apply the app type label to all load balancers in that namespace.

Perform the following to assign the app type label to your load balancers.

Step 1: Navigate to load balancer management.
  • Select the Load Balancers service.

  • Select the desired namespace from the Namespace drop-down menu.

  • Navigate to Manage > Load Balancers > HTTP Load Balancers.

  • Select ... > Manage Configuration for the load balancer for which the app type label needs to be assigned.

  • Select Edit Configuration.

lb edit 2
Figure: Navigate to load balancer Edit Configuration

Step 2: Assign the app type label.
  • Select ves.io/app_type for the Labels field and type.

  • Type the name of the app type object created in the previous step, and select Assign Custom Value to add the app type label.

label value 2
Figure: App Type Label Addition

  • Select Save and Exit to apply the label to the load balancer. This enables the learning and API EP markup for the load balancer.

Adding app type label to more than one load balancer groups the data of all such load balancers into a single learning model and presents the API EP markup analysis in the service mesh.


Monitor API EPs and PDFs

After enabling the load balancers for API EP markup, you can monitor and inspect the API EP markup and PDFs in the following two ways:

  • From the load balancers option in your namespace - This displays all endpoints for that load balancer.
  • From the service mesh option in your namespace - This displays all endpoints of all load balancers with the app type label.

Note: Learning of the API EPs and associated PDFs get updated every 4 hours and is incremental in nature. The APIs that do not get traffic for a time interval required for the model, those APIs are aged out and will not be displayed in the markup.

Service Mesh Monitoring

Note: The three types of APIs are distinguished only in Load Balancer API EP view.

Perform the following to inspect the API EPs and PDFs:

Step 1: Navigate to service mesh.
  • Select Load Balancers service.

  • Select the desired namespace from the Namespace drop-down menu.

  • Navigate to Service Mesh > Mesh.

  • Select ... > Edit for the load balancer for which the app type label needs to be assigned.

  • Select on your application tile from the displayed list to load its service mesh monitoring.

NAV SM2 2
Figure: Navigate to Service Mesh

Step 2: Load the endpoints view.
  • The service mesh loads service graph by default. Select Endpoints tab to load the API EP markup view. The endpoint paths are shown in a hierarchical structure with root and leaf relation ships presented in segments.

SM APIEP2 2
Figure: Service Mesh Endpoints View

  • Select a service edge from the All Endpoints drop-down to display the API endpoints specific for that service interaction.

srv edge apiep
Figure: API EP Markup for Specific Service Interaction

Note: You can also load API EP markup for a specific service interaction from the Service Graph view. Select an edge to load the quick view for that edge and select on Endpoints in the quick view to load the endpoints view for that specific service interaction. By inspecting the edge level API EP markup, you can determine which APIs are supposed to be functional between those nodes(services) and apply further security using service policies to restrict the traffic to those APIs only.

  • Select the Search drop down and then select a specific API to display the hierarchy for that path.

specific apiep
Figure: API EP Markup for Specific API

The following sample shows collapsed URL presented as a dynamic component:

collapsed url
Figure: Dynamic Component in API EP Markup

Note: The API EP markup also displays static resources. These are regular static resources such as javascript files that a web application uses.

  • Select any path to expand or collapse it. If a path displays the method or PDFs|<METHOD>, select it to display a quick PDF view.

Note: Refresh page to view current API results when any changes are made in the console.

endpoint details pdf
Figure: Quick PDF View from the API EP View

  • Hover over any PDF in the quick view to display the PDF percentile and mean values. Select any PDF to display full PDF view for that specific metric.

specific pdf
Figure: PDF Information for a Metric in PDF Quick View

  • Select the PII & Learnt API tab to view the PII information and also learnt schema at that API level. Select the Request Body, Headers, Query Parameters, and Swagger tabs to view the respective information related to that API. You can also select the Learnt Schema option in case of request body, headers, and query parameters tabs to view the examples in a drop-down list. Select on an entry to view an example API request.

pii learntapi
Figure: Per API PII and Schema

  • Download the swagger specification for the APIs at the AppType level by selecting the Download Swagger option. You can use the specification with a swagger editor to view the API documentation.

swagger spec
Figure: Download Swagger Spec

Note: You can select the download option next to the Learnt Schema option in the Swagger tab to download the swagger specification at that API level.

Step 3: Load the PDFs view.
  • Select the Table option to display the PDFs view for the API endpoints. This view shows tabular list for all API endpoints and displays the collapsed URLs, PDFs for metrics, and last updated time.

pdfs full 2
Figure: API EP PDFs Full View

  • Hover over any PDF to display the PDF percentile and mean value for that metric. Select any PDF to display that specific PDF's full view in graph format.

reqsize pdf
Figure: Detailed PDF View for a Specific Metric

Note: The X-axis represents the metric value and Y-axis represents the probability density.

  • Hover anywhere on the PDF graph to display the probability density for a given metric value. You can also change metric from here using the X Axis drop-down menu.

Note: In case the learning model does not get enough data for an API, it displays blank entries for the PDFs for that API and displays a message on the tool tip mentioning that not enough data is available.

Note: After APIs are learnt, one can download the swagger, edit it if needed and import. Imported swagger will define API Inventory and groups, which can be used to create API protection rules.

Load Balancer Monitoring

Perform the following to inspect the API EPs and PDFs:

Step 1: Open Load Balancers Monitoring.

Note: Configuration also available with WAAP in Web Apps & API Protection service > Apps & APIs > Security.

  • Switch to Load Balancers service and change to desired namespace.

  • Select Virtual Hosts > HTTP Load Balancers. A list of load balancers is presented.

  • Under the name of your load balancer, select either Performance Monitoring or Security Monitoring.

Step 2: Open API Endpoints View.
  • Next to the name of your load balancer at the top, use the monitoring pull down to select Security Monitoring.

  • Select the API Endpoints tab.

Step 3: Observe All API Types.

Note: Refresh page to view current API results when any changes are made in the console.

Note: Select an endpoint to view the Endpoint Details slide-out page.

NEYAPI2 2
Figure: Observe All API Types

Step 4: Configure protection rules and API rate limiting rules.
  • Select Table view above the graph.

NEYAPI3 3
Figure: Observe All API Types

  • Find the path in the Path column that you want to apply protection and rate limiting rules to.

  • Select ... > Edit Protection Rule.

  • Select Configure for Protection Rules.

  • Enter a Name and edit the configuration as needed.

  • Select Apply to save the configuration.

  • Select Apply to save the API protection rules.

  • Select Save and Exit to save the updated load balancer.

  • Select ... > Edit Rate Limit.

  • Select Configure for Rate Limit.

  • In the Rate Limiter Values section, update the parameters as required.

  • Select Apply to save the configuration.

  • Select Save and Exit to save the load balancer.


Concepts


API References