Setting Up API Discovery
Objective
This document provides instructions on configuring API Discovery on the F5® Distributed Cloud Platform. This core capability is essential for dynamically identifying and monitoring your application's API endpoints.
Prerequisites
-
F5 Distributed Cloud Console SaaS account.
Note: If you do not have an account, see Create an Account.
-
Active HTTP Load Balancer configured for your application within the Distributed Cloud environment. See Setting up a load balancer on WAAP.
-
For code discovery - Source Code Management API Key
-
For API Crawling - Standard user, password
Concepts
- API Discovery: The process of dynamically identifying and classifying all API endpoints within your application, including Inventory, Discovered, and Shadow APIs. It involves both detecting APIs from traffic and integrating OpenAPI Specifications to create API definitions.
- Sensitive Data Discovery: This feature identifies and masks sensitive data such as Personally Identifiable Information (PII) within API traffic. This helps ensure data privacy and compliance with regulations.
- Code Scanning API Discovery: This process involves analyzing your application's source code to identify all API endpoints, including those not observable through runtime traffic. By examining code repositories, it uncovers hidden or deprecated APIs, providing a complete API inventory and aiding in security assessments.
- API Crawling: A discovery technique that independently simulates user behavior to detect exposed APIs within your web applications. It focuses on identifying shadow APIs that are not monitored by traditional API discovery methods or code scanning, providing a more comprehensive view of your API landscape.
Configuration
The configuration process is divided into two sections: API Discovery and API Protection.
Activity | Description |
---|---|
Find HTTP Load Balancer | Locate the specific HTTP Load Balancer where API Discovery will be enabled. |
Enable API Discovery from Traffic | Enable API Discovery on the identified HTTP Load Balancer. |
Enable API Discovery from Code | Upload OpenAPI Specification files, create API definitions, and apply them to the load balancer. |
Enable API Discovery from API Crawling | Enable API Discovery on the identified web applications using crawling. |
Set Up API Discovery
Step 1: Find your HTTP Load Balancer.
- Log into Console.
Figure: Console Homepage
-
Click
Web App & API Protection
. -
Select your namespace.
Figure: Select Namespace
- Click
Manage
>Load Balancers
>HTTP Load Balancers
and locate your HTTP load balancer in the list.
Figure: Load Balancers
Step 2: Enable API Discovery.
Figure: Manage Load Balancer Configuration
-
For the target load balancer, click
...
>Manage Configuration
in theAction
column. -
Click
Edit Configuration
in the top right section of the page. -
Scroll down to the
API Protection
section. -
Select
Enable
from theAPI Discovery
drop-down menu. This will also expose the API discovery settings.
Figure: Enable API Discovery
-
Select your desired option from the
Learn From Traffic With Direct Response
drop-down menu. By default, API Discovery learns only from traffic with 2xx (successful) responses. You can choose to selectEnable Learning From Redirect Traffic
to also learn from traffic with 3xx (redirect) responses. -
Use the
Purge Duration for Inactive Discovered APIs
field to specify how often to delete inactive discovered APIs. The purging occurs after the number of days you specify expires. -
Scroll to the bottom of the page and click
Save and Exit
.
Step 3: Add Code Scanning API Discovery.
Figure: Add Code Base Integration
-
Navigate to
Manage
>API Management
>Code Base Integration
. -
Click
Add Code Base Integration
.
Figure: Add Code Base Integration Form
-
Enter a name for this integration and optionally enter labels and a description.
-
Select the code base you want to integrate (GitHub, GitLab, azure, etc.) from the
Code Base
drop-down menu. -
Based on your selected code base, you will need to add the required integration information (i.e. API key/username + password) to the Console. You can obtain integration information for your code base as described in one of the options shown below.
Azure Repos Integration
In Azure DevOps:
-
Create a personal access token in Azure DevOps (Under
User Settings
>Personal Access tokens
). -
Click
New Token
. -
Set a label
F5 integration
. -
Scopes: Ensure
Custom defined
is selected and select the following options:-
Code:
Read
-
User Profile:
Read
-
-
Click
Create
. Then copy your newly created personal access token into the paste buffer.
Bitbucket Cloud/Server Integration
In Bitbucket:
-
Create an app password in Bitbucket.
-
Set a label
F5 integration
. -
Select the following permissions:
- Code:
Read
- Workspace:
Read
- Project:
Read
- Repositories:
Read
- Pull Requests:
Read
- Webhooks:
Read
- Code:
-
Click
Create
. Then copy your newly created app password into the paste buffer.
Back in Distributed Cloud Console done one of the following:
-
Bitbucket Cloud Integration: Enter your Bitbucket username into the
Bitbucket Username
field. -
Bitbucket Server Integration: Fill in the
Bitbucket Server URL
andBitBucket Server Username
fields.
GitHub or GitHub Enterprise Integration
In GitHub:
-
Generate a new token (classic or fine-grained) in GitHub (
Settings
>Developer Settings
>Personal Access tokens
). -
Set a Note
F5 integration
. -
Set expiration date.
-
Configure the scopes as follow:
-
Classic token
- Read_api
- User > read:user
- User > user:email
-
Fine-grained
- Repository permission > Contents:read-only
- Repository permissions > Metadata:read-only
-
Click Generate Token
. Then copy your newly created token into the paste buffer.
Back in Distributed Cloud Console done one of the following:
-
GitHub Integration: Enter your GitHub username into the
GitHub Username
field. -
GitHub Enterprise Integration: Fill in the
GitHub Hostname
andGitHub Username
fields.
GitLab Cloud or GitLab Enterprise Integration
In GitLab:
-
Create an access token (Under
Preferences
>Access Tokens
). -
Set a label
F5 integration
. -
Set expiration date
-
Configure the scope as follows:
- Read_api
- Read_repository
- Read_user
Click Create Personal access token
. Then copy your newly created token into the paste buffer.
Back in Distributed Cloud Console, if you selected GitLab Enterprise Integration, Enter your GitLab URL in the
GitLab URL` field.
In Distributed Cloud Console:
-
Click
Configure
to bring up theSecret Type
form. You can either Blindfold your token/password or you can store it in the clear. -
Paste your token/password into either the
Secret to Blindfold
orSecret
field. -
Click
Apply
to save your token. -
Click
Save and Exit
. -
Go back to your load balancer (from Step 1) and scroll down to the
API Protection
section.
Figure: Enable API Discovery
-
Make sure API discovery is enabled and click
Configure
underAPI Repositories
. This will bring up an empty list of code base integrations. ClickAdd Item
to start the list. -
Select the code base you created above.
-
Select the API repositories you wish to include from the code base.
Figure: Select Code Base
-
Scroll to the bottom and click
Save and Exit
. -
You can now see in the API inventory the source of detection of an API endpoints. (Note: it can take up to two hours for the discovery process to run and show results).
Figure: Monitor API Endpoints
Step 4: Set Up API Crawling.
- Click
Manage
>Load Balancers
>HTTP Load Balancers
and locate your HTTP load balancer in the list.
Figure: Load Balancers
Figure: Manage Load Balancer Configuration
-
For the target load balancer, click
...
>Manage Configuration
in theAction
column. -
Click
Edit Configuration
in the top right section of the page. -
Scroll down to the
API Protection
section. -
Select
Enable
from theAPI Crawling
drop-down menu (API Discovery
must be enabled to see theAPI Crawling
drop-down menu). This will also expose theDomains to Crawl
list.
Figure: Enable API Discovery
-
For each domain you want to crawl,
- Click
Add Item
in theDomains to Crawl
list. - Enter the domain you want to crawl. Optionally you can click in the
Domain
field and then clickSee Suggestions
. - Enter the username in the
User
field. - To enter the password, click
Configure
to bring up theSecret Type
form. You can either Blindfold your token/password or you can store it in the clear.- Paste your token/password into either the
Secret to Blindfold
orSecret
field. - Click
Apply
to save your secret (password).
- Paste your token/password into either the
- Click
Apply
to save your domain to theDomains to Crawl
list.
- Click
-
Scroll to the bottom of the page and click
Save and Exit
. -
Scanning will begin shortly. You will soon (up to 4 hours) start seeing results in your
API Endpoints
page for this load balancer under theDiscovery Source
column.
Figure: Monitor API Endpoint Sources
Limitations
Discovery From Code
-
Code scan intervals:
Type of Scan Frequency Discovering new repositories once a day Scanning existing repositories once a day Checking integration token validity every 5 minutes -
Supported languages and frameworks:
Language Frameworks Java EE,
Spring.Net Versions 7.0, 6.0, 5.0, 3.1 Python Flask,
Django - function-based viewsJavascript Express,
HapiGo Gin -
Supported SCM platforms:
- Azure Repos
- Bitbucket Cloud
- Bitbucket Server
- GitHub
- GitHub Enterprise
- GitLab
- GitLab Enterprise
-
Code base integration will scan repos from the organization level, not from the user level.
Discovery from Crawling
-
API Crawling currently doesn't support MFA.
-
Scan interval: Monthly, triggered upon each configuration change/setup.
-
Scan Requirements: API crawling requires credentials; it cannot run without them.
-
Scan Time: Scans may take up to 4 hours to complete. If expected results are not visible after 4 hours, please contact our support team.