Configure Streaming of Logs
Objective
This guide provides instructions on how to enable streaming of all system and application logs to an external log collection or monitoring system in F5® Distributed Cloud Services. This is so that you can analyze events and measure application performance while establishing correlation with respect to other systems of your organization.
Streaming of logs is supported only for the following types of logs:
- Layer 7 request logs
- Network logs (if enabled through network policy)
Using the instructions provided in this guide, you can configure log streaming through a fleet and enable the streaming of logs for the sites that are part of that fleet.
Prerequisites
-
A Distributed Cloud Services Account. If you do not have an account, see Create an Account.
-
One or more sites. See Site Management for information on site creation.
-
An external log collection or monitoring system reachable from the site. Ensure that the hostname is resolved and reachable from your F5 site.
Enable Streaming of Logs
The example shown in this guide sets up a syslog server in a Docker container, and creates a log receiver object in the F5® Distributed Cloud Console for the syslog server. This example shown creates log receiver as part of the F5 fleet configuration. However, you can also create a log receiver object individually, and later can add it to a fleet.
Note: Streaming of logs is also supported for AWS Elasticsearch and Splunk.
This example starts a syslog server in a Docker container.
Step 1: Start external log collection server.
- Start Docker container for syslog and expose ports 514 and 601 for UDP and TCP connections, respectively. F5® Distributed Cloud Console uses these ports by default for streaming logs when log streaming is enabled.
Note: You can configure a log collection system of your choice. However, ensure that the host is reachable from your site from which logs are intended to be streamed.
docker run -it --rm -p 514:514/udp -p 601:601 --name syslog-ng balabit/syslog-ng:latest
- Optionally, log into the container to start displaying tail logs.
docker exec -it syslog-ng tail -f /var/log/messages
- Ensure that the syslog host is resolved from the site. This example adds an entry in the
/etc/hosts
file on your site with the hostname resolution.
192.168.1.33 syslog.mydomain
Step 2: Start creating new fleet.
- Open
F5® Distributed Cloud Console
and selectMulti-Cloud Network Connect
.
Figure: Homepage
-
Select
Manage
in left-menu > selectSite Management
. -
Select
Fleets
option in pop-up window. -
Select
Add Fleet
.
Figure: Fleets
- Enter a name for your fleet and enter a label string for the
Fleet Label Value
field. TheFleet Label Value
field is not available for editing an existing fleet.
Note: For detailed instructions on fleet configuration, see Create Fleet guide.
Step 3: Start creating new log receiver.
-
Go to the
Advanced Configuration
section and enable theShow Advanced Fields
option. -
Select
Enable Logs Streaming
option from theLogs Streaming
menu. -
Select on the
Enable Logs Streaming
drop-down menu. -
Select
Add Item
from the displayed options in the drop-down list. This starts a log receiver creation form.
Note: The drop-down list displays log receiver objects if those are already created. You can also select an existing log receiver. This example shows creating a new receiver.
Figure: Enabling Log Streaming in Fleet
Step 4: Configure options and apply it to fleet.
-
Enter a
Name
for the log receiver. -
From the
Syslog Transport Mode
menu, select whether to use UDP or TCP for transport. The default port numbers for UDP and TCP are 541 and 601, respectively. The system streams the logs using these ports by default. -
In the
Server name
field, enter the name of your external log collection host. -
Select
Continue
to create the new log receiver.
Figure: Log Receiver Creation
Step 5: Complete fleet creation.
-
In the
Advanced Configuration
section, enable theEnable Default Fleet Config Download
option. -
Select
Save and Exit
. The sites that are part of this fleet start streaming logs to the external syslog server specified in the log receiver configuration. -
Optionally, apply the fleet label to your sites from which you want to stream logs to the syslog server. For instructions on adding sites to a fleet, see Create Fleet guide.
-
Verify that the logs are streamed by logging into the Docker container.