Create KVM Site
On This Page:
- Objective
- Prerequisites
- Create a Site Token
- Check for Virtualization Support
- Install Packages
- Configure HugePages
- Create a Virtual Network
- Install Node using Terminal
- Install Node Using the Virtual Machine Manager
- Post-Install Node Parameter Configuration
- Register the Site
- Single-Node Site Registration
- Multi-Node Site Registration
- Deploy Site Using Terraform
- Access Site Local UI
- Concepts
Objective
F5® Distributed Cloud Services support site deployment for a Kernel-based Virtual Machine (KVM) with libvirt. Use the instructions provided in this document to perform Distributed Cloud Services node installation on a server using a KVM with libvirt and to perform site registration on F5® Distributed Cloud Console (Console).
The following is a sample topology for the hypervisor and guest machine setup. The sample shows three guest virtual machines installed with F5 Distributed Cloud Services Site software running on the hypervisor host:
Prerequisites
-
A Distributed Cloud Services Account. If you do not have an account, see Create an Account.
-
A server supporting hardware virtualization. The KVM will only work if the server CPU has hardware virtualization support (Intel VT or AMD-V).
-
An operating system installed on the host:
-
Ubuntu 16.x/18.x/20.x
-
CentOS 7.x/RHEL 7.x
-
-
At least one interface with Internet reachability.
-
Distributed Cloud Services Node Zero Touch Provisioning requires Internet connectivity to Console.
-
A KVM image file. Click here to download.
-
Resources required per node: Minimum 4 vCPUs and 14 GB RAM.
-
45 GB is the minimum amount required for storage. However, if you are deploying an F5® Distributed Cloud App Stack Site, 100 GB is the recommended minimum amount of storage.
-
By proceeding with the installation, download and/or access and use, as applicable, of the Distributed Cloud Services software, and/or Distributed Cloud Services platform, you acknowledge that you have read, understand, and agree to be bound by this agreement.
Create a Site Token
Create a site token or use an existing token. If you are configuring a multi-node site, use the same token for all nodes.
Step 1: Sign into Console and navigate to site tokens.
- Click
Multi-Cloud Network Connect
.
- Select
Manage
>Site Management
>Site Tokens
.
Step 2: Generate a new site token.
-
Click
Add site token
to create a new token. -
In the
Name
field, enter the token name. -
In the
Description
field, enter a description for the token. -
Click
Add site token
.
Step 3: Note down the new token.
-
Find the token previously created or choose an existing token from the list of tokens displayed.
-
Click
>
to expand the token details in JSON format and note down the value of theuid
field.
Check for Virtualization Support
Check if your system supports virtualization. The commands differ and are based on your particular Linux distribution.
Step 1: Check if virtualization is enabled on Ubuntu or CentOS/RHEL.
Enter egrep '(vmx|svm)' /proc/cpuinfo | wc -l
. This command returns a non-zero number if virtualization is enabled.
egrep '(vmx|svm)' /proc/cpuinfo | wc -l
Note: For Ubuntu, you may need to run the
sudo apt install cpu-checker
command prior to continuing. This action will install thecpu-checker
package.
Step 2: Verify if KVM acceleration is available on Ubuntu only.
Enter sudo kvm-ok
. This command returns a message that confirms that KVM acceleration can be used
.
sudo kvm-ok
KVM acceleration can be used
Install Packages
Install all required packages. You need root privileges to update or install packages.
Install the required packages for Ubuntu
- Enter
sudo apt update
.
sudo apt update
- Enter
sudo apt install qemu qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager
.
sudo apt install qemu qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager
Install the required packages for CentOS/RHEL
- Enter
sudo yum update
.
sudo yum update
-
Confirm with
y
to perform the update. -
Enter
sudo yum install qemu-kvm qemu-img virt-manager libvirt libvirt-client virt-install virt-viewer libvirt-python bridge-utils
.
sudo yum install qemu-kvm qemu-img virt-manager libvirt libvirt-client virt-install virt-viewer libvirt-python bridge-utils
- Confirm with
y
to perform the installation.
Configure HugePages
Configure the HugePages to support increased virtual memory management that is greater than the default for your Linux system.
The steps below provide an example for setting the HugePages configuration for a cluster of three nodes. The required number of HugePages per node is 400.
The commands differ and are based on your particular Linux distribution.
Note: You can use a text editor, such as
vim
ornano
, to open the configuration file and make edits.
Step 1: Add HugePages configuration.
-
For the hypervisor host on Ubuntu, add the following in the Grub configuration file (
/etc/default/grub
):GRUB_CMDLINE_LINUX="default_hugepagesz=2M hugepagesz=2M hugepages=1200"
GRUB_CMDLINE_LINUX="default_hugepagesz=2M hugepagesz=2M hugepages=1200"
-
For the hypervisor host on CentOS/RHEL, add the following to the
/etc/sysctl.conf
file:vm.nr_hugepages = 1200
vm.nr_hugepages = 1200
Step 2: Update the configuration to make HugePages effective.
-
For Ubuntu, run the following commands in the terminal:
-
sudo update-grub
-
sudo grub-mkconfig -o /boot/grub/grub.cfg
-
sudo grub-mkconfig -o /boot/grub/grub.cfg
-
Reboot your system.
-
For CentOS/RHEL, run the following commands in the terminal:
-
sysctl -p
. -
Confirm the output is correct.
-
Reboot your system.
-
Step 3: Check the HugePages configuration after the host reboot.
- Enter
cat /proc/meminfo | grep Huge
.
cat /proc/meminfo | grep Huge
- Confirm the configuration settings are correct.
AnonHugePages: 6100992 kB
ShmemHugePages: 0 kB
HugePages_Total: 1200
HugePages_Free: 1200
HugePages_Rsvd: 0
HugePages_Surp: 0
HugePagesize: 2048 kB
Create a Virtual Network
Create a new virtual network to use with your KVM site. Perform these steps only if you are creating a KVM site using the terminal.
Step 1: Create a new libvirt network.
-
In the terminal, use the
touch
command to create an XML file with a.xml
extension. -
Open the XML file using
vim
ornano
text editors. -
Populate the network fields. Set a different subnet other than the underlay network.
This example provides sample file entries:
<network>
<name>virtualnetwork1</name>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='bridge1' stp='on' delay='0'/>
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254'/>
</dhcp>
</ip>
</network>
- Save your information and close the text editor.
Step 2: Define the virtual network from the XML file created in Step 1.
In the terminal, enter sudo virsh net-define <virtual-network-name>.xml
.
sudo virsh net-define <virtual-network-name>.xml
Step 3: Start the virtual network and enable it for autostart.
- To start the network, enter
sudo virsh net-start <virtual-network-name>
.
sudo virsh net-start <virtual-network-name>
- To enable autostart, enter
sudo virsh net-autostart <virtual-network-name>
.
sudo virsh net-autostart <virtual-network-name>
Step 4: List the libvirt networks to verify that the virtual network was created.
To list the libvirt networks, enter sudo virsh net-list
.
sudo virsh net-list
This example provides sample output:
Name State Autostart Persistent
----------------------------------------------------------------
default active yes yes
virtualnetwork1 active yes yes
Step 5: Optionally, list your bridge devices.
Enter brctl show
.
brctl show
This example provides sample output:
bridge name bridge id STP enabled interfaces
virbr0 8000.5254003339b3 yes virbr0-nic
virbr1 8000.52540060f86e yes virbr1-nic
Install Node using Terminal
Create and install a node the using the terminal.
Step 1: Create a Virtual Disk Image (VDI) file.
- In the terminal, enter
sudo qemu-img create /var/lib/libvirt/images/volterra.qcow2 45G
.
This example creates a virtual disk file with name volterra.qcow2 45G
:
sudo qemu-img create /var/lib/libvirt/images/volterra.qcow2 45G
Step 2: Create a new virtual machine.
This list provides the option descriptions for the virt-install
command:
-
name
: Name of the guest virtual machine. -
ram
: RAM allocation for guest virtual machine (minimum 8 GB, recommended 16 GB). -
vcpus
: The CPU allocation for guest virtual machine (minimum 4). -
network
: The name of the virtual network that the Customer Edge (CE) server should be attached to. You can also specify bridge for network. -
disk path
: The virtual disk path. -
cache
: Ensure that thecache=none
option is specified in the command.
Note: KVM installation is supported only for the SCSI disk type and virtio model for
network
type.
- Run the
virt-install
command with the following options set:
sudo virt-install --name Volterra --ram 14000 --vcpus=4 --network network=virtualnetwork1 model=virtio --accelerate -–disk path=/var/lib/libvirt/images/volterra.qcow2 bus=virtio cache=none size=64 --cdrom /scratch/vsb-ves-ce-certifiedhw-generic-production-centos-7.2006.9-202010131432.1602604079.iso --noautoconsole --noreboot
Step 3: Optionally, start the virtual machine.
Start the virtual machine using the virsh
command. Note that in some cases the virtual machine may not start yet.
- Enter
sudo virsh start <node-name>
.
sudo virsh start <node-name>
Step 4: Verify the status of the virtual machine.
- Enter
sudo virsh list –all
.
sudo virsh list --all
This example provides the sample output of the command:
Id Name State
----------------------------------------------------
31 ce-volterra running
Step 5: Connect to the virtual machine using console.
- Enter
sudo virsh console 31
.
sudo virsh console 31
This example provides the sample output of the command:
Connected to domain ce-volterra
Escape character is ^]
This is localhost (Linux x86_64 4.19.56-coreos-r1) 22:53:47
SSH host key: SHA256:aIFHRiKcjFWaF47+9AAd6IGzCQ5M6suHzL0xH3QUaS4 (ECDSA)
SSH host key: SHA256:hmWTu9M5GYKDQcTgks7LDTkQK9LXKhQjSodVO14JICo (ED25519)
SSH host key: SHA256:ewojrrFELDNOZWIW+jp2UgSaQVikpnGupdbu9+ObgZw (RSA)
SSH host key: SHA256:dM0wUNcFqfJxVRGwwzOJBLtTY1JdptmtjyHXthnBKqY (DSA)
eth0: 192.168.122.197 fe80::5054:ff:fe89:abb8
Step 6: Optionally, connect to the virtual machine using IP address.
Enter sudo virsh domifaddr ce-volterra
.
sudo virsh domifaddr ce-volterra
This example provides the sample output of the command:
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet0 52:54:00:89:ab:b8 ipv4 192.168.122.197/24
Install Node Using the Virtual Machine Manager
Create and install a node using the Virtual Machine Manager application. The Virtual Machine Manager application requires that you pre-install the virt-manager
package.
Step 1: Initiate the Virtual Machine Manager on the hypervisor host.
-
For CentOS/RHEL: Launch the application from
Applications
>System Tools
>Virtual Machine Manager
. -
For Ubuntu: Launch the application from
Applications
>Virtual Machine Manager
. -
From a terminal, enter
virt-manager
.
Step 2: Create a virtual network.
-
Click
Edit
>Connection Details
. -
Follow the instructions at CREATING A VIRTUAL NETWORK to complete the virtual network set up.
Step 3: Create a virtual machine.
- Click
Create a new virtual machine
.
- Confirm the installation information is correct, and then click
Forward
.
Step 4: Select the node ISO file for installation.
- Click
Browse
to navigate and select the ISO image file.
- Click
Forward
.
Step 5: Configure memory and CPU settings.
-
In the
Memory (RAM)
field, use the+
or–
buttons to set the memory amount. -
In the
CPUs
field, use the+
or–
buttons to set the number of CPU cores. -
Click
Forward
.
Step 6: Configure the disk image settings.
-
Confirm the option for
Enable storage for this virtual machine
is enabled. -
Under the
Create a disk image for the virtual machine
field, use the+
or–
buttons to set the amount of storage. -
Click
Forward
.
Step 7: Configure the network.
-
From the
Network selection
drop-down menu, selectVirtual network ‘default’ : NAT
. -
Confirm all other settings are correct.
-
Click
Finish
.
Step 8: Configure the NIC driver and CPU.
- For the NIC driver, confirm the settings are correct, or adjust them and then click
Apply
.
- For the CPU(s), confirm the settings are correct, or adjust them and then click
Apply
.
Step 9: Complete the creation of the virtual machine.
-
Click
Finish
. -
Start the virtual machine from the Virtual Machine Manager options.
Post-Install Node Parameter Configuration
At any point, you can sign in to a node via SSH using the admin
username and Volterra123
password to configure parameters.
Note: If you did not sign in previously, you will be prompted to update the default password for the
admin
username. Follow the instructions to update the default password.
Step 1: Sign in to the node using your credentials.
The login shell loads with different options to select for configuration.
Press the Tab
key to select parameters to configure.
Step 2: Verify the configuration.
-
Select
get-config
. -
Confirm the settings are correct.
Step 3: Optionally, perform network configuration.
-
Select
configure-network
. -
Follow the prompts to configure network settings.
Note: You cannot change an IP address for a registered node for a multi-node site. You must use fixed IP addresses or DHCP addresses with a fixed lease.
Step 4: Optionally, configure your Wi-Fi network.
Configuring the network is optional. If you want to apply static configuration, then this option can be used.
Optionally, enter Y
for Do you want to configure wifi?
.
Step 5: Configure the main options.
-
Press the
Tab
key to select theconfigure
option. -
Enter a cluster name.
-
Enter the registration token.
-
Enter a hostname. This option is set to
master-0
by default.
Note: Ensure that hostnames are unique if you are installing nodes for a multi-node site.
-
Enter the longitude and latitude information.
-
Select
kvm-voltmesh
for the certified hardware.
Note: You must first perform network configuration using the
configure-network
option before setting the other fields using theconfigure
option in case you are applying static configuration for the network. Also, note that changing the assigned IP address after the successful registration of the node is not supported in cases of multi-node sites.
-
Certified Hardware: The supported options are as follows.
-
kvm-voltmesh
: Single service level objective eth0 interface hardware. -
kvm-voltstack-combo
: For managed k8s and vk8s use cases. -
kvm-regular-nic-voltmesh
: Two virtual NICs, when eth0 is SLO dedicated and eth1 is binded as regular interface, which can be optionally configured from Console post registration.
-
Note: The default option is
kvm-regular-nic-voltmesh
, and this supports 2 interfaces (ensure that both eht0 and eth1 are available). However, this is loaded by default with eth0 interface set as a dedicated site local outside interface. The other interface is an optional regular interface and can be only configured from Console using fleet after the site is provisioned. For instructions on configuring interfaces using fleet, see Create a Fleet document.
Note: For more information on choosing an appropriate image for your certified hardware, see How to choose an image for your site deployment?.
Step 6: Confirm configuration.
Enter Y
to confirm configuration.
Step 7: Verify configuration status.
-
Press the
Tab
key to select thehealth
option. -
Verify your Wi-Fi configuration and registration status.
Note: You can select the
factory-reset
option to perform a configuration reset and repeat the registration process again per the instructions below.
Note: If you use an NTP server, ensure that the server is reachable. Else, leave the NTP server configuration empty so that the F5 Distributed Cloud NTP servers are used.
Register the Site
Register the Distributed Cloud Services node(s) installed previously as a Site in Console.
Note: The USB allowlist is enabled by default. If you change a USB device, such as a keyboard after registration, the device will not function.
Single-Node Site Registration
Step 1: Navigate to the site registration page.
-
Sign in to Console.
-
Click
Multi-Cloud Network Connect
. -
Click
Manage
>Site Management
>Registrations
.
Step 2: Complete site registration.
-
Under
Pending Registrations
, find your node name and then click the blue checkmark. -
In the form that appears, fill in all required fields with the asterisk symbol (
*
). -
Enter a latitude value and a longitude value.
-
Enter other configuration information, if needed.
-
Click
Save and Exit
.
Step 3: Check site status and health.
It may take a few minutes for the site health and connectivity score information to update.
-
Click
Sites
>Site List
. -
Click on your site name. The
Dashboard
tab appears, along with many other tabs to inspect your site. -
Click the
Site Status
tab to verify the following:-
The
Update Status
field has aSuccessful
value for theF5 OS Status
section. -
The
Update Status
field has aSuccessful
value for theF5 Software Status
section. -
The
Tunnel status
andControl Plane
fields under theRE Connectivity
section haveup
values.
-
Multi-Node Site Registration
Step 1: Navigate to the site registration page.
-
Sign in to Console.
-
Click
Multi-Cloud Network Connect
.
- Click
Manage
>Site Management
>Registrations
.
Step 2: Accept the registration requests.
Registration requests are displayed in the Pending Registrations
tab.
-
Click
Accept
to accept the registration requests from themaster-0
,master-1
, andmaster-2
nodes. -
Enter the same values for the following parameters for all the registration requests:
-
In the
Cluster name
field, enter a name for the cluster. Ensure that all master nodes have the same name. -
In the
Cluster size
field, enter3
. Ensure that all master nodes have the same cluster size.
-
-
Enter all mandatory fields marked with the asterisk (
*
) character.
Step 3: Check site status and health.
It may take a few minutes for the site health and connectivity score information to update.
-
Click
Sites
>Site List
. -
Click on your site name. The
Dashboard
tab appears, along with many other tabs to inspect your site. -
Click the
Site Status
tab to verify the following:-
The
Update Status
field has aSuccessful
value for theF5 OS Status
section. -
The
Update Status
field has aSuccessful
value for theF5 Software Status
section. -
The
Tunnel status
andControl Plane
fields under theRE Connectivity
section haveup
values.
-
Deploy Site Using Terraform
Create and deploy a single-node KVM site using Terraform.
Step 1: Confirm Terraform is installed.
Terraform can be located on a different node than the host running KVM. In the main.tf
file cloned in Step 4, you can change the uri
variable under provider
to point to your KVM location.
In a terminal, enter terraform version
. If you need to install, follow the instructions at the official guide.
Step 2: Confirm hardware virtualization is enabled.
-
To confirm hardware virtualization is supported by your CPU, enter
egrep -c '(vmx|svm)' /proc/cpuinfo
. This command returns a non-zero number if virtualization is enabled. For example, if a2
is returned, then your system supports virtualization. If a0
is returned, then your system does not support virtualization. -
To confirm KVM acceleration for Ubuntu-based systems only, enter
sudo kvm-ok
. Confirm the output is the following:
INFO: /dev/kvm exists
KVM acceleration can be used
-
For Ubuntu only:
- If
sudo kvm-ok
does not work, entersudo apt install cpu-checker
. Then rerun thesudo kvm-ok
command.
- If
Step 3: Confirm KVM is installed.
For your specific Linux distribution, the installation will be different.
For information on Ubuntu, see KVM Installation.
For information on Red Hat, see Installing KVM packages on an existing Red Hat Enterprise Linux system.
Ubuntu
-
Update your software package repository. Enter
sudo apt update
. -
To install the necessary packages, enter
sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils
. -
If asked to confirm installation, enter
y
. -
Confirm your username is added to these groups:
kvm
andlibvirt
:
sudo adduser `id -un` libvirt
sudo adduser `id -un` kvm
-
Reboot your machine for the changes to take place.
-
Confirm installation and configuration is applied correctly. Open a terminal and enter
virsh list --all
. -
To use libvirt with the XML extension, install with
sudo apt install xsltproc
.
CentOS
-
Update your software package repository. Enter
sudo yum update
. Entery
to confirm. -
To install the necessary packages, enter
sudo yum install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils
. -
If asked to confirm installation, enter
y
. -
Confirm your username is added to these groups:
kvm
andlibvirt
. If not, add them. -
Reboot your machine for the changes to take place.
-
Confirm installation and configuration is applied correctly. Open a terminal and enter
virsh list --all
. -
To use libvirt with the XML extension, install with
sudo yum install xsltproc
.
Step 4: Clone repository folder.
-
Navigate to KVM-CE and use Git to clone the repository folder to your desired location.
-
In a terminal, navigate to the directory where the cloned repository folder is.
-
Enter
cd kvm-ce-main/terraform/
. -
Confirm that the following folder and three files now exist in the current directory with
ls -lh
:-
cloudinit
-
cpu-pinning.xsl
-
main.tf
-
var.tf
-
Step 5: Provide values for the variables.
Open the var.tf
file and provide all necessary values for the variables. For information on the values, see KVM CE Deployment.
Step 6: Optionally, add another network interface.
- Add another interface (eth1 - inside) in the
main.tf
(kvm-ce libvirt_domain resource).
network_interface {
network_name = "default"
}
network_interface {
network_name = "inside"
}
- In Console, configure your Fleet with interface assignment to outside and inside. See example below.
Step 7: Increase file system partition.
-
Use the
wget
command to download the KVM CE qcow2 image. -
Enter
qemu-img resize
command to resize the virtual disk storage. For example:qemu-img resize centos-7.2009.5-202103011045.qcow2 50G
. -
In the
var.tf
file, update thekvm-ce-qcow2
variable to point to the local qcow2 image downloaded previously.
Step 8: Apply Terraform parameters.
-
Enter
terraform init
. -
Enter
terraform apply
.
Step 9: Confirm site deployed.
In Console, navigate to the list of sites to register and confirm your site is on the list.
Access Site Local UI
After you create and register your site, you can access its local user interface (UI) to perform certain configuration and management functions. For more information, see Site Local UI Usage.