Create AWS Site with TGW

Objective

This guide provides instructions on how to create an F5® Distributed Cloud Services Amazon Web Services (AWS) Transit Gateway (TGW) site using F5® Distributed Cloud Console (Console). For more information on sites, see Site.

Using the instructions provided in this guide, you can create an AWS TGW site object in Console and deploy the virtual private cloud (VPC) with TGW site using the object.

Note: Configuring site mesh group is not supported for the sites deployed from Console.


Design

The AWS TGW Site is a way to orchestrate/automate the deployment and management of AWS TGW related resources and other resources needed to deploy an F5® Distributed Cloud Services AWS site on a new or existing VPC, which is called a Services VPC.

AWS TGW - Site Deployment & TGW Creation

An AWS TGW site does the following:

  • Automates the creation of the TGW resource, the TGW route table, and the VPN connection between the TGW and Distributed Cloud Services site.

  • Attaches the VPN connection to the TGW.

  • Adds the default routes to the main route table of the attached VPCs.

There are two TGW route tables which are created:

  1. VPC route table.

  2. Services route table.

The VPC route table is where all the VPC attachments will be attached, and the route will be propagated from the site via BGP over VPN attachment. The site advertised default route will be installed in the VPC route table so that it can attract all the traffic coming from the VPC attachments attached to the VPC route table.

The Services route table is where the VPN connection to the site is attached; the routes of VPC attached to the TGW will be propagated into the services route table. The same VPC CIDR routes will be learned by the site via the BGP connection to the TGW.

The following shows North-South traffic from the Spoke VPC, as indicated in Figure: AWS TGW - Services VPC + TGW + Single VPC Attachment.

  • Egress FROM Spoke VPC (HR): Traffic originating from source 192.168.100.0/22 lands in VPC route table. Traffic destined to ANY will match the 0.0.0.0/0 route pointing to an Equal Cost Multi-Path (ECMP) toward interfaces vpn-att-4 and vpn-att-5 (which are the Distributed Cloud Services nodes installed in the service VPC), and is eventually sent out toward the TGW (after F5® Distributed Cloud Mesh features and policies are applied).

  • Ingress TO Spoke VPC (HR): Traffic originating from source (anywhere) landing in the services route table will match on destination 192.168.100.0/22. This has a matching interface of vpc-att-3. Traffic is then forwarded to the spoke VPC.

Figure: AWS TGW - Services VPC + TGW + Single VPC Attachment
Figure: AWS TGW - Services VPC + TGW + Single VPC Attachment

AWS TGW - VPC Attachments

You can create a VPC attachment of spoke VPC to TGW not only while creating the initial AWS TGW Site but also after the site is deployed. You can go to the VPC attachments section and add vpc-id, and then you can assign a key-value label for each vpc-id. These labels can be used while creating network policy to allow traffic between the VPCs and to the Internet.

Once VPC attachments are added to the AWS TGW Site and apply action is completed, all these VPCs will be attached to the TGW. These VPC attachments will be associated with the VPC route table so that all traffic coming from the VPC will be routed to the site because of the default route pointing to the VPN attachment. Same VPC attachments will be added to the services route table in a way that VPC CIDR routes are propagated to the Customer Edge (CE).

Site deployment workflow will create a default route pointing to the transit gateway in the main route table of all VPCs attached to the TGW.

Figure: AWS TGW with Additional VPC Attachments
Figure: AWS TGW with Additional VPC Attachments

East-West Traffic:

  1. From the main route table of the VPC, the traffic will be directed toward the transit gateway because of the default route.

  2. In the transit gateway’s VPC route table, a route lookup is done and moves to the site, which is the next hop (NH).

  3. In the site’s route table, it will have all the VPC routes learned from TGW. The NH is set as TGW.

  4. Next, the lookup is done in the services route table and goes to the destination VPC using the attachment.

North-South Traffic:

The following is the ingress/egress traffic flow from VPC to the Internet:

  1. From the main route table of the VPC, the traffic will be directed toward the transit gateway because of the default route.

  2. In the transit gateway’s VPC route table, a route lookup is done and then moves to the site, which is the next hop.

  3. In the site route table, the default route points to the forward proxy, which connects the inside network to outside network. SNAT is performed on the outside interface and traffic is sent to the Internet.

Network Policies

The site can be your ingress/egress and East/West security policy enforcement point, as all the traffic coming from attached VPCs will flow through the site. If the traffic does not match the type defined in your network policy, then the default action will be to deny it.

Network Policies Between Attached VPCs (East-West Traffic)

It is a common use case for enterprises to have workloads of one department or environment spread across multiple VPCs, and one must be able to create a single network policy which could be applied for multiple VPCs attached.

For such scenarios, you can assign the same labels to group the VPCs attached to the TGW. The exact same labels can be used as the label selector while selecting an endpoint during network policy. You can then define ingress and egress policies with respect to that endpoint. This network policy will be applied for all traffic going towards or coming from the VPCs which match the label selector labels.

Network Policies for Ingress/Egress Traffic

Even for ingress/egress traffic, you can continue using a label selector to select the VPCs for which you are defining the network policy. You can define the egress policy by adding the egress rules from the point of VPC to deny/allow a specific traffic pattern. You can also add ingress rules to deny/allow traffic coming toward the endpoint based on the intent.

Forward Proxy Policy for Attached VPCs

Using a forward proxy policy, you can specify allowed/denied TLS domains or HTTP URLs. The traffic from workloads on private subnets toward the Internet via the AWS TGW site is allowed or denied accordingly.

AWS Direct Connect Orchestration

Direct Connect enables you to connect your on-premise data centers to a VPC in which the Distributed Cloud Services sites are hosted. Distributed Cloud Services automatically discovers the on-premise data center routes advertised by on-premise routers connected to AWS routers via Direct Connect. These routes will be learned on the inside network of the Site. There are two supported modes of Direct Connect private Virtual Interface (VIF).

Note: The prerequisite is that the Direct Connect connection is managed by the user.

Standard VIF: In this mode, site orchestration creates the Direct Connect gateway (DCGW) and Virtual Private Gateway (VGW). Ensure that you connect one or multiple VIFs to the DCGW.

Hosted VIF: In this mode, site orchestration accepts the configured list of VIFs delegated from the Direct Connect connection owner account to the hosted VIF acceptor account. You can set a list of VIF IDs to be accepted. The site orchestration then creates the DCGW, VGW, and connects the VIFs to the DCGW.


Prerequisites

The following prerequisites apply:


Deployment

The following video shows the AWS TGW site creation and site deployment workflow using Console:

AWS TGW site creation and management requires performing the following sequence of actions:

Phase Description
Create AWS TGW Site Object Create the TGW site object in Console using the guided wizard.
Deploy Site Deploy the VPC and site configured in the TGW site object using automated method.

Create AWS TGW Site Object

The wizard to create the TGW site object guides you through the steps for required configuration. This document covers each guided step and explains the required actions to be performed for each step.

Perform the following steps:

Step 1: Start AWS TGW site object creation.
  • Log into Console.

  • Click Cloud and Edge Sites.

Figure: Console Homepage
Figure: Console Homepage

  • Click Manage > Site Management > AWS TGW Sites.

  • Click Add AWS TGW Site.

Figure: Create AWS TGW Object
Figure: Create AWS TGW Object

  • In the Metadata section, enter a name for your TGW site object.
Step 2: Configure the TGW and VPC settings.
  • In the AWS Configuration section, perform the following:

    • Click Configure.

    • From the AWS Region menu, select the region based on your AWS account.

Step 2.1: Configure services VPC.
  • From the Select Services VPC menu, select an option and configure per the following guidelines:

    • For the New VPC Parameters option, select an option from the AWS VPC Name menu. The Autogenerate VPC Name option is selected by default. If you select the Choose VPC Name option, enter a VPC name in the Choose VPC Name field.

    • Enter the Classless Inter-Domain Routing (CIDR) block in the Primary IPv4 CIDR block field.

Figure: New Services VPC Configuration
Figure: New Services VPC Configuration

  • For the Existing VPC option, enter an existing VPC name in the Existing VPC field. If you use this option, you are required to enable the enable_dns_hostnames field in the existing VPC configuration.
Step 2.2: Configure the TGW settings.
  • In the Transit Gateway section, select an option from the Select Transit Gateway menu, and configure per the following guidelines:

    • For the New TGW Parameters option, select an option for the Select BGP ASN field. If you select Automatic, Distributed Cloud Services assign the ASNs for the TGW and site. For the User will assign ASN for TGW and Volterra Site option, enter the ASNs for Enter TGW ASN and Enter Volterra Site ASN fields. The supported ASN range is from 64513 to 65534.

    • For the Existing TGW option, enter the TGW ID in the Existing TGW ID field. Enter the ASNs for Enter TGW ASN and Enter Volterra Site ASN fields.

Figure: TGW Configuration
Figure: TGW Configuration

Step 2.3: Configure site node parameters.
  • In the Site Node Parameters section, configure per the following guidelines:

    • Select an option from the AWS Instance Type for Node menu.

    • Enter your public key in the Public SSH key field for SSH access to your node later.

    • In the Ingress/Egress Gateway (two Interface) Nodes in AZ field, click Add Item.

    • Select an option from the AWS AZ Name menu that matches the configured AWS Region.

    • From the Workload Subnet menu, select New Subnet or Existing Subnet ID.

    • Enter either a subnet address in the IPv4 Subnet field or a subnet ID in Existing Subnet ID field.

    • From the Subnet for Outside Interface menu, select New Subnet or Existing Subnet ID.

    • Enter either a subnet address in the IPv4 Subnet field or a subnet ID in the Existing Subnet ID field.

Note: Workload subnet is the network where your application workloads are hosted. For successful routing towards applications running in workload subnet, an inside static route to the workload subnet CIDR needs to be added on the respective site object.

Note: The AWS Certified Hardware option is set to aws-byol-multi-nic-voltmesh by default. You can add more than one node using the Add item option.

  • Click Add Item.
Step 2.4: Set the deployment type.
  • In the Deployment section, select Automatic Deployment from the Automatic Deployment menu.

  • Select an existing AWS credentials object or click Create new Cloud Credential to the load new credential creation wizard.

Note: Refer to the Cloud Credentials guide for more information. Ensure that the AWS credentials are applied with required access policies in accordance with the Policy Requirements document.

  • Click Continue.

  • Click Apply to apply the settings to the AWS TGW object.

Step 3: Optionally, configure VPC attachments.
  • In the VPC attachments section, click Configure.

  • Click Add Item.

  • Enter the VPC ID in the VPC ID field. Select labels from the Labels For VPC ID menu.

  • Click Add Item.

  • Click Apply.

Note: You can add multiple VPC attachments using the Add Item button. You can add VPC attachments during AWS TGW site creation, or you can edit an existing TGW site configuration to add VPC attachments.

Step 4: Optionally, perform TGW network configuration.
  • In the Network Configuration section, click Configure.

  • Click Show Advanced Fields to enable the advanced options.

  • From the Manage Static Routes for Inside Network menu, select Manage Static Routes and perform configuration per the following guidelines for the List of Static Routes field:

    • Click Add Item.

    • Select Simple Static Route and enter a static route in the Simple Static Route field. Or, select Custom Static Route and then click Configure and perform the following steps:

      • In the Subnets section, click Add Item and then select IPv4 or IPv6 from the Version menu. Enter a prefix and prefix length for your subnet. Click Add Item to set more subnets.

      • In the Nexthop section, select a next-hop type from the Type menu. Select IPv4 or IPv6 from the Version menu in the Address section, and enter an IP address accordingly.

      • From the Network Interface menu, select a network interface or click Create new network interface to create and apply a new network interface.

      • In the Static Route Labels section, select supported labels from the Static Route Labels menu. You can select more than one from this list.

      • In the Attributes section, select supported attributes from the Attributes menu. You can select more than one from this list.

      • Click Apply to add the custom route.

  • Select Manage Static routes from the Manage Static Routes for Outside Network menu, and click Add Item for the Static route list field. Follow the same procedure as that of managing the static routes for inside network.

  • Click Apply.

Note: You can use the Add Item button to add multiple inside and outside networks.

  • Select Connect Global Networks from the Select Global Networks to Connect menu.

  • Click Add Item.

  • Select an option for the Select Network Connection Type menu.

  • Select a global network object from the displayed list or select Create new global vn option from the Global Virtual Network menu. If you select Create new virtual network, the global network creation form opens. Create a global network using the guided form and click Continue to apply the network to the global network connection configuration.

  • Click Apply to apply the TGW network configuration.

Step 5: Optionally, perform TGW security configuration.
  • In the Security Configuration section, click Configure.

  • From the Manage Forward Proxy menu, select a forwarding policy:

    • Disable Forward Proxy: This is the default option and will not forward traffic.

    • Enable Forward Proxy with Allow All Policy: This option forwards all traffic.

    • Enable Forward Proxy and Manage Policies: This option forwards traffic based on the policy you select or specify. In the List of Forward proxy policy menu, select an existing proxy or select Create new forward proxy policy to create a new policy.

  • From the Firewall Policy menu, select Active Firewall Policies.

  • Select an existing firewall policy or select Create new Firewall Policy. After creating the policy, click Continue to apply.

  • In the Manage East-West Service Policy section, select an existing policy from the Manage East-West Service Policy drop-down menu:

    • Disable East-West Service Policy: This is the default option and will not use a proxy for East-West traffic.

    • Enable East-West Service Policy: This option uses a proxy for East-West traffic. Click Add item to select an existing policy or select Create new service policy from the menu for a new policy.

    • Enable East-West traffic Proxy with Allow All Policy: This option sends all East-West traffic through a proxy for monitoring.

  • Click Apply.

Step 6: Optionally, setup geographical site information.
  • In the Software Configuration section, click Show Advanced Fields.

  • Enter the geographical address and/or latitude and longitude for the site location.

Step 7: Optionally, configure more advanced settings.
  • In the Advanced Configuration section, click Show Advanced Fields.

  • From the Logs Streaming menu, select Enable Logs Streaming and then select a log receiver or create a new log receiver.

  • From the Select Volterra Software Version menu, select an option.

  • From the Select Operating System Version menu, select an option.

  • Optionally, add a site to site tunnel IP and additional AWS tags.

Step 8: Optionally, configure direct connect choice.
  • In the Direct Connect Choice section, enable Show Advanced Fields.

  • From the Direct Connect Choice drop-down menu, select an option:

    • Disable Direct Connect: Default option.

    • Enable Direct Connect: Select to configure for the AWS TGW site.

  • Click Configure.

  • From the VIF Configuration drop-down menu, select an option for the Virtual Interface (VIF):

    • Hosted VIF mode: With this mode, F5 will provision an AWS Direct Connect Gateway and a Virtual Private Gateway. The hosted VIP you provide will be automatically associated and will set up BGP peering.

    • Standard VIF mode: With this mode, F5 will provision an AWS Direct Connect Gateway and a Virtual Private Gateway, a user associate VIP, and will set up BGP peering.

  • For the Hosted VIF mode option:

    • Click Add item.

    • Enter a list of VIF IDs.

    • Click Apply.

  • For the Standard VIF mode option:

    • Click Apply.
Step 9: Complete AWS TGW site object creation.
  • Click Save and Exit to complete creating the AWS TGW site object.

Figure: Complete Creating Object
Figure: Complete Creating Object

The Status field for the AWS TGW object displays Generated.


Deploy Site

Creating the AWS TGW site object in Console generates the Terraform parameters.

Note: Site upgrades may take up to 10 minutes per site node. Once site upgrade has completed, you must apply the Terraform parameters to site via Action menu on cloud site management page.

  • Navigate to the AWS TGW site object by clicking Manage > Site Management > AWS TGW Sites.

  • Find your AWS TGW site object and click Apply under the Actions column. The Status field for your AWS TGW site object changes to Apply Planning.

Note: Optionally, you can perform Terraform plan activity before deployment. Find your AWS TGW site object and click ... > Plan to start the action of Terraform plan. This creates the execution plan for Terraform.

  • Wait for the apply process to complete and the status to change to Applied.

Note: You can check the status for the apply action. Click ... > Terraform Parameters for your AWS TGW site object and then click the Apply Status tab.

  • Navigate to Sites > Sites List.

  • Verify status is Online. It takes a few minutes for the site to deploy and status to change to Online.

Note: When you update worker nodes for a site object, the Terraform Apply button is enabled. Click Apply.


Delete AWS TGW Site

Perform the following steps to delete the AWS TGW site:

  • Navigate to the AWS TGW site object by clicking Manage > Site Management > AWS TGW Sites.

  • Find your AWS TGW site object and click ... > Delete.

  • Click Delete in the confirmation window.

Note: Deleting the AWS TGW site object deletes the sites and nodes from the VPC and deletes the VPC. If the delete operation does not remove the object and returns any errors, check the errors from the status, fix the errors, and then re-attempt the delete operation. If the problem persists, contact technical support. You can check the status using the ... > Terraform Parameters > Apply status option.


Deploy Site Using Terraform

This chapter provides instructions on how to create a single-node or multi-node site on Amazon Elastic Compute Cloud (EC2) using a custom Amazon Machine Image (AMI) with Terraform.

Perform the following procedure to deploy a site using Terraform:

Step 1: Confirm Terraform is installed.

In a terminal, enter terraform version. If you need to install, follow the instructions at the official guide.

Step 2: Create API credentials file.

Log into Console and create an API 12 certificate file and then download it. Use the instructions at Credentials for more help.

Step 3: Create a new directory on your system to place files for deployment.

Create a new directory on your system to place files for deployment.

Step 4: Create the deployment file.
  • Create a file and name it main.tf file, and place it in the newly created directory.

  • Copy and paste the following information into the file:

terraform {
  required_version = ">= 0.13.1"
  required_providers {
    volterra = {
      source = "volterraedge/volterra"
    }
  }
}
variable "site_name" {}
variable "aws_access_key" {}
variable "b64_aws_secret_key" {}
variable "aws_region" {
  default = "us-east-2"
}
variable "aws_vpc_cidr" {
  default = "192.168.0.0/20"
}
variable "aws_az" {
  default = "us-east-2a"
}
variable "outside_subnet_cidr_block" {
  default = "192.168.0.0/25"
}
variable "workload_subnet_cidr_block" {
  default = "192.168.0.128/25"
}
resource "volterra_cloud_credentials" "aws_cred" {
  name      = format("%s-cred", var.site_name)
  namespace = "system"
  aws_secret_key {
    access_key = var.aws_access_key
    secret_key {
      clear_secret_info {
        url = format("string:///%s", var.b64_aws_secret_key)
      }
    }
  }
}
resource "volterra_aws_tgw_site" "site" {
  name      = var.site_name
  namespace = "system"
  aws_parameters {
    aws_certified_hw = "aws-byol-multi-nic-voltmesh"
    aws_region       = var.aws_region
    az_nodes {
      aws_az_name            = var.aws_az
      reserved_inside_subnet = true
      outside_subnet {
        subnet_param {
          ipv4 = var.outside_subnet_cidr_block
        }
      }
      workload_subnet {
        subnet_param {
          ipv4 = var.workload_subnet_cidr_block
        }
      }
    }
    aws_cred {
      name      = volterra_cloud_credentials.aws_cred.name
      namespace = "system"
    }
    instance_type = "t3.xlarge"
    new_vpc {
      name_tag     = var.site_name
      primary_ipv4 = var.aws_vpc_cidr
    }
    new_tgw {
      system_generated = true
    }
  }
  lifecycle {
    ignore_changes = [labels]
  }
}
resource "volterra_tf_params_action" "apply_aws_vpc" {
  site_name        = volterra_aws_tgw_site.site.name
  site_kind        = "aws_tgw_site"
  action           = "apply"
  wait_for_action  = true
  ignore_on_update = true
}
  • Open the file and configure any necessary fields. The configuration above is an example. You can change the parameters for your particular setup.

  • Save the changes and then close the file.

Step 5: Create file for variables.
  • In the same directory, create another file for variables and name it terraform.tfvars.

  • Create and assign the following variables:

    • For your site name, type a name within double quotes: site_name = "<site-name>"

    • For the AWS region, type the name within double quotes: aws_region = "<region>"

    • For the AWS region subtype, type the name within double quotes: aws_az = "<region-subtype>"

site_name = "<site-name>"
aws_region = "<region>"
aws_az = "<region-subtype>"
Step 6: Create and export variables for credentials and secret keys.
  • In the terminal, create and export the following variables:

    • Create this variable and assign it your API credentials password: export VES_P12_PASSWORD=<credential password>

    • Create this variable and assign it the path to the API credential file previously created and downloaded from Console: export VOLT_API_P12_FILE=<path to your local p12 file>

    • Create this variable and assign it the URL for your tenant. For example: export VOLT_API_URL=https://example.console.ves.volterra.io/api

    • Create this variable and assign it your AWS secret key that has been encoded with Base64: export TF_VAR_b64_aws_secret_key=<base64 encoded value>

    • Create this variable and assign it your AWS access key: export TF_VAR_aws_access_key=<access key>

Note: You can also create and save these variables in the terraform.tfvars file. However, this may pose a security risk. Use caution when working with your credentials and secret keys.

export VES_P12_PASSWORD=<credential password>
export VOLT_API_P12_FILE=<path to your local p12 file>
export VOLT_API_URL=https://example.console.ves.volterra.io/api
export TF_VAR_b64_aws_secret_key=<base64 encoded value>
export TF_VAR_aws_access_key=<access key>
Step 7: Initiate Terraform process.

Enter terraform init.

Step 8: Apply Terraform process.
  • Enter terraform apply.

  • If prompted for the access key and secret key encoded in Base64, enter both.

  • Enter yes to confirm. This may take a few minutes to complete. After the process is complete, the output will state Apply complete!.

  • In Console, navigate to the list of sites and confirm the site was applied.


Destroy Site

Perform the following procedure to destroy the site using Terraform:

  • Enter terraform destroy.

  • If prompted for the access key and secret key encoded in Base64, enter both.

  • Enter yes to confirm. This may take a few minutes to complete. After the process is complete, the output will state Destroy complete!.

  • In Console, navigate to the list of sites and confirm the site was destroyed.


Concepts


API References