NSX-T Architecture in vSphere with Tanzu – Part 5 – Edge Node Networking

Welcome back!!!

We are at the final part of the blog series on NSX-T architecture in vSphere with Tanzu. In this article, we will take a look at the edge node networking design on Workload clusters with 2 pnic and 4-pnic host network adapters.

If you missed the previous parts, you can read it here:

Part 1 : https://vxplanet.com/2021/01/01/nsx-t-architecture-in-vsphere-with-tanzu-part-1-per-tkg-tier1-vs-per-namespace-tier1/

Part 2 : https://vxplanet.com/2021/01/05/nsx-t-architecture-in-vsphere-with-tanzu-part-2-multisupervisor-shared-t0-vs-dedicated-t0/

Part 3 : https://vxplanet.com/2021/02/04/nsx-t-architecture-in-vsphere-with-tanzu-part-3-dedicated-tier-1-edge-clusters/

Part 4 : https://vxplanet.com/2021/02/12/nsx-t-architecture-in-vsphere-with-tanzu-part-4-proxy-arp-gateways/

The below designs are applicable for edges nodes that are co-located with the Workload cluster (where workload management is enabled). For edge nodes on dedicated vsphere edge clusters or on shared ‘management and edge’ vsphere clusters, the design may vary as the host VDS do not participate in overlay networking and as such they don’t have TEP interfaces.

Let’s get started:

Edge networking on 2-pnic workload cluster hosts

The below design represents single-NVDS multi-TEP edge nodes attached to a converged VDS (c-VDS) on Workload ESXi hosts with 2-pnic network adapters. All the previous parts of this blog series used the below design.

This is called a Converged management and Workload design. This design has a single VDS (c-DVS) that handles both the infrastructure traffic and the workload traffic. With traffic steering via teaming policies, we achieve deterministic traffic flows northbound. Note that bonding techniques like LACP is not used.

The edge design is single-NVDS multi-TEP which is a consistent design across both VM and baremetal edge form factors.
Edge TEP VLAN tags are assigned by the edge uplink profile. eBGP peering VLAN tags are applied by the VLAN logical segments.
Named teaming policies are used to steer BGP peering across different edge uplinks. In this way, the BGP peering with the Leaf switches is deterministic. Peering VLAN X steers via uplink1 to Leaf 1 and VLAN Y steers via uplink 2 to Leaf 2.
Edge management (eth0) attaches to the c-DVS management Port group.
Edge uplink interfaces (fp-eth0 and fp-eth1) attach to trunk port groups on the converged VDS.
All the VLAN tags for TEP and BGP peering are applied at the edge NVDS
Each c-DVS trunk port group has a failover teaming policy and each trunk port group has their active uplinks over separate physical host pnics.
The infrastructure traffic (vSAN, vMotion, Replication etc) are traffic steered northbound using teaming policies to achieve a deterministic traffic flow.
The c-VDS also has host TEP interfaces that is bound to the pnics. The NSX-T uplink profile normally used is “Load balance Source” that gives us a multi-TEP design (Active-Active) for the Workload hosts.
The Host TEP and Edge TEP VLANs can be either on the same VLAN (in newer releases of NSX-T) or on different VLANs. (Personally I prefer using separate VLANs so that when we do a traceflow we would be able to correctly identify the transport nodes based on the TEP subnet)
The c-DVS uplinks (p-nics) are attached to separate Leaf Switches northbound (Standalone or VPC/VLT)

Edge networking on 4-pnic workload cluster hosts

The below sketch represents the edge and host networking design for workload clusters with 4-pnic adapters. In this design we have two host VDS with 2-pnics each:

Management & Infrastructure VDS – This handles all the management (ESXi host and edge management) and the infrastructure (vSAN, vMotion, Replication, iSCSI, FT etc) traffic
Workload VDS – This handles all the TEP (Host TEP and Edge TEP) and workload (overlay) traffic

This design is often used for environments that require isolation of workload traffic from the management and infrastructure traffic.

All notes mentioned for the above 2-pnic design applies to the 4-pnic design as well but with the below changes:

Edge management (eth0) attaches to the management Port group on the ‘Management & Infrastructure VDS’.
Edge uplink interfaces (fp-eth0 and fp-eth1) attach to trunk port groups on the Workload VDS
The pnics on each VDS attach to separate Leaf Switches northbound.

Choosing the VDS for Edge Transport node configuration

Correct VDS need to be chosen while configuring the Edges in a 4-pnic host networking design. This is where we select the edge management port group while deploying edge nodes.

This is where we select the Workload VDS (Trunk port groups) for edge TEP and uplink interfaces.

Choosing the VDS for Workload Management configuration

We have the below Management and Workload VDS in the Workload cluster.

Workload management configuration workflow has the VDS selection option where the Workload VDS (the one prepared for NSX-T) needs to be selected. The workflow picks the edge clusters configured on the same Overlay transport zone as the Workload VDS. We could choose either a shared edge cluster for T0 & T1 or a dedicated edge cluster for T1 based on the requirement.