NSX-T Edges have flexible deployment options based on the host networking that is used. We can deploy NSX-T Edges on Distributed vSwitches (which is managed by vCenter Server) or on host N-VDS (which is managed by NSX-T). NSX-T Edges deployment and configuration on DvSwitches has been covered in my previous post, you can find it here:
Whether to use DvSwitch or N-VDS or both depends upon the number of pNICs available on the host networking. If the vSphere environment is based on a 2 pNIC host configuration, we could migrate all the host networking to N-VDS and disassociate the DVS from the ESXi hosts. I have covered the migration from DVS to N-VDS in my earlier article, you can find it here:
In this article, we will walk though the steps to deploy and configure the NSX-T Edges on N-VDS host networking. This approach is suitable if you are deploying on a 2 pNIC host platform. This also works on an NSX-T based Workload domain in VMware Cloud Foundation. Let’s get started.
Environment Details
- 4 x Dell EMC PE R640 nodes as ESXi hosts
- Collapsed Management, Compute and Edge vSphere 6.7U2 vSAN cluster
- 2 X 25G host networking connected to Dell Networking L3 ToR switches in VLT
- NSX-T 2.4.1 with a 3 node management cluster. All the 4 ESXi hosts are configured as NSX-T Transport nodes.
- Host networking is completely decoupled from vCenter DVS and migrated to N-VDS
Current State
This is the current state of the NSX-T platform. The hosts are configured with 2 Transport Zones – One Overlay and another VLAN, both leveraging the same N-VDS.
The host Uplink Profile configures the host TEP VLAN and the teaming policies. We use VLAN 40 for TEP encapsulation and Load balancing as the Teaming policy.
We create 4 VLAN Segments on the N-VDS which is used for host networking (management, vSAN, vMotion, VMNetwork etc)
The N-VDS is visible to all the ESXi hosts and the host networking (vmk ports, virtual machines etc) are migrated to N-VDS. vCenter DVS is disassociated from the ESXi hosts.
Creating VLAN Transport Zones for the Edge Uplinks
We have to create VLAN Transport Zones based on the Uplink Interfaces that we use on the Edge nodes. We have two uplink VLANs for the Edges – VLAN 60 and VLAN 70, so we create two Transport zones. These Transport zones are not a part of the ESXi Transport nodes, they are only available to the Edge nodes. Each Uplink Transport zone will have a separate N-VDS on the Edges.
Creating Logical Segments for Edge TEP and Uplinks
Since NSX-T Edges are deployed on host N-VDS, the VLAN for the Edge TEP should be different from the ESXi host TEP. The host TEPs are on VLAN 40, so we choose the Edge TEPs to be in VLAN 80. We create 3 Logical Segments for the Edge Connectivity to the host N-VDS.
- One for the Edge TEP – VLAN 80
- Another for the Edge Uplink 1 – VLAN 60
- Another for the Edge Uplink 2 – VLAN 70
These VLAN Logical Segments should now reflect on the host N-VDS.
Creating the Edge Uplink Profile
We could use the pre-created single nic Edge uplink profile. Make sure NOT to put VLAN tag on the Edge Uplink Profile. The Tag for the Edge TEP is applied by the host N-VDS.
Deploying the first Edge Node
We will use the Edge ova file to deploy the Edge nodes. I’ve already downloaded the ova from my VMware account.
I’ve used Medium form factor for the deployment.
Select the datastore. Since our cluster is vSAN enabled, let’s place it on the vSAN datastore.
Configure the Networking. Edges are deployed with 4 vnics. We have to map the vnics to the appropriate Logical segments on the host N-VDS. Just in case, we use only one edge uplink, disconnect the 4th vnic. This is how the networks are attached:
- Network 0 → Management VLAN Logical segment (VLAN 10)
- Network 1 → Edge TEP (VLAN 80)
- Network 2 → Uplink 1 Logical Segment (VLAN 60)
- Network 2 → Uplink 2 Logical Segment (VLAN 70)
Configure the Management network, passwords, DNS, NTP and other basic settings.
Review the settings and click Finish to start the deployment of the first Edge node VM.
Power on the Edge VM and wait for it to initialize.
SSH to the Edge VM and perform basic connectivity checks.
Joining Edge to the NSX-T Management Plane
Generate the Certificate Thumbprint from the NSX-T manager.
Join Edge to the management plane. [Click here for HQ Image]
The Edge should now appear under the “Edge Transport Nodes” section in the NSX-T Manager UI.[Click here for HQ Image]
Configuring the Edge as an NSX-T Transport Node
The NSX-T Edge VM will be a part of minimum 2 transport zones – one will be the overlay Transport zone and the other one will be the Uplink VLAN Transport zone. In our case, we have 2 Edge Uplinks – each on separate VLANs and with separate ports, hence we configure the Edges as a Transport node for the 3 Transport zones.
We have to configure 3 N-VDS here:
- Overlay N-VDS is where the Edge TEP is configured. It’s uplink maps to fp-eth0 (This is the second adapter on the Edge VM)
- Uplink1 VLAN N-VDS – It maps to the third interface fp-eth1
- Uplink2 VLAN N-VDS – It maps to the forth interface fp-eth2
We could either use a Static entry or use IP Pool for the Edge TEPs. We can create an IP Pool directly from this configuration box.
Once configured, verify the Edge transport node status.
Deploying the second NSX-T Edge and configuring as a Transport node
The procedure is exactly similar to the above except that management IP of 192.168.10.172/24 is used for the second node.
Once deployed and configured as a transport node, both Edges should display a healthy status.
Configuring the Edge Cluster
We need to create an Edge cluster and add both Edge nodes as its members.
Creating VLAN Logical Segments for Tier 0 Gateway Uplinks
We need two VLAN Logical segments created on the Edge Uplink Transport Zones, so that the Tier 0 Gateway can attach to the Edge Uplinks.
Note that we need not have to put a VLAN tag here, as the tagging is applied at the host N-VDS level ie, on the VLAN Uplink Logical Segments that we created earlier.
Creating Tier 0 Gateway
We will now create a Tier 0 gateway on the Edge Cluster.
Configuring Tier 0 Gateway Uplinks
We will create two uplinks for the T0 gateway – One on VLAN 60 via the first Edge node and second on VLAN 70 via the second Edge node.
Confirm that the Uplink interfaces are initialized and are up.
Validating the External Connectivity
We will ssh to the Edge nodes and perform a connectivity test to the external ToR switches.
SUCCESS!!! Tier 0 gateway can establish communication to the ToRs via Edge node 1 over VLAN 60. Lets try the other Edge node.
SUCCESS!!! Tier 0 gateway can establish communication to the ToRs via Edge node 2 over VLAN 70.
We are now good to deploy the overlay logical segments, Tenant T1 routers and BGP peering with the L3 ToRs.
I hope the post was informative. Thanks for reading
Thanks, this is very useful!
Thanks Ronald 🙂 I followed your site as well, lots of information there.
Hey,
first thanks for this great article. 🙂
I have 2 questions:
1) Why edges overlay VLAN and hosts overlay VLAN can’t be the same?
2) Why do we need to create 2 NVDS for edge uplinks?
thanks.
Ishai.
Thanks Ishai and sorry I couldn’t get back to you last week as I was on a festive occasion. To answer your questions:
1. This is a special case. Here, the Edges and the Compute Transport nodes (on which Edge VM is deployed) are part of the same Transport zone and connected to same N-VDS. We can visualize the Edge VM networking as a nested N-VDS scenario. We would need the Edges and Host TEPs on different VLANs to avoid a possible encapsulation / de-capsulation issue in few scenarios.
2. As of now, Edge VMs don’t support a single N-VDS design. We use 2 NVDS for the Edge uplinks to eBGP peer with the ToR switches over over two different VLANs. A single N-VDS would also work, but this is what the VMware Validated Design recommends. With the upcoming NSX-T 2.5 version, edge networking is greatly simplified. We could have a single NVDS Multi-TEP designs for Edge VMs. Please see my tweet for the VVD design:
https://twitter.com/hari5611/status/1148603256186331137
Thanks
Hari
Thanks Hari, very helpful article!
If you’re always configuring T0-UplinkA on edge01 and T0-UplinkB on edge02, then do both edge nodes need to belong to both TZ?
Could you have edge01 only on TZ-Edge-Uplink01 (and Overlay TZ of course) and edge02 only on TZ-Edge-Uplink02 (and overlay), so only two N-VDS on each node (3 vnic per VM)?
Thanks Manu
If the T0 LR has only 2 Uplinks each sitting on separate edges, then as you said we don’t need edges to be on both VLAN TZ. But in Production we deploy T0 with min of 2 uplinks per edge node for redundancy. So we require edges to be in min of 2 VLAN TZ. I will add a comment to the article to avoid the confusion.
Starting from v2.5 onwards, NSX-T edges support a single NVDS Multi-TEP design which is the recommended design going forward.
https://vxplanet.com/2019/09/23/nsx-t-single-nvds-multi-tep-edge-vm-deployment-configuration-on-vsphere-dvs/
Excellent guide for a 2-pNIC deployment of NSX. Translates pretty well in NSX 3.0 as well. Thanks very much, this is probably the best guide I have found for this deployment.
Thank you Luke. For NSX-T 3.0, it is recommended to use the Converged VDS instead of NVDS as only c-VDS is supported for Workload management in vSphere 7.0. This keeps the configuration simpler. and also addresses an opaque port group issue we had with NVDS.
Hi Hari,
Thanks and For the Vlogs. But we are unable to reach the NSX Manager from the NSX Edge Node. NSX Manager is connected to VM Network . Could you tell us what we missing here.
Regards,
vignesh
Hi Vignesh, if we follow the VVD way, NSX-T Manager, vCenter Server and NSX-T Edge management networks reside on the ESXi management network. Not a hard requirement though, but this will help us avoid any L3 routing related issues.
Can you share me your topology, I can take a look.
Thanks
Hari
Hi Hari, thanks for this article. it’s superuseful. I have a doubt. I am using in the lab NSX-T 3.0 and i can’t see on vCenter the VLAN segments created using NSX. Furthermore I can’t see the N-VDS on vCenter. Any clue why?
Hi Stefano, Thanks for the feedback
Which is the vCenter version that you are using? From vSphere 7.0, you can use the converged VDS mode, which prepares the vSphere DVS for NSX-T, and depending on the host switch options, you can decide how and where the VLAN segments need to be defined.
https://serveritpro.files.wordpress.com/2021/01/45.png
Thanks
Hari