NSX-T Edge Nodes come in two form factors – VM and Baremetal both leveraging Intel DPDK (Data Plane Development Kit) acceleration for the transport and Uplink networks. Deciding which form factor to use depends upon on our use case requirements and it is good to understand the workload behavior and virtualized services requirement properly before finalizing the Edge deployment form factor. One we have deployed the logical routers (SR Components) and the centralized components (NAT, Edge firewalls, Loadbalancers etc) on an Edge cluster (with VM Form factor for example), it’s really complicated to migrate the services to another Edge cluster with a different form factor (Eg: baremetal). There are different ways to handle Edge VM bottlenecks like maintaining dedicated Edge Clusters – one for N-S Routing and the other for Centralized Services like NAT, Edge Firewalling, VPN etc but that is beyond the scope of our discussion.
This article is about a comparison between the Baremetal and VM form factor for the Edge nodes. Some key areas to look at before deciding the Edge deployment type are:
- North bound traffic pattern
- Any L2 Bridging requirements
- External storage access requirements for the overlay VMs
- Number of NSX services to be deployed (NAT, Load balancers, Edge firewalls etc)
- Future scalability of the platform
- SSL Offloading on the NSX Load balancers
- Edge Performance
- Plans to have a dedicated Edge cluster?
- Networking / Design simplicity
I hope this would give you a good understanding before choosing the right Edge deployment type for your workloads. Let’s get started.
VM Form factor : Edge VMs are deployed with 4 vNICs – one for management, another for Transport network (Geneve encapsulation), and last two for Uplinks. All these vNICs attach to Port groups on the vSphere DvSwitch on the ESXi host. Each Edge node will have Overlay and Uplink N-VDS (when configured as Transport node) which attaches to the vNICs which infact is attached to the DvSwitch Port Groups. This could be realized as a Nested networking scenario. When compared to Baremetal networking, I think this adds a bit to networking complexity. This below sketch for the Edge VM networking depicts this scenario.
Baremetal Form factor : As per recommendation, the Baremetal edges are also deployed with 4 pNICs but here the Overlay and Uplink N-VDS attaches directly to the pNICS. No Nested networking in this case. Networking complexity is simplified. This is how the Baremetal Edge networking looks like:
Both Baremetal and VM Edges support DPDK acceleration for the Transport and Uplink networks.
VM Form factor: In most cases, the deployment mode is Active-Active to take advantage of ECMP and higher throughput. In Active-Active mode, stateful NSX services are not supported (For eg: you can have only Reflexive NAT on the T0 router). This becomes like a decision choice to us on whether we need better throughput or support stateful services. If we need both, we need to deploy the VM Edges in Active-Passive mode but it can’t guarantee the same throughput levels when compared with Baremetal Edges.
Baremetal Form factor : With NSX-T 2.4, 25G NIC support is available for the baremetal Edges. We can deploy the Edges in Active-Passive and take advantage of stateful services. A single Edge (1x25G uplink or 2x25G uplink in LACP) could satisfy the North bound traffic requirements and the second Edge can take over when the first instance fails.
VM Form factor : It is recommended to attach VM Edges to vSphere DVS/VSS and not to the N-VDS. Based on the design, the Edge VMs can sit on a dedicated Edge cluster, on a collapsed Management & Edge cluster or on a collapsed Compute & Edge cluster. We can deploy the Edges either on the infrastructure DVS (having management, vMotion, vSAN and other infrastructure traffic) or on a separate DVS with dedicated uplinks. When deploying on the infrastructure DVS, we need to look at options for bandwidth guarantees like QoS for the Edge VMs. When using dedicated DVS, we need additional pNICs (min. of 2 for redundancy) . Whichever option we choose, that need to be identical across the ESXi hosts in the cluster to take advantage of vMotion for the Edge VMs.
Baremetal Form factor: Edges get dedicated high bandwidth pNICs for the Overlay and Uplink N-VDS.
VM Form factor : Can leverage the existing ESXi clusters for deployment. This can be a dedicated Edge cluster or on a collapsed Edge and Management cluster.
Baremetal Form factor: Adds to the Capex as it requires a dedicated hardware and that listed in the compatibility matrix.
BFD and Failover Convergence
VM Form factor : When deployed with BFD, failover and convergence takes approximately 3 seconds.
Baremetal Form factor : During a failover, the other edge node can take over in less than a second (~750 ms). This is true for convergence as well.
Having dedicated Baremetal Edges adds to a simpler Edge cluster design than having a dedicated vSphere Edge cluster with VM Edge nodes.
vSphere Upgrades and maintenance
VM Form factor : Some admin actions like vMotioning the Edges to other ESXi hosts for shorter maintenance and / or vSAN Storage evacuation depending up on the outage window. This is more of a vSphere admin activity but just to point out that there is a dependency.
Baremetal Form factor : vSphere Upgrades and maintenance can be performed independently. NSX-T Baremetal Edges are outside of the vSphere platform and there is no direct vSphere dependency.
Edges for KVM hosts
If the NSX-T platform is only KVM based, we have the option to deploy only Baremetal Edges. Edges in VM form factor cannot be deployed on KVM hosts.
Overlay-VLAN L2 Bridging
It is recommended to use Baremetal Edge clusters for Overlay-VLAN L2 bridging to achieve better throughput for the bridged network by leveraging DPDK acceleration. Having a bridge instance on the VM Edges for high data transfers could lead to performance bottlenecks.
NSX Loadbalancer SSL Offloading
It is recommended to use Baremetal Edges for NSX Loadbalancer SSL offloading as it supports higher TPS. SSL Offloading is resource intensive and VM Edges would possibly create performance issues.
Upper limits for NSX-T Services
Baremetal Edges got higher upper limits than VM form factor for the NSX Services like Loadbalancers, VPN, NAT etc
NSX-T Loadbalancer instances
VM Form factor : Per Large Edge VM Form factor, it supports 1 Large Loadbalancer Instance or 4 Medium Loadbalancer Instances or 40 Small Loadbalancer Instances
Baremetal Form factor : Per Baremetal Edge instance, it supports much higher numbers than VM Edges. It supports 18 Large Loadbalancer Instances or 75 Small Loadbalancer Instances or 750 Small Loadbalancer Instances.
NSX-T Loadbalancer Pool members
Baremetal Edges support as large as 30,000 pool members when compared to 7,500 on VM Edges with Large Form factor.
Recommended to use Baremetal Edges as per VMware Reference Architecture.
I hope there could be more comparison data, but this is all that I have for now. To conclude, Baremetal Edges provide better performance with sub-second convergence, faster failover, and greater throughput. Personally my suggestion would be to use Baremetal edges considering the future scalability of the platform and to support increasing NSX-T services. Workload nature and traffic patterns could vary over the time and could not be predicted accurately, this might put Edges in VM Form factor to end up in performance bottlenecks at a future point of time. If that is the case, we will have a tough time migrating services from Edge VM cluster to Edge Baremetal cluster.
I hope this post was informative.
Thanks for reading