NSX-T Edges – Baremetal vs VM Comparison

NSX-T Edge Nodes come in two form factors – VM and Baremetal both leveraging Intel DPDK (Data Plane Development Kit) acceleration for the transport and Uplink networks. Deciding which form factor to use depends upon on our use case requirements and it is good to understand the workload behavior and virtualized services requirement properly before finalizing the Edge deployment form factor. One we have deployed the logical routers (SR Components) and the centralized components (NAT, Edge firewalls, Loadbalancers etc) on an Edge cluster (with VM Form factor for example), it’s really complicated to migrate the services to another Edge cluster with a different form factor (Eg: baremetal). There are different ways to handle Edge VM bottlenecks like maintaining dedicated Edge Clusters – one for N-S Routing and the other for Centralized Services like NAT, Edge Firewalling, VPN etc but that is beyond the scope of our discussion.

This article is about a comparison between the Baremetal and VM form factor for the Edge nodes. Some key areas to look at before deciding the Edge deployment type are:

  • North bound traffic pattern
  • Any L2 Bridging requirements
  • External storage access requirements for the overlay VMs
  • Number of NSX services to be deployed (NAT, Load balancers, Edge firewalls etc)
  • Future scalability of the platform
  • SSL Offloading on the NSX Load balancers
  • Edge Performance
  • Plans to have a dedicated Edge cluster?
  • Networking / Design simplicity
  • Cost

I hope this would give you a good understanding before choosing the right Edge deployment type for your workloads. Let’s get started.

N-VDS Networking

VM Form factor : Edge VMs are deployed with 4 vNICs – one for management, another for Transport network (Geneve encapsulation), and last two for Uplinks. All these vNICs attach to Port groups on the vSphere DvSwitch on the ESXi host. Each Edge node will have Overlay and Uplink N-VDS (when configured as Transport node) which attaches to the vNICs which infact is attached to the DvSwitch Port Groups. This could be realized as a Nested networking scenario. When compared to Baremetal networking, I think this adds a bit to networking complexity. This below sketch for the Edge VM networking depicts this scenario.

 

EdgeNetworking

Baremetal Form factor : As per recommendation, the Baremetal edges are also deployed with 4 pNICs but here the Overlay and Uplink N-VDS attaches directly to the pNICS. No Nested networking in this case. Networking complexity is simplified. This is how the Baremetal Edge networking looks like:

BMEdge

[Click here for HQ Image]

DPDK Acceleration

Both Baremetal and VM Edges support DPDK acceleration for the Transport and Uplink networks.

Deployment Mode

VM Form factor: In most cases, the deployment mode is Active-Active to take advantage of ECMP and higher throughput. In Active-Active mode, stateful NSX services are not supported (For eg: you can have only Reflexive NAT on the T0 router). This becomes like a decision choice to us on whether we need better throughput or support stateful services. If we need both, we need to deploy the VM Edges in Active-Passive mode but it can’t guarantee the same throughput levels when compared with Baremetal Edges.

Baremetal Form factor : With NSX-T 2.4, 25G NIC support is available for the baremetal Edges. We can deploy the Edges in Active-Passive and take advantage of stateful services. A single Edge (1x25G uplink or 2x25G uplink in LACP) could satisfy the North bound traffic requirements and the second Edge can take over when the first instance fails.

Networking Complications

VM Form factor : It is recommended to attach VM Edges to vSphere DVS/VSS and not to the N-VDS. Based on the design, the Edge VMs can sit on a dedicated Edge cluster, on a collapsed Management & Edge cluster or on a collapsed Compute & Edge cluster. We can deploy the Edges either on the infrastructure DVS (having management, vMotion, vSAN and other infrastructure traffic) or on a separate DVS with dedicated uplinks. When deploying on the infrastructure DVS, we need to look at options for bandwidth guarantees like QoS for the Edge VMs. When using dedicated DVS, we need additional pNICs (min. of 2 for redundancy) . Whichever option we choose, that need to be identical across the ESXi hosts in the cluster to take advantage of vMotion for the Edge VMs.

Baremetal Form factor: Edges get dedicated high bandwidth pNICs for the Overlay and Uplink N-VDS.

Cost

VM Form factor : Can leverage the existing ESXi clusters for deployment. This can be a dedicated Edge cluster or on a collapsed Edge and Management cluster.

Baremetal Form factor: Adds to the Capex as it requires a dedicated hardware and that listed in the compatibility matrix.

BFD and Failover Convergence

VM Form factor : When deployed with BFD, failover and convergence takes approximately 3 seconds. 

Baremetal Form factor : During a failover, the other edge node can take over in less than a second (~750 ms). This is true for convergence as well. 

Design Simplicity

Having dedicated Baremetal Edges adds to a simpler Edge cluster design than having a dedicated vSphere Edge cluster with VM Edge nodes.

vSphere Upgrades and maintenance

VM Form factor : Some admin actions like vMotioning the Edges to other ESXi hosts for shorter maintenance and / or vSAN Storage evacuation depending up on the outage window. This is more of a vSphere admin activity but just to point out that there is a dependency.

Baremetal Form factor : vSphere Upgrades and maintenance can be performed independently. NSX-T Baremetal Edges are outside of the vSphere platform and there is no direct vSphere dependency.

Edges for KVM hosts

If the NSX-T platform is only KVM based, we have the option to deploy only Baremetal Edges. Edges in VM form factor cannot be deployed on KVM hosts.

Overlay-VLAN L2 Bridging

It is recommended to use Baremetal Edge clusters for Overlay-VLAN L2 bridging to achieve better throughput for the bridged network by leveraging DPDK acceleration. Having a bridge instance on the VM Edges for high data transfers could lead to performance bottlenecks.

NSX Loadbalancer SSL Offloading

It is recommended to use Baremetal Edges for NSX Loadbalancer SSL offloading as it supports higher TPS. SSL Offloading is resource intensive and VM Edges would possibly create performance issues.

Upper limits for NSX-T Services

Baremetal Edges got higher upper limits than VM form factor for the NSX Services like Loadbalancers, VPN, NAT etc

NSX-T Loadbalancer instances

VM Form factor : Per Large Edge VM Form factor, it supports 1 Large Loadbalancer Instance or 4 Medium Loadbalancer Instances or 40 Small Loadbalancer Instances

Baremetal Form factor : Per Baremetal Edge instance, it supports much higher numbers than VM Edges. It supports 18 Large Loadbalancer Instances or 75 Small Loadbalancer Instances or 750 Small Loadbalancer Instances.

NSX-T Loadbalancer Pool members

Baremetal Edges support as large as 30,000 pool members when compared to 7,500 on VM Edges with Large Form factor.

NFV Workloads

Recommended to use Baremetal Edges as per VMware Reference Architecture.

Conclusion

I hope there could be more comparison data, but this is all that I have for now. To conclude, Baremetal Edges provide better performance with sub-second convergence, faster failover, and greater throughput. Personally my suggestion would be to use Baremetal edges considering the future scalability of the platform and to support increasing NSX-T services. Workload nature and traffic patterns could vary over the time and could not be predicted accurately, this might put Edges in VM Form factor to end up in performance bottlenecks at a future point of time. If that is the case, we will have a tough time migrating services from Edge VM cluster to Edge Baremetal cluster. 

I hope this post was informative. 

Thanks for reading

 

nsxrun

 

2 thoughts on “NSX-T Edges – Baremetal vs VM Comparison

  1. Hi Harikrishnan T
    Your blog is not exactly written.
    -Please update your diagram. VTEP stands for VxLAN. But here we talk about a GENEVE Tunnel End Point (TEP)
    -We have support of deployments with only a single N-VDS, 3 N-VDS are typically used so far, but a single N-VDS is supported
    -Bare metal can have more than 4 DPDK interfaces, the current limitation is 8
    -Your bare metal design is something what we from VMware dont recommend when we have 4 interfaces. We use multi-TEP for the GENEVE traffic, so that we can achieve redundandcy when a single links fails, instead failover to the standby edge.
    -in the bare metal design we do the dot1q tagging at the segment level, this is unclear in your diagram.
    -we support as well 40G interface for bare metal, not only 25G
    -this statement is wrong: “It is recommended to attach VM Edges to vSphere DVS/VSS and not to the N-VDS”, just think about how VCF is deployed and how we do that there.
    -today (June 2019) we dont support vMotion on vm-based edge node. By the end of 2019, this should be fully supported

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s