A Tier 1 Gateway requires an SR Construct whenever a Stateful or non-Distributed services are deployed on it. The SR construct is always instantiated on the Edge nodes in an Active-Passive mode for the Tier 1 Gateway. The presence of an SR Construct can influence the East-West routing, North-South routing and ECMP depending on the way in which Edge Clusters are leveraged. We could either:
- Leverage the same Edge Cluster for both Tier1 and Tier0 SR Constructs
- Have dedicated Edge Clusters for Tier1 and Tier0 SR Constructs
In this 3-part blog series, we will walk through three different scenarios for the placement of Tier 1 Gateway and it’s influence on North-South Routing and ECMP. Each part covers one T1 deployment scenario. We will not cover the effect on East-West routing here, and will deal that as a separate article.
Part 1 -> Tier 1 Gateway without an SR Construct
Part 2 -> Tier 1 Gateway with an SR Construct on a Shared T0-T1 Edge Cluster
Part 3 -> Tier 1 Gateway with an SR Construct on a dedicated Edge Cluster.
Let’s get started.
Tier 1 Gateway without an SR Construct : North-South Routing and ECMP
The below sketch shows the traffic pattern for North bound flow from a Segment connected to the T1 Gateway.
[Click here for HQ Image] Sketch1
And the below sketch shows the traffic pattern for a South bound flow from the Leaf Switches to the Segment connected to the T1 Gateway.
[Click here for HQ Image] Sketch2
It is important to note that Routing happens closer to the source of traffic (as it is clear from the sketches). For Northbound, it would be the Compute Transport nodes and for Southbound it would be the Edge nodes. As such, the return path for a flow can be on a different route compared to the source flow.
Topology Details
For this article, we will use a two-tiered architecture with:
- Tier0 Gateway deployed in Active-Active mode with 4 uplinks on an Edge Cluster with 2 Edge nodes.
- Each Edge node hosts two uplinks – one on VLAN 60 and the other on VLAN 70
- The T0 Gateway establishes eBGP peering with two Dell EMC S 5048-ON Leaf switches. We will have 4 eBGP peerings in total
- T0 Gateway uses eBGP ECMP on the uplinks and Inter-SR Routing between the SR nodes.
- A single Tier 1 Gateway deployed without an SR Construct and uplinked to the Tier 0 Gateway
- A Logical Segment on 172.21.0.0/24 attached to the Tier 1 Gateway
We will not cover the Topology configuration here, as this has already been covered in my previous articles.
Northbound Traffic behavior
As shown in Sketch1, the Tier 1 DR Construct has two interfaces – One attached to the Logical Segment (over auto-plumbed VNI 71701) and the other attached to the Tier 0 DR Construct (over auto-plumbed VNI 71702). The DR Constructs are available on all the transport nodes in the Transport zone. So all the Northbound routing for workloads from Segment 172.21.0.0/24 on a Compute Transport node happens locally until the point where the SR Construct is involved in the path. At that point, traffic is tunneled to the relevant Edge node for SR Lookup and further upstream routing. Note that the VNI can be different based on deployments.
Let’s login to an ESXi Transport node and see the Interfaces and Next-hops
These are the T1 DR Interfaces
This is the Nex-hop to the T1 DR that points to T0 DR that is local to this ESXi Transport node.
Tier 0 DR Construct has two interfaces – One attached to the T1 DR Construct (over auto-plumbed VNI 71702) and the other attached to the Tier 0 SR Constructs (over auto-plumbed VNI 71703).
The T0 DR has two Default Routes on the Compute Transport node that points to the two SR Constructs on the Edge nodes to achieve ECMP between T0 DR & SR. Note that this T0 DR-SR ECMP is available only on the Compute Transport Nodes and not on the Edge nodes (with exceptions, as explained in upcoming Part 3).
This T0 DR-SR ECMP is scalable. If we introduce additional T0 SR Constructs on the T0 Gateway, the T0 DR forwarding table on the Compute Transport nodes are updated with default routes to the new T0 SR Constructs, thereby increasing the number of ECMP paths from the Compute transport nodes.
Once the traffic reaches a T0 SR Construct, it is routed to the external environment using its Forwarding table. Since ECMP is enabled for the eBGP Uplinks on the T0 SR Constructs, we get a total of 2 x 2 = 4 ECMP paths for Northbound routing on this topology as shown earlier.
Let’s try a Traceflow to see the Flows in action:
Our source is a VM attached to the Logical Segment (172.21.0.0/24) on the T1 Gateway. Destination is a machine outside the NSX-T Environment.
Note that all the DR Lookups (T1 & T0) and routing are completed within the Transport node (ESX02) before being forwarded to the T0 SR Construct on Edge node (BGGWEdge01).
Since T0 DR-SR ECMP is achieved in this topology, we should see traffic flowing to the other Edge node (BGGWEdge02) as well when we Traceflow to a different destination.
Southbound Traffic behavior
As shown in Sketch2, depending on how eBGP ECMP behavior is configured on the Leaf Switches, traffic can Ingress into the T0 SR Constructs over the 4 different paths. Each Edge node performs local routing lookup as the SR & DR Constructs are available to them. Traffic is tunneled to the Host transport node only after the T1 DR lookup. Another T1 DR lookup doesn’t happen on the Compute Transport nodes.
Let’s do a Traceflow from one of the Edge Uplinks to a VM on Segment 172.21.0.0/24 attached to the Tier 1 Gateway and confirm this.
Summary:
- Routing always happens closer to the Source
- For Northbound, it is the Compute Transport nodes
- For Southbound, it is the Edge Nodes.
- For a T1 Gateway without an SR Construct:
- Tier 0 DR-SR ECMP is achieved on the Compute Transport Nodes for Segments attached to the T1 Gateway
- Both Tier 0 SR Constructs are leveraged for Northbound routing
- All available Tier 0 eBGP Uplink ECMP are utilized for North bound routing (Provided ECMP is enabled under the BGP Process)
- As Tier 0 SR Constructs are scaled out (upto 8), more ECMP paths become available for the segments connected to the Tier 1 Gateway
- For any Asymmetric failures on the Edge Uplinks, Inter-SR Routing can be utilized here.
I hope the article was informative. We will continue in Part 2 where we discuss the Northbound routing behavior and ECMP for Tier 1 Gateway with an SR Construct on a Shared T0-T1 Edge Cluster.
Thanks for reading
Continue reading? Here are the other Parts:
Part 2 : https://vxplanet.com/2019/10/28/nsx-t-tier1-sr-placement-and-the-effect-on-northbound-ecmp-part-2/
Part 3 : https://vxplanet.com/2019/10/28/nsx-t-tier1-sr-placement-and-the-effect-on-northbound-ecmp-part-3/
VxPlanet 