This is the second part of the blog series on NSX-T Tier 1 Gateway SR Construct placement and it’s influence on North-South routing and ECMP. In case you missed Part 1, you can read it here:
https://vxplanet.com/2019/10/26/nsx-t-tier1-sr-placement-and-the-effect-on-northbound-ecmp-part-1/
In this article we will focus on Scenario 2 – Tier 1 Gateway with an SR Construct on a Shared T0-T1 Edge Cluster. We will not cover the effect on East-West routing here, and will deal that as a separate article. Let’s get started
Tier 1 Gateway with an SR Construct on Shared T0-T1 Edge Cluster: North-South Routing and ECMP
The below sketch shows the traffic pattern for North bound flow from a Segment connected to the T1 Gateway having an SR Construct on the Edge nodes.
[Click here for HQ Image] Sketch1
And the below sketch shows the traffic pattern for a South bound flow from the Leaf Switches to the Segment connected to the same T1 Gateway.
[Click here for HQ Image] Sketch2
As mentioned earlier, it is important to note that routing happens closer to the source of traffic. For Northbound, it would be the Compute Transport nodes and for Southbound it would be the Edge nodes. As such, the return path for a flow can be on a different route compared to the source flow.
Topology Details
The Logical Topology is same as Part 1 except that the Tier 1 Gateway has an SR Construct on the same Edge Cluster used by the Tier 0 Gateway.
- Tier0 Gateway is deployed in Active-Active mode with 4 uplinks on an Edge Cluster with 2 Edge nodes.
- Each Edge node hosts two uplinks – one on VLAN 60 and the other on VLAN 70
- The T0 Gateway establishes eBGP peering with two Dell EMC S 5048-ON Leaf switches. We will have 4 eBGP peerings in total
- T0 Gateway uses eBGP ECMP on the uplinks and Inter-SR Routing between the SR nodes.
- A single Tier 1 Gateway deployed with an SR Construct and uplinked to the Tier 0 Gateway. The SR Construct leverages the same Edge cluster used for the T0 Gateway
- A Logical Segment on 172.21.0.0/24 attached to the Tier 1 Gateway
We will not cover the Topology configuration here, as this has already been covered in my earlier articles.
Northbound Traffic behavior
As shown in Sketch1, the Tier 1 DR Construct has two interfaces – One attached to the Logical Segment (over auto-plumbed VNI 71701) and the other attached to it’s SR Construct sitting on the Edge Node (over auto-plumbed VNI 71704).
The T1 SR Construct is always deployed in Active-Passive mode on the Edge Nodes. The Passive SR Construct will have all it’s interfaces in an Operationally down state but is in sync with the Active SR Construct (NSX-T Manager maintains the state synchronization and desired state)
Traffic from the Compute Segments (172.21.0.0/24) attached to the T1 Gateway will be tunneled to the Edge node hosting the Active T1 SR Construct for Northbound reachability. This happens after the T1 DR Lookup on the Compute Transport Nodes and another T1 DR lookup is avoided at the Edge node (hosting Active T1 SR).
Let’s see the Next-hop of T1 DR and T1 SR Constructs.
This is the Forwarding table of the T1 DR which is next-hopping to it’s Active SR Construct on the Edge node.
This is the Forwarding table of the T1 SR next-hopping to Tier 0 DR Construct. This T0 DR is available locally on the Edge node, so the lookup happens locally and doesn’t need to leave the Edge node for northbound reachability.
The T0 DR Construct will next-hop to the T0 SR Construct which is again available locally on the Edge node. Note that within an Edge node we have only a single path for T0 DR-SR routing and all Northbound traffic takes this local route. This is different from the Part 1 Scenario where we saw T0 DR-SR ECMP behavior on the Compute Transport nodes. All Northbound traffic will Egress out of the eBGP uplinks on the SR Construct of the Edge node leveraging ECMP on it’s uplinks to the Leaf Switches.
Note that , in this Scenario, only the Active Edge Node hosting the Tier 1 SR Construct is involved in Northbound routing. The other Edge node (hosting the Passive Tier 1 SR Construct) doesn’t take part in Northbound routing. Hence ECMP at the T0 DR-SR level isn’t leveraged. The eBGP Uplinks on the Active Edge Node (for the T1 SR Construct) leverage ECMP. Hence the total number of ECMP paths available in this topology is 2 unlike Scenario 1 which was 4.
Let’s do a Traceflow to see the Flows in action:
Our source is a VM attached to the Logical Segment (172.21.0.0/24) on the T1 Gateway. Destination is a machine outside the NSX-T Environment. The Edge node named ‘bggwedge02’ is the Active node for the T1 SR Construct.
As you can see, once after the T1 DR lookup is completed on the Compute Transport node (esx02.orange.local), the traffic has been tunneled to the Edge node (bggwedge02) hosting the Active T1 SR Construct. Thereafter, the Edge node does all the upstream routing locally. Traffic gets egressed via it’s eBGP ECMP uplinks.
Southbound Traffic behavior
As shown in Sketch2, depending on how eBGP ECMP behavior is configured on the Leaf Switches, traffic can Ingress into the T0 SR Constructs over the 4 different paths. Traffic can Ingress either via
- The Edge node holding the Active T1 SR Construct or
- The Edge node holding the Passive T1 SR Construct
We will look at the two options here:
For Ingess traffic into the Edge node holding the Active T1 SR Construct (bggwedge02), it performs all the DR & SR lookups locally (as they are available locally) and the traffic is tunneled to the Compute Transport node (esx02.orange.local) only after the T1 DR routing has completed. No T1 DR routing lookup is performed at the Compute Transport node.
For Ingess traffic into the Edge node holding the Passive T1 SR Construct (bggwedge01), it is tunneled to the Active T1 SR Edge node after the T0 DR Lookup as it’s local T1 SR Construct is in an Operationally down state. This is seen in the Traceflow results below.
Summary:
- Routing always happens closer to the Source
- For Northbound, it is the Compute Transport nodes
- For Southbound, it is the Edge Nodes.
- For a T1 Gateway with an SR Construct leveraging the same T0 Edge Cluster:
- Tier 0 DR-SR ECMP is NOT available on the Compute Transport Nodes for Segments attached to the T1 Gateway. This is because the traffic has already been routed to the Edge node for T1 SR lookup and thereafter all upstream routing happens locally on the Edge node (holding the Active T1 SR Construct).
- Both Tier 0 SR Constructs are NOT leveraged for Northbound routing. Only the Edge node hosting the Active T1 SR Construct is involved in Northbound routing.
- All available Tier 0 eBGP Uplink ECMP are NOT utilized for North bound routing. ECMP for eBGP Uplinks are only utilized from the Active T1 SR Edge node. The Passive T1 SR Edge node is not utilized.
- As Tier 0 SR Constructs are scaled out (upto 8), the available ECMP paths Northbound remains the same. Any additional ECMP paths are NOT available for the Segments in the T1 Gateway
- For any Asymmetric failures on the Edge Uplinks, Inter-SR Routing can be utilized here.
I hope the article was informative. We will continue in Part 3 where we discuss the Northbound routing behavior and ECMP for Tier 1 Gateway with an SR Construct on a dedicated Edge Cluster.
Thanks for reading
Continue reading? Here are the other parts:
Part 1 : https://vxplanet.com/2019/10/26/nsx-t-tier1-sr-placement-and-the-effect-on-northbound-ecmp-part-1/
Part 3 : https://vxplanet.com/2019/10/28/nsx-t-tier1-sr-placement-and-the-effect-on-northbound-ecmp-part-3/
VxPlanet 