NSX-T supports BGP as the only dynamic routing protocol to peer the Edges (T0 Gateways) with the customer network. BGP is supported only on the Tier 0 gateways unlike NSX-V which supports dynamic routing on both the DLR and ESGs. Peering of T1 and T0 Gateways are fully managed by NSX-T and is transparent to the administrators. T1 and T0 Gateways attach to each other using NSX-T router links (on a pre-defined 100.64.0.0 subnet) which is fully managed by NSX-T manager. When we enable Route advertisement on the T1 Gateway, NSX-T manager automatically establishes static routing between the T1 & T0 Gateways for the advertised routes. These routes are named as NSX Static routes in the UI.
In this post, we will walk through on how to establish BGP Peering between T0 Gateways in Active-Active mode with Dell Networking S5048-ON switches in VLT.
Let’s get started.
Environment details
- vSphere 6.7 U2 environment running on 4 x Dell EMC PE R640 platform with 2 x 25G NICs
- NSX-T 2.4.1 – Single Tenant two-tiered hierarchy with two Edge nodes (VM Form factor)
- NSX-T T0 Gateway deployed in Active-Active mode with two Uplinks, each one via separate Edges
- Dell Networking 2 x S5048-ON ToR Switches in VLT (L3 Leaf)
- This is a Collapsed Management, Compute and Edge infrastructure
Initial BGP Configuration on the S5048-ON VLT ToR Switches
Let’s configure BGP peering between the two Dell S5048-ON VLT Swiches. We use ASN 65500 for the ToRs and they are configured as iBGP peers with each other and with the same networks advertised on both of them.
Lets configure BGP on the first VLT ToR
We would see the State as ‘Active’. Don’t get confused with this state. ‘Active’ is not good in BGP, it simply means that it can’t peer with the other switch. After successful peering, we should see the state as ‘Established’.
Lets configure BGP on the second VLT ToR
Now lets look at the BGP states of both the ToR switches. We should see that the state changed to ‘Established’.
We can also see the Capabilities exchanged between the BGP peers like Dynamic Route Refresh, MP BGP etc – just for information. Note that for successful BGP peering, the Hold time values should be same on both switches, by default it is 180 seconds.
Advertising Customer networks into BGP
The way in which networks are advertised depends upon the customer environment. In most cases, the ToR VLT Switches establish BGP peering (iBGP or eBGP) with the spines depending upon the datacenter architecture and receive the customer networks. In this post, lets advertise some networks on the ToR switches.
Now lets verify the BGP table.
Each ToR will select the route advertised from it as the preferred path because it is local to it. Locally originated routes have higher preference.
Enabling BFD on the S5048-ON VLT ToR Switches
It is always good to enable BFD to enable faster detection of communication path failures. BFD provides detection times in the order of milliseconds unlike BGP which is slower. BFD can be used in BGP both for iBGP peers and eBGP peers. Note that BFD doesn’t support multihop for eBGP, so the neighborship has to be established on directly connected interfaces.
Enabling BFD globally on the ToR switches
Enabling BFD support for BGP
Let’s enable BFD support for all the neighbors.
The BFD status should show up between the peers.
Enabling Route Advertisement on the Tier-1 Gateway
Once the T1 gateway is attached to the T0 Gateway, we have to enable Route advertisement on the T1 Gateway so that the networks can be seen from the T0 Gateway. NSX-T manages the uplinks and static routing between them and is transparent to the administrator.
Configuring Uplinks on the Tier-0 Gateway
The T0 Uplinks need to be configured according to our BGP architecture. In this post, I will use two Uplinks for the T0 gateway:
- First Uplink on VLAN 60 over Edge 1 (192.168.60.254/24)
- Second Uplink on VLAN 70 over Edge 2 (192.168.70.254/24)
These Uplinks establish separate eBGP peering with the ToR switches
Enabling Route Redistribution on the Tier-0 Gateway
Now that we have configured Route advertisement on the T1 Gateway, we need to advertise these networks into BGP from the T0 Gateway. In NSX-T , we do this via Route redistribution. This can be done before or after enabling BGP on the T0 Gateway
BGP Configuration on the Tier-0 Gateway
We will configure BGP on the T0 Gateway with a different ASN to establish eBGP peering. With Version 2.4, T0 Gateways also support iBGP peering, but the decision depends upon the architecture. iBGP peering usually requires a full-mesh connection to overcome the Split-horizon phenomenon with iBGP neighborship. We will use ASN 65400 for the T0 Gateway and eBGP peer it with the ToR VLT switches (which are on ASN 65500). Since T0 Gateway has two Uplinks each one via separate Edges, we will have two eBGP peerings.
T0 Active-Active mode and Inter-SR iBGP
Since we have deployed the T0 gateway in Active-Active mode, it is good to enable Inter-SR iBGP routing between them. NSX-T manager will automatically configure an iBGP peering between the Uplink SR Components of the T0 Gateway. Since the SR components sit on separate Edge nodes with dedicated Uplinks, failure of an Uplink on one Edge will not affect the North bound traffic reaching on it as the path will be switched over to the other Edge. The iBGP peering is established over the NSX managed network 169.254.0.0/16 network. This configuration is transparent to the administrator.
Configuring the eBGP neighbors on the T0 Gateway
We add the two ToR switches in VLT (AS 65500) as neighbors to the T0 Gateway (AS 65400). We will also enable BFD for the eBGP neighborship. Lets keep the defaults for the holddown and BFD timers.
We have to make sure that the neighborship is established over the correct Uplink interfaces of the To Gateway. We can confirm this via the Advanced UI.
BGP Peering S5048-ON VLT ToR Switches with Tier-0 Gateway
Now we will eBGP peer the ToR switches with the T0 Gateway
Let’s verify the BGP peering status
We can see that the eBGP peering status is successful and it has received 3 prefixes from the T0 Gateway peer. Since we use BFD with BGP neighbors, we have to confirm that the BFD status is up.
Now lets see how the BGP table on the ToR Switches look like.
As we can see, we got 3 prefixes on the ToR switches from the T0 Gateway. Each prefix has got two paths from the ToR switches – one via eBGP with T0 Gateway and the other via iBGP with the other ToR switch. According to the BGP path selection rule, eBGP paths are preferred over iBGP path and as we can see above the best path points to the one learned via eBGP neighborship.
The Origin code is marked as “Incomplete” because the prefixes are redistributed into BGP by the To Gateway. Nothing to worry here, as in our case it is not of a concern.
Verifying BGP status from the Edge Nodes
Let’s connect to one of the Edge nodes and confirm that the BFD status to the ToR BGP peer is up.
Let’s verify the BGP neighborship
Notice that the SR Component on each Edge has established an iBGP peering between themselves. This way, when one of the Edge Uplink goes down, traffic will get routed over to the other edge through the iBGP interface. iBGP learned routes take less preference over the eBGP learned routes. So for any networks to the customer environment, each SR Component chooses the eBGP path and in case the uplink fails, the iBGP path is chosen as the alternate.
Lets look at the BGP table on the To Gateway. It has now received all the customer routes that were advertised from the ToR switches.
Traffic to the customer networks are routed to the eBGP ToR neighbor. Traffic to individual Tenants (T1 Gateway) are routed to the respective T0-T1 router links.
This is all for this post. I hope you enjoyed the article.
Thanks for reading.
VxPlanet 