NSX-V supports dynamic routing with OSPF and BGP. Both of these protocols are supported at both DLR and ESG. With OSPF/BGP you can dynamically peer the tenants that you spin up in NSX environment with the ESGs as well as with the external physical networks. This is suitable when your private cloud architecture is on a routable network. That means, your private cloud tenants are visible to the external physical networks and are accessible directly without the need for SNAT/DNAT on the ESGs. Careful decisions need to be taken when designing the logical routing architecture to avoid unnecessary flooding of routing change updates or LSAs with dynamically changing Tenant networks.
This article is about performing a study on NSX-V routing with different OSPF area types – (Backbone, Inter Area and NSSA), understand the LSA flooding and filtering options on logical tenants and see why NSSA is the recommended option for DLR-ESG peering.
Environment Details
The environment I used had the below specs:
- 4 X DellEMC PowerEdge R640 vSAN nodes
- 2 X DellEMC Networking S4048-ON ToR switches in VLT
- 2 X DellEMC Networking S5048-ON Aggregation switches in VLT
- e-VLT between the S4048 ToR & S5048 Aggregation
- 1 X DellEMC Networking S3048-ON switch for Out-of-box management
- NSX-V Enterprise
- Two Tenants named Tenant A & Tenant B
- Tenant A:
- Logical Switch 1 – Segment 172.16.10.0/24
- Logical Switch 2 – Segment 172.16.20.0/24
- Transit Logical Switch – Segment 10.0.0.0/24
- Single DLR instance – 2 internal vNics & 1 Uplink
- Logical switches 1 & 2 attach to the internal vNics. The IP of the vNICs is the DG for the workloads on the respective logical switches
- The uplink vNIC attach to a Transit logical switch which in turn connects to the ESG.
- Tenant B:
- Logical Switch 1 – Segment 172.17.10.0/24
- Logical Switch 2 – Segment 172.17.20.0/24
- Transit Logical Switch – Segment 11.0.0.0/24
- Single DLR instance – 2 internal vNics & 1 Uplink
- Logical switches 1 & 2 attach to the internal vNics. The IP of the vNICs is the DG for the workloads on the respective logical switches
- The uplink vNIC attach to a Transit logical switch which in turn connects to the ESG.
- Tenant A:
- Single ESG instance for North bound traffic.
- 3 vNiCs – 2 internal vNiCs attaches to the transit segments of the DLRs. Third vNiC attaches as an uplink to the VLAN Port Group (VLAN 50).
- Two Tenants named Tenant A & Tenant B
- There are OSPF networks that are advertised from the physical environment and they will be seen in the routing tables of the NSX-V ESGs and DLRs
To see the configuration of the physical switching environment including VLT setup, eVLT setup, Port-channels, VLANs, SVIs and VRRP please refer to my earlier post
The below sketch shows the environment (Click here for the HQ Image). For simplicity in the sketch, I haven’t drawn the eVLT links and and the ToR switches (as it just sites on L2 network).
Scenario 1 – Everything in a Single OSPF Backbone Area
Here, the physical infrastructure, NSX ESGs and all the DLRs for the Tenants are in a single OSPF area 0 (The backbone area).
OSPF Configuration on the S5048-ON Aggregation switches
As the switches are in VLT, their OSPF configuration is identical except that some networks are redistributed from Core01. This is to present some external E2 routes to the NSX logical routers. The L3 routing tables (and the control plane) is synced through the VLTi interconnect. The two switches have VRRP configured for all the VLAN Interfaces, but the OSPF peering happens over the SVI address and not through the VRRP virtual address. Since in VLT, both VRRP switches work in Active-Active mode, we don’t need to worry about what the OSPF next-hop is on the switches. But it’s good to have ECMP, so we get a kind of load sharing there.
Core01
The three Loopback interfaces are redistributed to OSPF process, so they appear as E2 routes to other OSPF speakers.
Core02
OSPF Configuration on the NSX-V ESG
All the ESG interfaces are configured under Area 0. I have changed the priority of the Uplink interface (VLAN interface) to 0 just to avoid ESG becoming the DR for the Area 0 segment on the physical network. Graceful restart is enabled to allow OSPF process to be restarted while maintaining the adjacency and data plane traffic. [Click here for HQ image]
OSPF Configuration on Tenant01 (DLR_Tenant01)
On the DLRs, we can enable OSPF only on the Uplink interface. All the internal Interfaces to where the Logical Switches are attached needs to be redistributed into the OSPF process. ie, each Tenant network appears as an E2 network to other DLRs as well as to external routers. That means, each DLR acts as an ASBR for the OSPF area.
OSPF Configuration on Tenant02 (DLR_Tenant02)
This is identical to Tenant01. Uplink interface is configured under Area 0 and the internal interfaces connecting to logical switches are redistributed into the OSPF process.
Routing Tables, OSPF LSDB, Segments and Neighbors
In this topology that we used, all the routers are in a a single Area 0. There are 3 broadcast segments in this backbone area.
- Segment 1 -> between the Aggregation switches (external) and the Uplink Interface of ESG.
- Segment 2 -> Between Tenant 1 and ESG
- Segment 3 -> Between Tenant 2 and ESG
The segments increases with the number of Tenants as well as with additional ESG deployments.
For each segment, there will be an OSPF DR and BDR. So in our scenario, we have 3 DR/BDRs. I have changed the priority of the VLAN uplink of ESG to 0 to avoid it becoming the DR/BDR for the physical segment.
Lets look at the tables on the Aggregation switches:
See that all the logical networks advertised from the DLRs appear as Type-5 LSAs (or E2 routes). We also see the network information form all the 3 DRs in the Network LSA (Type 2 LSA) table.
The highlighted routes in the above figures are the ones advertised from the DLRs of tenants.
Lets look at the tables on a Tenant DLR:
Each Tenant DLR receives Type-5 LSAs from the DLRs of other tenants as well as from external networks. In the above figure,
1 means -> The routes that are redistributed into OSPF process from the local Tenant (Tenant 1)
2 means -> The External routes (E2) from other Tenants (Tenant 2) on the NSX environment.
3 means -> The External routes that are advertised from the physical network. In this example I did “redistribute connected” on the Core01 switch, so all the routes are seen here. You can use route maps or distribute list to filter out unwanted routes, I didnt do that in this case (excuse)
This is the routing table of that Tenant DLR.
Observations from Scenario 1
- A tenant DLR has only one uplink interface to ESG and enabled with OSPF and this is the single point of entry and exit for North bound traffic. It already has a default route pointing to the ESG. In that case why do we need to build up the routing table of a DLR with additional network information from other Tenants as well as from external networks?
- As seen above, as the number of Tenants and Tenant networks increases, the number of routes in the DLR routing table also increases – this is kind of a duplicate entry as there is already a default route pointing to the ESG. It’s good to keep the routing table size and the LSDB shorter.
- As the DLR routing table has E2 routes from other Tenants, a change to a network in other Tenants like deleting a Logical switch, Creating a logical switch, Deleting a Tenant etc causes LSA flooding to this DLR as everything is in a single OSPF area. Based on the Tenant link states, the DLR needs to maintain the LSDB accordingly as well as run SPF. This is unnecessary
- Logical Networks of a Tenant is seen by the DLR of another Tenant (as an E2 route). This is also unnecessary.
- Each DLR instance is an OSPF ASBR. For large scale Tenant deployments, we need to have a mechanism to control the External routes advertised.
Single Area deployment is not a good option for the above observations especially on large scale Tenant deployments. The solution to this is OSPF NSSA.
Lets move on to Part 2 and see how OSPF NSSA areas addresses these issues.
VxPlanet 