Building an NSX-T Bring Your Own Topology (BYOT) for VMware Enterprise PKS – Part 4


Happy New Year everyone.

This is the fourth and final part of my multi-part blog series on ‘Building an NSX-T Bring Your Own Topology (BYOT) for VMware Enterprise PKS’. In case you missed the previous parts, here are the links:

Part 1 -> https://vxplanet.com/2019/12/25/building-an-nsx-t-bring-your-own-topology-byot-for-vmware-enterprise-pks-part-1/

Part 2 -> https://vxplanet.com/2019/12/26/building-an-nsx-t-bring-your-own-topology-byot-for-vmware-enterprise-pks-part-2/

Part 3 -> https://vxplanet.com/2019/12/29/building-an-nsx-t-bring-your-own-topology-byot-for-vmware-enterprise-pks-part-3/

Here are the contents of this article:

  • Deploying the Enterprise PKS Management Console OVA
  • Configuring Enterprise PKS Management Console and presenting NSX-T BYOT for consumption
  • Deploying Enterprise PKS

Let’s get started

Deploying the Enterprise PKS Management Console OVA

Enterprise PKS Management Console comes as an OVA virtual appliance. This is imported and deployed on the ‘Shared Management and Edge Cluster’ on the vSphere DVS Management VLAN (VLAN 105). The PKS Management and Control plane VMs deployed from the Enterprise PKS Management console sits on the Shared AZ01 and on the Overlay Tier1 Management Network (192.168.101.0/24 non-routable)

This is how the vCenter clusters look like:

52.png

Deployment of Enterprise PKS Management Console is same like importing any other OVA appliances to vCenter. The procedure is available in the Pivotal documentation below:

https://docs.pivotal.io/pks/1-6/console/deploy-console-ova.html

Also have a look at the pre-requisites, we have already met them in Parts 1, 2 & 3 though.

https://docs.pivotal.io/pks/1-6/console/ova-deployment-prereqs.html

Enterprise PKS Management Console deploys the below PKS Components:

  • Deploys Pivotal Ops Manager (VM)
  • Deploys BOSH Director (VM)
  • Deploys Enterprise PKS (VM)
  • Deploys Harbor Registry (VM)
  • Stem cells (which are templates to build the PKS VMS)
  • Configures networking for Enterprise PKS (NSX-T or Flannel). We used NSX-T BYOT here which is presented for Consumption.

Configuring Enterprise PKS Management Console and presenting NSX-T BYOT for Consumption

It’s good to read through the pre-requisites for NSX-T BYOT in the Pivotal documentation below. We have already met this in our configuration in Parts 1, 2 & 3.

https://docs.pivotal.io/pks/1-6/console/prereqs-nsxt-byot.html

A detailed step-by-step procedure is already available in the Pivotal documentation below, we will do a quick walkthough here:

https://docs.pivotal.io/pks/1-6/console/deploy-ent-pks-wizard.html

PKS Management Console is available on WebUI port 80. Once the PKS Management Console OVA is deployed successfully, navigate to it’s WebUI and Click ‘Install’.

1

Review the deployment prerequisites and click ‘Start Configuration’.

2

Connect to the vCenter instance (IaaS Platform)

3.png

Under Networking, select ‘NSX-T Bring Your Own Topology’. Connect to NSX-T Manager. Select the PKS Management network, POD Network, K8S Node network and Floating Pool ranges that we have already defined. Select NAT mode because the PKS management network is on a non-routable subnet.

4.png

Select the Authentication Provider. It could be local UAA, LDAP or SAML depending upon the environment. In our case, we used the Integrated UAA available with the PKS API VM. 

5.png

Create Availability Zones. Availability Zones are a BOSH Construct which maps to vSphere Clusters, Host Groups or Resource Pools. Whenever a Kubernetes Cluster is deployed, the Master and Worker nodes are deployed across the Availability Zones for Load Sharing and Fault Tolerance. In our case, we have three Compute Clusters which maps to three Compute Availability Zones. AZ01 is shared to host the PKS Management components.

24.png

Select the Ephemeral Storage to store temporary VMs in case of installation, upgrade and operations. Ephemeral VMs are automatically created and deleted as needed by the BOSH Director. Since we have three Availability Zones (three vSAN clusters), select all the three vSAN datastores. Alternatively, if there is any datastore accessible across all the Availability Zones, that can be used.

25

Select the Permanent Storage where the PKS VMs are deployed. Since we have three Availability Zones (three vSAN clusters), select all the three vSAN datastores.

26

Select the datastore to store persistent volumes for Kubernetes Clusters. We already have a shared NFS datastore accessible across all the Availability Zones and will use that.

10.png

Create the plans for Kubernetes Clusters. Plans are configuration templates that specifies the master/worker counts, resources, VM sizes and other custom properties of the Kubernetes Clusters. Atleast one plan is mandatory. Make sure that all the Availability Zones are enabled for the plans for load sharing and better fault tolerance. We’ve created three plans – Small, Medium and Large.

11121314

Select Integrations if needed.

15

Optionally enable Harbor Registry as a repository to store container images. Install Notary and Clair if needed.

16.png

Join CEIP and Telemetry program, if needed.

17

Generate the configuration and if needed, export and save the yaml file.

1819

Deploying Enterprise PKS

Click on Apply Configuration, the deployment starts and wait for it to finish. It takes nearly an hour to complete.

202122

Once the deployment has succeeded, we are presented with a nice dashboard with links to access the deployed PKS services.

23.png

Deployment metadata tab shows the IPs, username and credentials to access the PKS services. Since we used NAT mode for the PKS Management network, the IPs shown here corresponds to the NAT IPs (taken from the floating pool)

27.png

We can see the relevant SNAT/DNAT rules created for the PKS Management Network under the NSX-T Tier 1 Gateway instance.

28

Navigate to the Pivotal Ops Manager WebUI and explore the tiles to confirm that the settings are mapped correctly.

29

Optionally set the default NAT mode for Kubernetes nodes. This setting can be overridden using PKS network profiles during cluster creation.

34

Also, confirm that the PKS Management VMs are deployed on the shared AZ01.

37

Finally as part of post-deployment validation, verify the Enterprise PKS Deployment on vSphere with NSX-T using the steps outlined in the Pivotal documentation.

https://docs.pivotal.io/pks/1-6/nsxt-install-verify-pks.html

This concludes the 4-part blog series.

I hope this was informative. Thanks for reading

Continue reading? Here are the other parts of this series:

Part 1 -> https://vxplanet.com/2019/12/25/building-an-nsx-t-bring-your-own-topology-byot-for-vmware-enterprise-pks-part-1/

Part 2 -> https://vxplanet.com/2019/12/26/building-an-nsx-t-bring-your-own-topology-byot-for-vmware-enterprise-pks-part-2/

Part 3 -> https://vxplanet.com/2019/12/29/building-an-nsx-t-bring-your-own-topology-byot-for-vmware-enterprise-pks-part-3/

vxplanet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s