Hello everyone, this is Part 2 of my post on “Getting Started with BDaaS Private Cloud”. Part 1 is available at the below link, in case you need to have a walkthrough.
In this post I will cover the below areas:
- Managing Host Tags for virtual cluster affinity.
- Adding Worker Hosts to the Platform (+ scaling out the cluster)
- Steps covered during the installation
- Handling a failed installation
- Adding Gateway Hosts to the Platform
- Steps covered during the installation
- Handling a failed installation
- Adding additional Gateway hosts for HA and redundancy
- Managing Gateway Port mappings
Managing Host tags
How do I manage host tags?
Setting up “host tags” and attaching to worker hosts helps in creating placement rules (affinity) for the virtual nodes at a later stage. For Eg: Creating a tag called “GPU Accelerated” for GPU Accelerated workers for TensorFlow / ML apps etc.
Under the installation menu, click on “Manage Tags” and create your own tags as per the requirement.
Adding Worker Hosts
How do I add worker hosts?
You can add workers by navigating to the “Installation” menu and providing the details under the “Worker” section.
Enter one or more IPs for the worker hosts and select the tags (if needed). Since you have configured password less ssh, select the ssh private key and click on “Add Hosts”.
The installer will perform the prechecks on the hosts, downloads and installs the necessary rpms. Once this is completed, you can see the staus against the worker hosts as “bundle completed”. If for any reasons, the installation fails you can review the setup log or login to the worker hosts and review the logs at /tmp/ and /var/log/bluedata directories.
It’s recommended to use symmetric configuration across the platform. If you have assymetric disk configuration on your worker hosts, you will notice a warning like the below.
What steps does the installer do?
The installer perform the below steps while adding a worker host to the platform:
- Worker prechecks
- The steps are exeactly the same as used for the controller prechecks. For a list of steps covered,see What does the prechecks script do?
- This generates 3 log files under the /tmp/ directory
- <PID>.log
- <PID>.log.xtrace
- <PID>.conf
- A precheck config file is generated which is used by the EPIC installer (bd_iu_prechecks.<PID>.conf)
- Worker installation
- Initiates the installation logging to /tmp/worker_setup_<IP address><Date><Time>.log
- Cleans up the yum metadata to remove any stale RPM version info.
- Installs the bluedata repository which contains the controller and worker rpms.
- Checks whether the dependencies are already installed. Else it downloads and installs the dependencies
- The list of all the required RPMs are found HERE
- Installer then installs the worker, docker,OVSwitch and haproxy RPMS.
- bluedata-common.x86_64 0:3.2-2177
- bluedata-controller.x86_64 0:3.2-2177
- bluedata-worker.x86_64 0:3.2-2177
- docker.x86_64 2:1.12.6-71.git3e8e77d.el7
- haproxy.x86_64 0:1.5.18-6.el7
- libxml2-devel.x86_64 0:2.9.1-6.el7_2.3
- libxslt-devel.x86_64 0:1.1.28-5.el7
- openssl-devel.x86_64 1:1.0.2k-8.el7
- openvswitch.x86_64 0:2.5.2-1.el7.centos
- python-cffi.x86_64 0:1.6.0-5.el7
- python-devel.x86_64 0:2.7.5-58.el7
- socat.x86_64 0:1.7.3.2-2.el7
- Does the worker configuration.
- Installer also sets up the VxLAN networking across the hosts. It deploys the bds-local bridge, bds-ex bridge and bds-gateway bridge
What if you end up with a failed worker installation?
The installer will automatically rollback if the installation fails for some reason. You can review the logs at the below locations and take corrective actions and retry the installer
- /tmp/ – Review the prechecks logs and installer logs
How do I install the worker host to the platform?
Once you have finished adding your worker hosts to the platform, you can see the status as “Bundle Completed”.
Enter “Site Lockdown”, select the worker hosts and under the action menu, click on “Install”.
Once the installation is successful, you can see the status as “Installed”
You can view the log files under /var/log/bluedata/install/worker_setup_<IP Address><Date><Time>.
The below actions are performed:
- Prepares the Node Storage.
- Creates physical volumes on the disks.
- Creates a Volume group called “VolBDSCStore” using those physical disks.
- Creates a logical volume “thinpool” on this volume group.
- This Linux logical volume is assigned to the Linux Docker subsystem, which then uses the Linux device mapper functionality to allocate portions of the thinpool logical volume to the containers running on that host for use as local storage within those containers.
- Prepares the Tenant HDFS Storage
- Creates a single partition on the disks (eg:/dev/sdg1), writes the xfs filesystem and mounts it under /opt/bds-hdfs-storage/jbod-X where X = 0 to n
- Creates the 3 containers which are explained under the “EPIC Platform Configuration”
- The container “epic/apache-hdfs-centos:2.7.2” runs the Standby name node and Data node for the local HDFS Tenant storage.
Once the worker installation is completed, exit from the Site lockdown.
Adding Gateway Hosts
How do I add Gateway hosts?
You can add Gateway hosts depending upon two factors:
- How many gateway sets you need.
- Whether you need HA for the gateways
You can add gateway hosts by navigating to the “Installation” menu and providing the details under the “Gateway” section. You need to enter “Site Lockdown” mode inorder to add the Gateway hosts.
Unlike Worker hosts, Gateway hosts do not run containers. Hence they don’t need a tenant storage or node storage.
Enter the IP for the gateway host. Enter the Gateway set name as the hostname. Make necessary DNS entries so that the gateway set hostname resolves to the gateway host. If you add additional gateway host for HA, make sure to add to the same gateway set.
Add only one gateway host at a time. Since you have configured password less ssh, select the ssh private key and click on “Add Hosts”.
The installer will perform the pre-checks on the hosts, downloads and installs the necessary rpms. Once this is completed, you can see the status against the gateway hosts as “bundle completed”. If for any reasons, the installation fails you can review the setup log or login to the gateway hosts and review the logs at /tmp/ and /var/log/bluedata directories.
What steps does the installer do?
The installer perform the below steps while adding a gateway host to the platform:
- Worker prechecks
- The steps are exeactly the same as used for the controller and worker prechecks. For a list of steps covered,see What does the prechecks script do?
- This generates 3 log files under the /tmp/ directory
- <PID>.log
- <PID>.log.xtrace
- <PID>.conf
- A precheck config file is generated which is used by the EPIC installer (bd_iu_prechecks.<PID>.conf)
- Worker installation
- Initiates the installation logging to /tmp/worker_setup_<IP address><Date><Time>.log
- Cleans up the yum metadata to remove any stale RPM version info.
- Installs the bluedata repository which contains the controller and worker rpms.
- Checks whether the dependencies are already installed. Else it downloads and installs the dependencies
- The list of all the required RPMs are found HERE
- Installer then installs the docker,OVSwitch and haproxy RPMS.
- bluedata-common.x86_64 0:3.2-2177
- bluedata-controller.x86_64 0:3.2-2177
- docker.x86_64 2:1.12.6-71.git3e8e77d.el7
- haproxy.x86_64 0:1.5.18-6.el7
- libxml2-devel.x86_64 0:2.9.1-6.el7_2.3
- libxslt-devel.x86_64 0:1.1.28-5.el7
- openssl-devel.x86_64 1:1.0.2k-8.el7
- openvswitch.x86_64 0:2.5.2-1.el7.centos
- python-cffi.x86_64 0:1.6.0-5.el7
- python-devel.x86_64 0:2.7.5-58.el7
- socat.x86_64 0:1.7.3.2-2.el7
- Does the gateway and HAProxy configuration.
- Installer also sets up the VxLAN networking across hosts. It deploys the bds-local bridge, bds-ex bridge and bds-gateway bridge
What if you end up with a failed gateway installation
The installer will automatically rollback if the installation fails for some reason. You can review the logs at the below locations and take corrective actions and retry the installer
- /tmp/ – Review the prechecks logs and installer logs
How do I install the gateway hosts to the platform?
Once you have finished adding your gateway hosts to the platform, you can see the status as “Bundle Completed”.
Select the worker hosts and under the action menu, click on “Install”.
Once the installation is successful, you can see the status as “Installed”
You can view the log files under /var/log/bluedata/install/addworker.out-<Date><Time>.
How do I add additional gateway hosts for HA and redundancy?
To achieve HA for gateway nodes, you could either use DNS round robin or external ADCs (like Kemp Loadmaster). In the below steps we use DNS Round robin.
- Add the additional gateway nodes to the same gateway set
- Create the host entries for the individual gateway nodes in your local DNS server.
- Create the host entries for the gateway set name to point to both the gateway nodes. Load balancing is achieved by DNS round robin.
- If any of the gateway node dies or needs to be take for maintenance, make sure to update the DNS records as it can’t dynamically detect host failures.
How do I manage gateway port mapping?
When you create the cluster, EPIC will automatically add the necessary mapping entries to the haproxy tables. When a cluster is deleted, the mapping tables are updated appropriately. In most cases you don’t need to manually deal with port mapping.
The haproxy config file is located at /etc/bluedata/haproxy.conf
You can use the “bdconfig –addmapping” utility to manually manage port mappings. Reference at http://docs.bluedata.com/32_manually-adding-port-mappings
This is all for now. Hope both Part 1 & Part 2 are informative. Thanks for reading and good luck on your BDaaS Private Cloud.
Excellent post, Good write up. Cheers