Creating Multi-Master BareMetal Cluster on Platform9 Managed OpenStack VMs

Problem

The purpose of this document is to provide a step-by-step guide for creating a BareMetal Multi-Master Kubernetes cluster on the VMs spawned in Platform9 Managed OpenStack. (Please note that this is not OpenStack Provider. The VMs would be treated as BareMetal nodes and it would be an agent-based install). This document would shed light on how to set up networking in OpenStack for Virtual IP to ensure the High Availability of the API server component running on the master servers.

Use Case: The use case assumed here is that the VMs created on Platform9 Managed Openstack are connected to the Platform9 Managed Kubernetes as nodes and are to be used as Kubernetes Cluster nodes.

Note: This document assumes that Virtual IP is in the same subnet as that of the masters. This document is specifically intended for VMs running on PMO and OpenStack in general. In the case of Physical BareMetal nodes, this document will not apply.

Environment

• Platform9 Managed OpenStack - v3.11.0 and Higher
• Platform9 Managed Kubernetes - v3.11.0 and Higher

Procedure

Let's take a look at an example for better understanding.

VIP      - 10.128.233.23 (MAC - fa:16:3e:71:2f:00)

master-0 - 10.128.233.20 (MAC -  fa:16:3e:e4:b2:3a)

master-1 - 10.128.233.45 (MAC - fa:16:3e:95:cf:02)

master-2 - 10.128.233.48 (MAC - fa:16:3e:e4:b2:3a)

1. Create a neutron port for reserving the Virtual IP. This can be done either using OpenStack CLI or Clarity UI.
   • Using CLI - https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/port.html
   • Using Clarity UI - Networks → Select Network → Ports → Create a New Port

2. Make sure that port Security is enabled for the ports associated with the master VMs and the VIP port as the allowed_address_pairs functionality of Neutron needs port security enabled.
3. Neutron ports associated with master VMs will need to have the following allowed_address_pair in the format - VIP, [MAC of the actual master node port]. This will make the neutron port responds to both the VIP as well as the master IP itself. Without this setting, default port security prevents any packet coming in or going out on qvo interface unless the IP and MAC match that of the physical/bound neutron port of the master node.

Here's an example of the neutron port configuration of a master node "master-1".

4. Configure all the master node neutron ports with a similar configuration and then trigger a cluster creation with the API endpoint parameter as the VIP.