Enabling Luigi Operator through Qbert API

Bootstrapping Cluster with Luigi NetworkOperator via API

Qbert-API Calls

In PMK version 4.5 new entries have been added to the qbert-api including: ipv6, networkplugin: calico, deployLuigiOperator, containersCidr, servicesCidr, calicoIPv6PoolCidr, privileged, calicoIPv4, calicoIPv6, calicoIPv6PoolNatOutgoing and calicoIPv6PoolBlockSize.

• ipv6: This is the most important parameter. It triggers the cluster components to use IPv6 addressing for various Kubernetes components like CoreDNS, KubeProxy, Canal, API server etc. (valid values are 0, 1 or false/true) Setting ipv6 must also set calicoIPv6 and** calicoIPv6PoolCidr** (more on this below).
• _deployLuigiOperator: _this boolean value will allow you to deploy a cluster with Luigi NetworkOperator Installed
• “networkplugin”: “calico”: Platform9 supports Flannel and Calico network plugins, for the IPv6 only Calico is supported.
• containersCidr & servicesCidr: Please specify the IPv6 CIDR when setting the “ipv6”: 1. Additionally, if the ipv6 flag is set, the value populated in containersCidr must also be populated in calicoIPv6PoolCidr. Calico only supports a subnet mask greater than /112 . Please make sure the CIDR specified is between /112 - /123. For example fd00:101::/64 is an invalid value but fd00:101::/112 is acceptable.
• privileged: This is a requirement for calico to run - so turning ipv6 on must turn this on automatically.
• calicoIPv4 and calicoIPv6: These are complimentary. If the ipv6 flag is set to true, we need to set calicoIPv4 to none and calicoIPv6 to** autodetect. Vice versa if **ipv6 is set to false. (valid values are none and autodetect).
• calicoIPv6PoolNatOutgoing: This is similar to the calicoNatOutgoing field that exists already. Need to turn it on if pod traffic leaving the host needs to be NAT’d. (valid values are 0/1)
• calicoIPv6PoolBlockSize: Block size to use for the IPv6 POOL created at startup. Block size for IPv6 should be in the range 116-128.
• calicoIPv4DetectionMethod & calicoIPv6DetectionMethod:

1. first-found Use the first valid IP address on the first enumerated interface (common known exceptions are filtered out, e.g. the docker bridge). It is not recommended to use this if you have multiple external interfaces on your host.
2. can-reach= Use the interface determined by your host routing tables that will be used to reach the supplied destination IP or domain name.
3. interface= Use the first valid IP address found on interfaces named as per the first matching supplied interface name regex. Regexes are separated by commas (e.g. eth.,enp0s.).
4. skip-interface= Use the first valid IP address on the first enumerated interface (same logic as first-found above) that does NOT match with any of the specified interface name regexes. Regexes are separated by commas (e.g. eth.,enp0s.).

Notes: In order to deploy Luigi Operator as part of the bootstrap process via qbert-api the networkPlugin *_entry allowed to use is *_calico.

Python Payload Example

cluster_create_params = {

 "name":"test-call-NetworkOperator",

 "externalDnsName":"10.128.146.127",

 "containersCidr":"10.20.0.0/16",

 "servicesCidr":"10.21.0.0/16",

 "mtuSize":1440,

 "privileged":True,

 "appCatalogEnabled":False,

 "nodePoolUuid":"3361eead-a5ce-435d-b9f6-8f4dee0621aa",

 "calicoIpIpMode":"Always",

 "calicoNatOutgoing":True,

 "calicoV4BlockSize":"24",

 "calicoIPv4DetectionMethod":"first-found",

 "networkPlugin":"calico",

 "deployLuigiOperator":True,

 "runtimeConfig":"api/all=true",   "etcdBackup":{"storageType":"local","isEtcdBackupEnabled":1,"storageProperties":{"localPath":"/etc/pf9/etcd-backup"},"intervalInMins":1440},

    "tags":{"pf9-system:monitoring":"true"}

}

Python Snippet to Bootstrap Cluster

Prerequisites

The easiest way to use this script is by deploying a virtual environment in a docker container, so please follow the next steps in order to set up the environment.

docker run -i -t --name python-bootstrapper centos:centos7 bash

Inside the container update packages and install python3 and python3-pip

yum update -y

yum python3 python3-pip

Create a virtual environment

python3 -m venv pf9-cluster-virtualenv

Activate virtual environment

source pf9-cluster-virtualenv/bin/activate

Requirements file

cat <<EOF > requirements.txt

certifi==2020.6.20

chardet==3.0.4

click==7.1.2

debtcollector==2.2.0

httplib2==0.18.1

idna==2.10

importlib-metadata==1.7.0

importlib-resources==3.0.0

iso8601==0.1.12

keystoneauth1==4.2.1

msgpack==1.0.0

netaddr==0.8.0

netifaces==0.10.9

oauth2client==4.1.3

os-service-types==1.7.0

oslo.config==8.3.1

oslo.i18n==5.0.0

oslo.serialization==4.0.0

oslo.utils==4.5.0

packaging==20.4

pbr==5.5.0

pyasn1==0.4.8

pyasn1-modules==0.2.8

pyparsing==2.4.7

python-keystoneclient==4.1.0

pytz==2020.1

PyYAML==5.3.1

qbertclient==0.0.12

requests==2.24.0

rfc3986==1.4.0

rsa==4.6

six==1.15.0

stevedore==3.2.1

urllib3==1.25.10

wrapt==1.12.1

zipp==3.1.0

EOF

Install module requirements.

pip install requirements.txt

Create python bootstrap script

Create python deployer script and update the parameters of DU_NAME, TENANT_NAME, TENANT_ID,USER, PASSWORD,NODE_POOL, MASTER_NODE_ID, WORKER1_NODE_ID, WORKER2_NODE_ID

cat <<EOF > cluster_deploy_with_luigi.py

from qbertclient import qbert

from qbertclient import keystone

import time

import requests

import json

du_fqdn = "DU_NAME"

username ="USERt"

password = "PASSWORD"

project_name = "TENANT_NAME"

keystone = keystone.Keystone(du_fqdn, username, password, project_name)

token = keystone.get_token()

project_id = keystone.get_project_id(project_name)

headers = {'X-Auth-Token' : token,'Content-Type': "application/json"}

cluster_create_params = {

    "name":"test-cal",

 "externalDnsName":"10.128.146.127",

 "containersCidr":"10.20.0.0/16",

 "servicesCidr":"10.21.0.0/16",

 "mtuSize":1440,

 "privileged":True,

 "appCatalogEnabled":False,

 "nodePoolUuid":"NODE_POOL",

 "calicoIpIpMode":"Always",

 "calicoNatOutgoing":True,

 "calicoV4BlockSize":"24",

 "calicoIPv4DetectionMethod":"first-found",

 "networkPlugin":"calico",

 "deployLuigiOperator":True,

 "runtimeConfig":"api/all=true",   "etcdBackup":{"storageType":"local","isEtcdBackupEnabled":1,"storageProperties":{"localPath":"/etc/pf9/etcd-backup"},"intervalInMins":1440},

    "tags":{"pf9-system:monitoring":"true"}

}

cluster_create = requests.post('https://DU_NAME/qbert/v3/TENANT_ID/clusters',data=json.dumps(cluster_create_params),headers = headers)

if cluster_create.status_code == 200:

     cc_json = json.loads(cluster_create.text)

     new_cluster_uuid = cc_json["uuid"]

     print(new_cluster_uuid)

node_list3 = [{

        "uuid": MASTER_NODE_UUID",

        "isMaster": True

        },

         {

        "uuid": "WORKER1_NODE_UUID",

            "isMaster": False

        },

        {

        "uuid": "WORKER2_NODE_UUID",

            "isMaster": False

        }

    ]

attach_nodes = requests.post('https://DU_NAME/qbert/v3/{}/clusters/{}/attach'.format(project_id,new_cluster_uuid),data=json.dumps(node_list3),headers = headers)

print(attach_nodes.status_code)

if attach_nodes.status_code == 200:

    print("Nodes are attachted")

​

EOF

Create new cluster via API

python cluster_deploy_with_luigi.py

Tips

MacVLAN

• When declaring the network attach definitions the master section can not use the same physical/virtual/vlan interface of another network-attach-definition that is being used for ipvlan.

IPVLAN

• In order for kubelet to create pods with ipvlan interface types Kernel version 4.1+ should be installed across all the nodes of the cluster, please follow the instructions to install Kernel 4.1+ on CentOS7

$ rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

$ rpm -Uvh https://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm

$ yum list available --disablerepo='*' --enablerepo=elrepo-kernel

$ yum --enablerepo=elrepo-kernel install kernel-lt

$ reboot

# SELECT NEW INSTALLED KERNEL TO BOOT WITH

$ cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-4.4.237-1.el7.elrepo.x86_64 root=UUID=6cd50e51-cfc6-40b9-9ec5-f32fa2e4ff02 ro console=tty0 console=ttyS0,115200n8 crashkernel=auto net.ifnames=0 console=ttyS0 LANG=en_US.UTF-8 intel_iommu=on iommu=pt

Kube-sriov-device-plugin

• A know issue with sriov-device-plugin pod that runs on every node is that if you make a change on a hostconfig object that will match a resource definition in your sriov-config map that links to an sriov networkattachdefinition the allocatable resources will not change; in order the sriov-device-plugin pod to re-read the new VFs resources and update the networkattach definition allocatable resources the sriov-device-plugin pod needs to be recreates by simply deleting the pod and let the daemonset to take care of it. https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/issues/276

SRIOV - DPDK

• NetworkManager needs to be disabled since NetworkManager auto DHCP all the Virtual Functions.
• Due to the way VFIO Driver works, there are certain limitations to which devices can be used with VFIO. Mainly it comes down to how IOMMU groups work. Any Virtual Function device can be used with VFIO on its own, but physical devices will require either all ports bound to VFIO, or some of them bound to VFIO while others not being bound to anything at all. If your device is behind a PCI-to-PCI bridge, the bridge will then be part of the IOMMU group in which your device is in. Therefore, the bridge driver should also be unbound from the bridge PCI device for VFIO to work with devices behind the bridge.
• IPAM not valid for DPDK enabled networks, see SRIOV-CNI section on DPDK: https://github.com/intel/sriov-cni
• In order for the test DPDK application to work successfully, you need hugepages enabled at the host level, you can enable it on CentOS7 by editing /etc/default/grub and add the following kernel boot parameters to enable iommu and create 8GB of 2M size hugepages. https://github.com/openshift/sriov-network-device-plugin/blob/master/docs/dpdk/README.md

GRUB_CMDLINE_LINUX="nofb nomodeset vga=normal iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=16"

#Rebuild grub.cfg

grub2-mkconfig -o /boot/grub2/grub.cfg && reboot

Repurpose Worker Nodes for a New Cluster

In order to repurpose a worker node once it has been dissociated from the cluster you should perform the following commands in order to fully clean the node.

yum erase -y pf9-hostagent && rm -rf /opt/pf9 && rm -rf /etc/pf9 && rm -rf /var/opt/pf9 && rm -rf /var/log/pf9 && rm -rf /tmp/* && rm -rf /var/spool/mail/pf9 && rm -rf /opt/cni/bin/* && rm -rf /etc/cni && rm -rf /var/log/messages-* && rm -rf /var/lib/docker && yum clean all && rm -rf /opt/cni && rm -rf /etc/cni

References

SR-IOV - DPDK Drivers

https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html

https://github.com/ceph/dpdk/blob/master/tools/dpdk-devbind.py

NetworkAttachDefinition Examples

https://github.com/intel/sriov-network-device-plugin#configurations