Luigi Networking Quickstart

This document makes the following assumptions, you are running a BareOS PMK 4.5.1 Kubernetes cluster v 1.17+ , the Calico CNI is enabled, privileged pods options is enabled, no MetalLB is installed and every worker node should have an additional nic/port up and running with no ip assigned to it if intended for ipvlan/l2 and macvlan, or an additional interface with an ip for ipvlan/l3.

Luigi

Luigi Installation

The first step is to download the Luigi Operator definition in your master node or from where you are able to execute kubectl commands against this cluster, a local copy of the manifest is at the end of the document.

As of 4.5, this is available through qbert API as well. Please go to the Bootstrapping Cluster with Network Operator section.

Then simply install it by using the following command:

$ kubectl apply -f luigi-plugins-operator.yaml

Luigi Install Validation

After executing the command above, let’s review our work, please consider that some of the pods are going to take some time to fully be up, let’s a couple of minutes.

pf9-0102:~ carlos$  kubectl get all -n luigi-system

NAME                                            READY   STATUS    RESTARTS   AGE

pod/luigi-controller-manager-74bdbf9cc9-2g2tf   2/2     Running   0          41h

​

NAME                                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

service/luigi-controller-manager-metrics-service   ClusterIP   10.168.113.206   <none>        8443/TCP   41h

​

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/luigi-controller-manager   1/1     1            1           41h

​

NAME                                                  DESIRED   CURRENT   READY   AGE

replicaset.apps/luigi-controller-manager-74bdbf9cc9   1         1         1       41h

Luigi Networking Plugins

Luigi Networking Plugins Installation (hostPlumber)

The sampleplugins.yaml manifest will help to deploy or undeploy different CNI plugins, in our K8s cluster, the different components are going to be deployed as daemon sets and will run in every node including workers and masters.

Please use the following manifest sampleplugins.yaml

apiVersion: plumber.k8s.pf9.io/v1

kind: NetworkPlugins

metadata:

  name: networkplugins-sample11

spec:

  # Add fields here

  plugins:

    hostPlumber: {}

      #hostPlumberImage: "platform9/luigi-plumber:v0.1.0"

    #nodeFeatureDiscovery: {}

    #multus: {}

    #whereabouts: {}

    # COMMENT/UNCOMMENT - requires all host networking/VFs to be configured first

    #sriov: {}

As you may have observed all the plugins are commented out but hostPlumber, first we will install hostPlumber plugin and we will revisit this manifest later and re-apply it with the rest of the CNI plugins uncommented after creating our first HostNetworkTemplate object. HostPlumber is not required, but it provides a mechanism to configure SRIOV and host networking, as well as view the Node SRIOV and interface state via a K8S operator. If you’re configuring host networking and SRIOV VFs thru other means, hostPlumber plugin is not needed and can be skipped

The reason we deploy this first, is that the SRIOV CNI requires VF and network configuration to be done first before deploying the SRIOV plugin.

Execute the following command to install hostPlumber plugin

$ kubectl  apply -f sampleplugins.yaml

Luigi Networking Plugins Install Validation (hostPlumber)

After executing the command above, let’s review our work.

# pf9-0102:~ carlos$ kubectl get ds -n kube-system

NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE

calico-node                      4         4         4       4            4           kubernetes.io/os=linux          2d9h

hostconfig-controller-manager    4         4         4       4            4           <none>                          41h

Luigi HostNetworkTemplate Object

As mentioned before, HostNetworkTemplate will help us to enable the number of VirtualFunctions in our nodes as well which driver will be loaded on the newly enabled VirtualFunctions, on this new release we have added the nodeSelector feature in order to perform SRIOV configurations only on SRIOV capable nodes based Labels .

Note: NFD plugin (Node Feature Discovery) should be installed first in order to leverage on the feature.node.kubernetes.io/network-sriov.capable: "true" label

For example:

apiVersion: plumber.k8s.pf9.io/v1

kind: HostNetworkTemplate

metadata:

  name: sriovconfig-enp3s0f1

spec:

  # Add fields here

  nodeSelector:

    feature.node.kubernetes.io/network-sriov.capable: "true"

    testlabelA: "123"

  sriovConfig:

    - pfName: enp3s0f1

      numVfs: 8

      vfDriver: vfio-pci

      mtu: 9000

---

apiVersion: plumber.k8s.pf9.io/v1

kind: HostNetworkTemplate

metadata:

  name: sriovconfig-enp3s0f0

spec:

  # Add fields here

  sriovConfig:

    - pfName: enp3s0f0

      numVfs: 4

      vfDriver: ixgbev

In this definition, we are creating two HostNetworkTemplates one for sriov using vfio-pci drivers and the other one sriov using kernel drivers, the first one will only configure the VFs on nodes that has 3 things, SRIOV capable, a PF called enp3s0f1 and a label with key-value to testlabelA: "123"

The second will attempt to configure 4 VFs with the ixgbevf drive across all the nodes no matter what labels they have.

It is also possible to merge the two sriovConfig’s for the two interfaces into the same HostNetworkTemplate CRD, rather than a separate one for each PF. For example:

apiVersion: plumber.k8s.pf9.io/v1

kind: HostNetworkTemplate

metadata:

  name: HostNetworkTemplate-kernel-enp3

spec:

  # Add fields here

  nodeSelector:

    feature.node.kubernetes.io/network-sriov.capable: "true"

    testlabelA: "123"

  sriovConfig:

    - pfName: enp3s0f1

      numVfs: 8

      vfDriver: ixgbevf

      mtu: 9000

    - pfName: enp3s0f0

      numVfs: 4

      vfDriver: ixgbevf

Configuring via vendor and device ID

The above will search for all interfaces matching vendor ID 8086 (Intel) and device ID 1528 (representing a particular model of NIC). It will then create 32 VFs on each matching device and bind all of them to the vfio-pci (DPDK driver). This might be useful if you don’t know the interface naming scheme across your hosts or PCI addresses, but you just want to target all particular NIC by vendor and device ID.

apiVersion: plumber.k8s.pf9.io/v1

kind: HostNetworkTemplate

metadata:

  name: HostNetworkTemplate-1528-dev

spec:

  # Add fields here

  nodeSelector:

    feature.node.kubernetes.io/network-sriov.capable: "true"

    testlabelA: "123"

  sriovConfig:

    - vendorId: "8086"

      deviceId: "1528"

      numVfs: 32

      vfDriver: vfio-pci

Configuring via PCI address

apiVersion: plumber.k8s.pf9.io/v1

kind: HostNetworkTemplate

metadata:

  name: HostNetworkTemplate-sample

spec:

  # Add fields here

  nodeSelector:

    feature.node.kubernetes.io/network-sriov.capable: "true"

    testlabelA: "123"

  sriovConfig:

    - pciAddr: 0000:03:00.0

      numVfs: 32

      vfDriver: vfio-pci

    - pciAddr: 0000:03:00.1

      numVfs: 32

      vfDriver: vfio-pci

The above will configure 32 VFs on PF matching PCI address “0000:03:00.0” and 32 VFs on PCI address “0000:03.00.1”, for a total of 64 VFs, and bind each VF to the vfio-pci driver.

HostNetwork CRD

This is not to be confused with the status section of the HostNetworkTemplate CRD(For latter phases, each Node will append it’s nodename and the sucess/failure of application of the HostNetworkTemplate policy to the Status section)

The HostNetwork CRD, which is different, will not be created by the user. Instead, this is intended to be a read-only CRD and the Daemonset operator on each node will discover and populate various host settings to this CRD:

• Created: First upon the HostPlumber plugin being deployed

• Updated: After each application of the HostNetworkTemplate CRD

• For Phase3 and later: Monitored as a periodic task, updating and discovering host changes

There will be one HostNetwork CRD automatically created for each node. The Name will correspond to the Node’s name, which in PMK is the IP.

[root@arjunpmk-master ~]$ kubectl get HostNetwork -n luigi-system

NAME             AGE

10.128.144.14    28s

10.128.144.43    25s

10.128.237.203   27s

10.128.237.204   26s

Fetching the first Node in -o yaml reveals

[root@arjunpmk-master ~]$ kubectl get HostNetwork 10.128.237.204 -n luigi-system -o yaml

apiVersion: plumber.k8s.pf9.io/v1

kind: HostNetwork

metadata:

  creationTimestamp: "2020-10-20T08:22:58Z"

  generation: 1

  name: 10.128.237.204

  namespace: luigi-system

  resourceVersion: "3751"

  selfLink: /apis/plumber.k8s.pf9.io/v1/namespaces/luigi-system/HostNetworks/10.128.237.204

  uid: 13bfa4ca-2546-43dc-8074-9943f054674e

spec:

  interfaceStatus:

  - currentSriovConfig:

      totalVfs: 63

    deviceId: "1528"

    mac: a0:36:9f:43:54:54

    mtu: 1500

    pciAddr: "0000:03:00.0"

    pfDriver: ixgbe

    pfName: enp3s0f0

    sriovEnabled: true

    vendorId: "8086"

  - currentSriovConfig:

      totalVfs: 63

    deviceId: "1528"

    mac: a0:36:9f:43:54:56

    mtu: 9000

    pciAddr: "0000:03:00.1"

    pfDriver: ixgbe

    pfName: enp3s0f1

    sriovEnabled: true

    vendorId: "8086"

status: {}

This node just has one interface, eth0, using virtio driver. This node happens to be an Openstack VM. Let’s look at a bare metal SRIOV node

[root@arjunpmk-master ~]# kubectl get HostNetwork 10.128.237.204 -n default -o yaml

apiVersion: plumber.k8s.pf9.io/v1

kind: HostNetwork

metadata:...​

spec:...​

status: {}

There are 2 interfaces. Each of them has 4 VFs. For each PF, and each VF underneath, you can see the device and vendor IDs, PCI addresses, driver enabled, MAC address allocated (if assigned to a Pod), VLAN, and other Link level information.

In future phases, more information will be reported such as IP information, ip-route information, and other networking related host-state.

Validate Luigi HostNetworkTemplate Object (hostPlumber)

$ pf9-0102:DPDK carlos$ kubectl get hostnetworktemplate

NAME                            AGE

sriovconfig-enp3s0f0-kernel     13s

sriovconfig-enp3s0f1-vfio-pci   13s

Luigi Networking Plugins Installation (multus, sriov, whereabouts, and nodeFeatureDiscovery)

Let’s revisit sampleplugins.yaml and uncomment the rest of the lines and reapply the manifest, this will install multus, sriov and whereabouts.

$ kubectl apply -f sampleplugins.yaml

apiVersion: plumber.k8s.pf9.io/v1

kind: NetworkPlugins

metadata:

  name: networkplugins-sample11

spec:

  # Add fields here

  plugins:

    hostPlumber: {}

      #hostPlumberImage: "platform9/luigi-plumber:v0.1.0"

    nodeFeatureDiscovery: {}

    multus: {}

    whereabouts: {}

    # COMMENT/UNCOMMENT - requires all host networking/VFs to be configured first

    sriov: {}

Validate Luigi Networking Plugins Installation (multus,sriov,whereabouts and nodeFeatureDiscovery)

Let’s review our work by listing the new daemonsets created in the kube-system namespace.

$ pf9-0102:DPDK carlos$ kubectl get ds -n kube-system

NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE

calico-node                      4         4         4       4            4           kubernetes.io/os=linux          2d10h

hostconfig-controller-manager    4         4         4       4            4           <none>                          42h

kube-multus-ds-amd64             4         4         4       4            4           kubernetes.io/arch=amd64        46h

kube-sriov-cni-ds-amd64          4         4         4       4            4           beta.kubernetes.io/arch=amd64   47h

kube-sriov-device-plugin-amd64   4         4         4       4            4           beta.kubernetes.io/arch=amd64   47h

whereabouts                      4         4         4       4            4           beta.kubernetes.io/arch=amd64   47h

​

$ pf9-0102:DPDK carlos$ kubectl get ds -n node-feature-discovery

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE

nfd-worker   3         3         3       3            3           <none>          66s

Note: The kube-sriov-device-plugin-amd64 pods will be in CreatingContainer or Pending state since the sriov-config-map hasn’t been created the sriov-config-map is created in the sriov section of this document.