VF's PCI Address is Exposed as an Env Variable and Can be Consumed by Userspace DPDK Applications Di

Problem

Multus interface is not getting created when using type "sriov".

Pod Network Output

k8s.v1.cni.cncf.io/networks: sriovwhereabout
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "10.40.242.136"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "kube-system/sriovwhereabout",
                    "interface": "net1",
                    "ips": [
                        "172.21.14.50"
                    ],
                    "dns": {},
                    "device-info": {
                        "type": "pci",
                        "version": "1.0.0",
                        "pci": {
                            "pci-address": "0000:af:04.5"

Pod Interface Output

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if306: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether 66:2c:23:55:0b:31 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.242.136/32 brd 10.20.242.136 scope global eth0
       valid_lft forever preferred_lft forever

Environment

• Platform9 Edge Cloud - All Versions

• SRIOV

• DPDK

Answer

The vfio-pci DPDK driver used in this case bypasses the linux layer as the vfio interfaces are userspace interfaces and one cannot see them as kernel devices (ip link). The VF's pci address is exposed as an env variable and can be consumed by userspace dpdk applications directly.

SRIOV Pod Describe Output

$ cat pod-describe.txt

Name:         sriov-deployment-86c6b7bcbc-w9fh9
Namespace:    kube-system
Priority:     0
Node:         xyz-master0/10.69.44.199
Start Time:   Mon, 13 Jun 2022 13:34:20 +0000
Labels:       app=sriov-pod
              pod-template-hash=86c6b7bcbc
Annotations:  cni.projectcalico.org/podIP: 10.40.242.136/32
              cni.projectcalico.org/podIPs: 10.40.242.136/32
              k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "10.40.242.136"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "kube-system/sriovwhereabout",
                    "interface": "net1",
                    "ips": [
                        "172.21.14.50"
                    ],
                    "dns": {},
                    "device-info": {
                        "type": "pci",
                        "version": "1.0.0",
                        "pci": {
                            "pci-address": "0000:af:04.5"
                        }
                    }
                }]
              k8s.v1.cni.cncf.io/networks: sriovwhereabout
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "10.40.242.136"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "kube-system/sriovwhereabout",
                    "interface": "net1",
                    "ips": [
                        "172.21.14.50"
                    ],
                    "dns": {},
                    "device-info": {
                        "type": "pci",
                        "version": "1.0.0",
                        "pci": {
                            "pci-address": "0000:af:04.5"
                        }
                    }
                }]
...

SRIOV Pod Inspect Output

$ cat sriov-pod-docker-inspect.txt | grep -i -C 1 env

            "StdinOnce": false,
            "Env": [
                "PCIDEVICE_INTEL_COM_INTEL_SRIOV3_ETH10=0000:af:04.5",

Additional Information

You can refer to sriov-network-device-plugin/issues/401 for additional information.