Enable SRIOV with DPDK

SRIOV and DPDK

SRI-IOV WorkerNode Prerequisites

In order for the cni-sriov-plugin to start reading Virtual Functions resources as well HostNetworkTemplate objects can configure virtual functions you need to perform the following steps.

Enable** VT-D** at BIOS level.
Enable SRIOV feature globally and/or per nic according to the nic you are working with, for example Intel x710 should be enabled per nic at BIOS level.
Upgrade Kernel boot line to enable intel_iommu=on and iommu=pt

Huge page Support

Most network applications use huge pages, so you may want to enable that. Please edit /etc/default/grub and add huge-ages

Bash
    
 
GRUB_CMDLINE_LINUX="nofb nomodeset vga=normal iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=16"#Rebuild grub.cfggrub2-mkconfig -o /boot/grub2/grub.cfg && reboot
Copy

Please copy paste the following definition update the values of your Physical Functions and ranges or Virtual Functions as well as Drivers and apply it to our cluster, in case that you have multiple nics, that each one uses different kernels drivers please omit the drivers section and just mentioned pfName within the selectors section.

SR-IOV Kernel Driver plus DPDK ConfigMap

YAML
    
 
apiVersion: v1kind: ConfigMapmetadata:  name: sriovdp-config  namespace: kube-systemdata:  config.json: |    {        "resourceList": [            {                "resourceName": "intel_sriov_dpdk0",                "selectors": {                    "drivers": ["vfio-pci"],                    "pfNameis": ["enp3s0f0#0-7"]                }            },            {                "resourceName": "intel_sriov_kernel0",                "selectors": {                    "drivers": ["ixgbevf"],                    "pfNames": ["enp3s0f0#8-15"]                }            },            {                "resourceName": "intel_sriov_dpdk1",                "selectors": {                    "drivers": ["vfio-pci"],                    "pfNames": ["enp3s0f1#0-7"]                }            },            {                "resourceName": "intel_sriov_kernel1",                "selectors": {                    "drivers": ["ixgbevf"],                    "pfNames": ["enp3s0f1#8-15"]                }            }        ]    }
Copy

Create NetworkAttachDefinition of DPDK type

YAML
    
 
kind: NetworkAttachmentDefinitionmetadata:  name: sriov-net-a  annotations:    k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_dpdk_aspec:  config: '{  "type": "sriov",  "cniVersion": "0.3.1",  "name": "sriov-network-a",  "vlan": 100,  "vlanQoS": 1,  "max_tx_rate": 100,  "spoofchk": "off",  "trust": "on"}'
Copy

Network Attach Definition Validation SR-IOV-DPDK type

Let’s validate our work by listing and describing our new Network Attach Definition

Bash
    
 
$ kubectl  get net-attach-defNAME                      AGEsriov-dpdk0               15s
Copy

Bash
    
$  kubectl describe net-attach-def sriov-dpdk0Name:         sriov-dpdk0Namespace:    defaultLabels:       <none>Annotations:  k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_dpdk0              kubectl.kubernetes.io/last-applied-configuration:                {"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{"k8s.v1.cni.cncf.io/resourceName":"inte...API Version:  k8s.cni.cncf.io/v1Kind:         NetworkAttachmentDefinitionMetadata:  Creation Timestamp:  2020-10-05T20:25:57Z  Generation:          1  Resource Version:    2310115  Self Link:           /apis/k8s.cni.cncf.io/v1/namespaces/default/network-attachment-definitions/sriov-dpdk0  UID:                 5307140a-3771-41b4-845e-1aa8460acffcSpec:  Config:  { "cniVersion": "0.3.1", "name": "sriov-dpdk0", "type": "sriov", "vlan": 90 }Events:    <none>
Copy

Create Pods with SR-IOV interfaces (DPDK Driver - vfio-pci)

YAML
    
 
# cat sriov-pod-2.yamlapiVersion: v1kind: Podmetadata:  name: sriov-pod-2  annotations:    k8s.v1.cni.cncf.io/networks: sriov-net-aspec:  containers:  - name: sriov-example    image: rparikh/dpdk-app-centos    securityContext:      privileged: true    volumeMounts:    - mountPath: /etc/podnetinfo      name: podnetinfo      readOnly: false    - mountPath: /dev/hugepages      name: hugepage    resources:      requests:        memory: 1Gi        #cpu: "4"        intel.com/intel_sriov_dpdk_a: '1'      limits:        hugepages-1Gi: 2Gi        #cpu: "4"        intel.com/intel_sriov_dpdk_a: '1'    # Uncomment to control which DPDK App is running in container.    # If not provided, l3fwd is default.    #   Options: l2fwd l3fwd testpmd    env:    - name: DPDK_SAMPLE_APP      value: "testpmd"    #    # Uncomment to debug DPDK App or to run manually to change    # DPDK command line options.    command: ["sleep", "infinity"]  volumes:  - name: podnetinfo    downwardAPI:      items:        - path: "labels"          fieldRef:            fieldPath: metadata.labels        - path: "annotations"          fieldRef:            fieldPath: metadata.annotations  - name: hugepage    emptyDir:      medium: HugePages
Copy

Deploy the new pods

Bash
    
 
$ kubectl apply -f sriov-pod-2.yaml
Copy

Validate Pods Creation with SR-IOV interfaces (DPDK - Driver vfio-pci)

Let’s validate your work by confirming that the created pods got successfully created by doing the following commands:

Bash
    
 
$ kubectl get pods -o wideNAME          READY   STATUS    RESTARTS   AGE     IP              NODE             NOMINATED NODE   READINESS GATESsriov-pod-2   1/1     Running   0          5d21h   10.167.181.11   10.128.237.204   <none>           <none>
Copy

SR-IOV DPDK Driver - vfio-pci Validation

Bash
    
 
$ kubectl exec -it sriov-pod-2 -- shsh-4.4# dpdk-appENTER dpdk-app: argc=1 dpdk-app  cpuRsp.CPUSet = 0-31  Interface[0]:    IfName=""  Name=""  Type=SR-IOV    MAC=""  IP="10.143.181.4"    PCIAddress=0000:03:10.7 myArgc=14 dpdk-app -n 4 -l 1 --master-lcore 1 -w 0000:03:10.7 -- -p 0x1 -P --config="(0,0,1)" --parse-ptype
Copy

Last updated on

Was this page helpful?