Enable SRIOV - Kernel Driver
SRIOV and DPDK
SRI-IOV WorkerNode Prerequisites
In order for the cni-sriov-plugin to start reading Virtual Functions resources as well HostNetworkTemplate objects can configure virtual functions you need to perform the following steps.
- Enable** VT-D** at BIOS level.
- Enable SRIOV feature globally and/or per nic according to the nic you are working with, for example Intel x710 should be enabled per nic at BIOS level.
- Upgrade Kernel boot line to enable intel_iommu=on and iommu=pt
Huge page Support
Most network applications use huge pages, so you may want to enable that. Please edit /etc/default/grub and add huge-pages
GRUB_CMDLINE_LINUX="nofb nomodeset vga=normal iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=16"#Rebuild grub.cfggrub2-mkconfig -o /boot/grub2/grub.cfg && rebootCreate SR-IOV ConfigMap
SR-IOV ConfigMap
Please copy paste the following definition update the values of your Physical Functions and ranges or Virtual Functions as well as Drivers and apply it to our cluster, in case that you have multiple nics, that each one uses different kernels drivers please omit the drivers section and just mentioned pfName within the selectors section.
SR-IOV - Kernel Driver ConfigMap
apiVersionv1kindConfigMapmetadata namesriovdp-config namespacedefaultdata config.json { "resourceList": [{ "resourceName": "intel_sriov_kernel0", "selectors": { "pfNames": ["em1#0-6"], "drivers": ["igbvf"] } } ] }Create Network Attach Definition SR-IOV type
Please copy paste the following definition update the values of your subnet and apply it to our cluster:
apiVersion"k8s.cni.cncf.io/v1"kindNetworkAttachmentDefinitionmetadata namesriov-kernelnet0 annotations k8s.v1.cni.cncf.io/resourceNameintel.com/intel_sriov_kernel0spec config'{ "type""sriov", "cniVersion""0.3.1", "name""sriov-kernelnet0", "spoofchk""off", "type""sriov", "vlan"80, "ipam" "type""whereabouts" "range""192.168.80.0/24" "range_start""192.168.80.20" "range_end""192.168.80.50" "gateway""192.168.80.1" 'The master key value is the reference to the *_second nic *_in our worker nodes.
Network Attach Definition Validation SR-IOV type
Let’s validate our work by listing and describing our new Network Attach Definition
$ kubectl get net-attach-defNAME AGEsriov-net-a 5d21h $ kubectl describe net-attach-def sriov-kernelnet0Name: sriov-kernelnet0Namespace: defaultLabels: <none>Annotations: k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_kernel0 kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{"k8s.v1.cni.cncf.io/resourceName":"inte...API Version: k8s.cni.cncf.io/v1Kind: NetworkAttachmentDefinitionMetadata: Creation Timestamp: 2020-09-23T18:53:31Z Generation: 1 Resource Version: 174547 Self Link: /apis/k8s.cni.cncf.io/v1/namespaces/default/network-attachment-definitions/sriov-kernelnet0 UID: c6fd410a-97d8-42d4-86d5-56ef447e1dfaSpec: Config: { "type": "sriov", "name": "sriov-kernelnet0", "spoofchk": "off", "type": "sriov", "vlan": 80, "ipam": { "type": "whereabouts", "range": "192.168.80.0/24", "range_start": "192.168.80.20", "range_end": "192.168.80.50", "gateway": "192.168.80.1" } }Events: <none>Create Pods with SR-IOV interfaces (Kernel-Driver)
$ cat pod0-case3.yamlapiVersion: v1kind: Podmetadata: name: pod0-case-03-sriov-kernel annotations: k8s.v1.cni.cncf.io/networks: sriov-kernelnet0spec: containers: - name: pod0-case-03 image: docker.io/centos/tools:latest command: - /sbin/init resources: requests: intel.com/intel_sriov_kernel0: '1' limits: intel.com/intel_sriov_kernel0: '1'Deploy the new pods
$ kubectl apply -f pod0-case3.yamlValidate Pods Creation with SR-IOV interfaces (Kernel Driver)
Let’s validate your work by confirming that the pods got created with an additional interface by doing the following commands:
$ kubectl get pods -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod0-case-02 1/1 Running 0 66m 10.135.1.147 192.168.50.14 <none> <none>pod1-case-02 1/1 Running 0 62m 10.135.1.148 192.168.50.14 <none> <none>x
$ kubectl exec -it pod0-case-02 -- ip -d address1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 numtxqueues 1 numrxqueues 1 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever3: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc noqueue state UP group default link/ether de:ff:c2:57:c6:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth numtxqueues 1 numrxqueues 1 inet 10.135.1.147/24 brd 10.135.1.255 scope global eth0 valid_lft forever preferred_lft forever4: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether fa:16:3e:10:31:25 brd ff:ff:ff:ff:ff:ff promiscuity 0 ipvlan mode l2 numtxqueues 1 numrxqueues 1 inet 192.168.80.20/24 brd 192.168.70.255 scope global net1 valid_lft forever preferred_lft forever$ kubectl exec -it pod1-case-02 -- ip -d address1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 numtxqueues 1 numrxqueues 1 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever3: eth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc noqueue state UP group default link/ether aa:2d:2e:e8:5e:19 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth numtxqueues 1 numrxqueues 1 inet 10.135.1.148/24 brd 10.135.1.255 scope global eth0 valid_lft forever preferred_lft forever4: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether fa:16:3e:10:31:25 brd ff:ff:ff:ff:ff:ff promiscuity 0 ipvlan mode l2 numtxqueues 1 numrxqueues 1 inet 192.168.80.21/24 brd 192.168.70.255 scope global net1 valid_lft forever preferred_lft foreverSR-IOV VF Testing
SR-IOV Kernel Driver Validation
$ kubectl exec pod0-case-03 -- ethtool -i net2driver: igbvfversion: 2.4.0-kfirmware-version:expansion-rom-version:bus-info: 0000:01:10.2supports-statistics: yessupports-test: yessupports-eeprom-access: nosupports-register-dump: yessupports-priv-flags: no$ kubectl exec pod0-case-03 -- ethtool -i net1driver: igbvfversion: 2.4.0-kfirmware-version:expansion-rom-version:bus-info: 0000:01:10.0supports-statistics: yessupports-test: yessupports-eeprom-access: nosupports-register-dump: yessupports-priv-flags: noWas this page helpful?
On This Page
Enable SRIOV - Kernel DriverSRIOV and DPDKSRI-IOV WorkerNode PrerequisitesHuge page SupportCreate SR-IOV ConfigMapSR-IOV ConfigMapSR-IOV - Kernel Driver ConfigMapCreate Network Attach Definition SR-IOV typeNetwork Attach Definition Validation SR-IOV typeCreate Pods with SR-IOV interfaces (Kernel-Driver)Validate Pods Creation with SR-IOV interfaces (Kernel Driver)SR-IOV VF TestingSR-IOV Kernel Driver Validation