How to Implement PCI Requirement Of “Data Encryption In Flight” In Kubernetes Clusters
As I talk with customers and prospects that are interested in Platform9 Managed Kubernetes offering, PCI requirements come up frequently. One critical PCI requirement is data encryption in flight within the Kubernetes cluster. This blog is a guided tutorial on exactly how to implement data encryption within a k8s cluster using Calico and WireGuard on top of Platform9 Managed Kubernetes.
Installation
Create a PMK Cluster
The following values were used to create a PMK Cluster:
Once that we have our PMK Cluster up and running, only two steps are required to enable wireguard on a Kubernetes cluster, WireGuard at OS level & WireGuard at Kubernetes, let’s go through each of these steps.
WireGuard at OS (CentOS 7 in this example)
Let’s install the prerequisites needed according to your operating system on each of the nodes forming your PMK cluster this involves masters and workers, let’s follow the next link https://www.wireguard.com/install, reboot your nodes as necessary according to the packages altered, in this environment we are using Centos 7.
$ sudo yum install epel-release elrepo-release
$ sudo yum install yum-plugin-elrepo
$ sudo yum install kmod-wireguard wireguard-toolsWireGuard at Kubernetes
In order to enable Wireguard at our PMK cluster, we should patch the Calico Felix configuration right after our nodes have the wireguard packages installed at the O.S level
To patch the Felix configuration you will need to leverage on calicoctl binary, this binary needs a kubeconfig file to work properly, by default Platform9 ships this binary in the master nodes, you just need to create a folder structure for calico and provide a kubeconfig file, Platform9 also provides an internal admin kubeconfig for other internal purposes to demonstrate the WireGuard feature you can leverage on the next steps to achieve it.
NOTE: calicoctl can also be used outside the cluster as long as you provide a kubeconfig file that targets your PMK cluster.
Create calico folder path
[root@master00 ~]# mkdir /etc/calicoCreate Calico configuration file
[root@master00 ~]# cat <<'EOF' > /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
datastoreType: "kubernetes"
kubeconfig: "/etc/pf9/kube.d/kubeconfigs/admin.yaml"
EOF
Now Let’s patch the felixconfiguration to enable WireGuard across all the clusters:
[root@master00 ~]# /opt/pf9/pf9-kube/bin/calicoctl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":true}}'Testing
Let’s validate that Public Keys have been generated on each of our nodes, by running the following command
$ calicoctl get node <NODE-NAME> -o yaml
...
status:
...
wireguardPublicKey: jlkVyQYooZYzI2wFfNhSZez5eWh44yfq1wKVjLvSXgY=
...
Let’s deploy a couple of test pods to validate that east-west traffic is being encrypted, in this environment, we will be using the following manifests as the pods originating and receiving the traffic.
Deploy some pods that we will use to generate traffic from the CentOS pods towards the Nginx pods.
[root@master00 ~]# cat << 'EOF' > wireguard-deploy-pf9.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: wireguard-test-deploy
spec:
replicas: 4
selector:
matchLabels:
app: wireguard
template:
metadata:
labels:
app: wireguard
spec:
containers:
- name: pf9-tooler
image: docker.io/centos/tools:latest
command:
- /sbin/init
EOF[root@master00 ~]# kubectl apply -f wireguard-deploy-pf9.yamlDeploy some nginx pods that will receive the traffic from the centos pods.
[root@master00 ~]# cat << 'EOF' > nginx-deploy-pf9.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 4 # tells deployment to run 4 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
EOF[root@master00 ~]# kubectl apply -f nginx-deploy-pf9.yamlGenerate some traffic from the centos-tool(wireguard-test-deploy) pod towards the Nginx over port 80 via a watch curl command.
These are the two pods participating in the traffic:
wireguard-test-deploy-f7985fcdf-mcz2p 1/1 Running 0 13m 10.20.115.9 192.168.50.55 <none> <none>
nginx-deployment-66b6c48dd5-fd7k2 1/1 Running 3 5d2h 10.20.246.118 192.168.50.53 <none> <none>The centOS (wireguard-test-deployment) pod is in worker 192.168.50.55
The Nginx pod is in worker 192.168.50.53
Let’s login inside the centos pod, and generate traffic against Pod’s IP 10.20.246.118
[root@master00 ~]# kubectl exec -it wireguard-test-deploy-f7985fcdf-mcz2p -- bash
[root@wireguard-test-deploy-f7985fcdf-mcz2p /]# watch curl -n 0.1 10.20.246.118Log into one of the workers that are participating in this communication and confirm via the wg command that packet transfer is increasing over the encrypted session, wg command is present on each of our nodes as a result of installing WireGuard packages, let’s do a watch with the default time so you can see how the received encrypted traffic is increasing over time.
[root@worker02 ~]# watch wg
Every 2.0s: wg Tue Apr 6 20:57:57 2021interface: wireguard.cali
public key: Vg1/L/b9jGIPixk6p0H9/bDkNK3l9sDyDvJSYx85LgU=
private key: (hidden)
listening port: 51820
fwmark: 0x40000peer: TDtpx4DNBAIg9dJ5dtt2lECeg4iIJurt3ZMYLbUZliQ=
endpoint: 192.168.50.59:51820
allowed ips: 10.20.178.12/32, 10.20.178.0/26
latest handshake: 23 seconds ago
transfer: 436.28 KiB received, 439.27 KiB sentpeer: MPkoh0erajxIBylgx48mJOidNyzxUhYTCjOg9DO7r3k=
endpoint: 192.168.50.55:51820
allowed ips: 10.20.115.0/26, 10.20.115.28/32
latest handshake: 1 minute, 44 seconds ago
transfer: 1.50 MiB received, 4.05 MiB sentWe can also validate by tcpdump for port 80 on the external interface of your worker node where traffic is flowing and you shouldn’t be able to see any port 80 traffic at all since this is encrypted.
Install tcpdump in case it is not installed in your O.S
[root@worker02 ~]# yum install tcpdump -y[root@worker02 ~]# tcpdump -i eth0 -nn port 80 -vvv
<EMPTY>Lastly, if you desire to disable WireGuard encryption, this action can be performed in a declarative way, lets disable WireGuard encryption while we are still hitting our Nginx Pod and sniffing for port 80, you should be able to see now port 80 traffic flowing.
[root@master00 ~]# calicoctl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":false}}'[root@worker01 ~]# tcpdump -i eth0 -nn port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:17:10.529020 IP 10.20.115.9.39636 > 10.20.246.118.80: Flags [S], seq 4249531524, win 28000, options [mss 1400,sackOK,TS val 8444466 ecr 0,nop,wscale 7], length 0
21:17:10.529455 IP 10.20.246.118.80 > 10.20.115.9.39636: Flags [S.], seq 763109226, ack 4249531525, win 27760, options [mss 1400,sackOK,TS val 8443383 ecr 8444466,nop,wscale 7], length 0
21:17:10.529966 IP 10.20.115.9.39636 > 10.20.246.118.80: Flags [.], ack 1, win 219, options [nop,nop,TS val 8444467 ecr 8443383], length 0Let’s enable WireGuard back in our cluster so our east-west traffic is encrypted.
[root@master00 ~]# calicoctl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":true}}'That’s all folks!
Benefits
Calico + WireGuard on Platform9 Managed Kubernetes provides an easy declarative way to encrypt data in-flight for east-west traffic in Kubernetes environments.
Conclusion
Platform9’s customers need to tackle the encryption challenge for data in flight for PCI compliance. The blog illustrates that by leveraging Platform9 Managed Kubernetes, calico, and WireGuard just how easily this can be accomplished. While this is only one of the steps for hardening a k8s cluster it is of critical importance and can be easily accomplished.
- Beyond Kubernetes Operations: Discover Platform9’s Always-On Assurance™ - November 29, 2023
- KubeCon 2023 Through Platform9’s Lens: Key Takeaways and Innovative Demos - November 14, 2023
- Getting to know Nate Conger: A candid conversation - June 12, 2023