How To Run Kube-Bench In Platform9 Managed Kubernetes Clusters
Problem
How To Run Kube-Bench In Platform9 Managed Kubernetes Clusters?
Environment
- Platform9 Managed Kubernetes - 5.3
- K8s version 1.20
Procedure
- Checkout the kube-bench repo~# git clone --single-branch --branch private/platform9-1.20.5 https://github.com/platform9/kube-bench.git
- Switch to the kube-bench directory~# cd kube-bench
- Create a kube-bench image and ensure it's available for cluster nodes. 3A. By pushing image to repository: a. Enter your docker hub username and password when prompted ~# docker login b. Build the image locally: ~# docker build -t kube-bench:latest .
Note: You will have to replace <UserName> with the docker hub username if you want to push it later else you can use the following command: ~# docker build -t <UserName>/kube-bench:latest .
c. Once the above command is successful, push the image to your dockerhub repo ~# docker push <UserName>/kube-bench:latest
- To change the username - https://docs.docker.com/engine/reference/commandline/push/ - To push image to a private registry replace <UserName> with localhost:5100
Note: Skip this step(step #3A) if you have no intention of pushing the image to your dockerhub repo. You may proceed with the below. OR 3B. Without pushing image to repository: a. Check the accurate image name with tag - https://docs.docker.com/engine/reference/commandline/save/ ~# docker images
b. Create a gzip tarball file of your image: ~# docker save <UserName>/kube-bench:latest | gzip > kube-bench.tar.gz
c. Copy the image tarball to destination node using SCP. ~# scp kube-bench.tar.gz <user@destination:/path/to/user/dir>
d. Load this image on a different node, use this: ~# docker load -i kube-bench.tar.gz
e. Run docker images on the destination node and you should see the kube-bench image
~# docker images | grep "kube-bench"
- Ensure the following files are present in kubebench/cfg/pmk-1.0/ kube-bench]# ls cfg/pmk-1.0/ config.yaml controlplane.yaml etcd.yaml master.yaml node.yaml policies.yaml
- Running the benchmark test:
- Make sure the image is present on Master and Worker nodes before testing.
- Test individually on each Master and worker nodes.
a. For Master Node: Master]$ sudo docker run -t --network=host --rm --pid=host -v /etc:/etc:ro -v /var:/var:ro -v /run:/run:ro -v $HOME/kube-bench/cfg:/opt/kube-bench/cfg -v /usr/local/bin/kubectl:/usr/local/mount-from-host/bin/kubectl -v /etc/pf9/kube.d/kubeconfigs/admin.yaml:/.kube/admin.yaml -e KUBECONFIG=/.kube/admin.yaml <UserName>/kube-bench:latest run --benchmark=pmk-1.0 --targets=master,etcd,controlplane,policies
b. For worker nodes: Worker]$ sudo docker run -t --network=host --rm --pid=host -v /etc:/etc:ro -v /var:/var:ro -v /run:/run:ro -v $HOME/kube-bench/cfg:/opt/kube-bench/cfg -v /usr/local/bin/kubectl:/usr/local/mount-from-host/bin/kubectl -v /etc/pf9/kube.d/kubeconfigs/admin.yaml:/.kube/admin.yaml -e KUBECONFIG=/.kube/admin.yaml <UserName>/kube-bench:latest run --benchmark=pmk-1.0 --targets=node
##