How to Enable Audit Logging in PMK Cluster

Problem

• Enable Audit Logging in PMK Cluster.

Environment

• Platform9 Managed Kubernetes v5.5 & Platform9 Kube Version 1.21.3-pmk.183 and Higher
• Platform9 Edge Cloud v5.3 LTS Patch #12 v-5.3.0-1911578 & Platform9 Kube Version 1.20.15-pmk.2100 and Higher

Procedure

• For New Clusters, ensure that during cluster creation the API Server Flags parameter is configured with the required Audit logging capability flags.
• For existing Clusters, the flags can be added later by contacting Platform9 Support team.
• Platform9 Edge Cloud users who have access to their own management plane can do the same on their own as well.

• Ensure that the /var/opt/pf9/kube/apiserver-config/audit-policy.yaml file is populated with required resources that need to be audited on each master node part of the cluster. Also, to enable auditing for other resources and different audit levels, check the official documentation at kubernetes.io-audit .

$ cat /var/opt/pf9/kube/apiserver-config/audit-policy.yaml

apiVersion: audit.k8s.io/v1

kind: Policy

rules:

- level: Metadata

  resources:

  - group: "k8s.cni.cncf.io"

    resources: ["network-attachment-definitions"]

• Check if all the required Audit Logging Flags passed to apiServerFlags.

curl -H "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/clusters/<CLUSTER_UUID> | jq .cloudProperties.apiServerFlags

# curl -H "X-Auth-Token: $TOKEN" https://airctl-support-lts.pf9.localnet/qbert/v4/c1e62e73dfb64e12bee58e9e09ddfbc8/clusters/8355678a-8d42-4b2b-9aab-150a9d95fd34 | jq .cloudProperties.apiServerFlags

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  3490  100  3490    0     0   9656      0 --:--:-- --:--:-- --:--:--  9667

​

"\"--audit-policy-file=/var/opt/pf9/kube/apiserver-config/audit-policy.yaml\",\"--audit-log-path=/var/opt/pf9/kube/audit/audit.log\",\"--audit-log-maxage=18\",\"--audit-log-maxsize=18\",\"--audit-log-maxbackup=18\""

• Restart the nodelet phases on all master nodes one node at a time, so that new Audit Logging Flags passed to apiServerFlags will get applied. This step is only required if the audit logging is being enabled on existing clusters.

sudo systemctl stop pf9-hostagent pf9-nodeletd

sudo /opt/pf9/nodelet/nodeletd phases stop

sudo systemctl start pf9-hostagent

• Check for audit logs.

$ less /var/opt/pf9/kube/audit/audit.log

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4d4a7598-6035-4a98-9b0a-d76c0097dc1b","stage":"RequestReceived","requestURI":"/apis/k8s.cni.cncf.io/v1/namespaces/kube-system/network-attachment-definitions/whereabouts-conf","verb":"delete","user":{"username":"admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["::1"],"userAgent":"kubectl/v1.20.15 (linux/amd64) kubernetes/8f1e5bf","objectRef":{"resource":"network-attachment-definitions","namespace":"kube-system","name":"whereabouts-conf","apiGroup":"k8s.cni.cncf.io","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-14T01:57:34.801014Z","stageTimestamp":"2022-06-14T01:57:34.801014Z"}

Environment

• Platform9 Managed Kubernetes v5.5 & Platform9 Kube Version 1.21.3-pmk.111 and Below
• Platform9 Edge Cloud v5.3 LTS Patch #11 v-5.3.0-1806225 & Platform9 Kube Version 1.20.11-pmk.2038 and Below

Note: Below instructions WONT persist post cluster upgrade.

Procedure

• Create a file/etc/kubernetes/audit-policy.yaml with required content the all the master nodes. Also, to enable auditing for other resources and different audit levels check the document at kubernetes.io-audit

# Log all requests at the Metadata level.

apiVersion: audit.k8s.io/v1

kind: Policy

rules:

- level: Metadata

  resources:

  - group: ""

      # Resource "pods" doesn't match requests to any subresource of pods,

      # which is consistent with the RBAC policy.

    resources: ["pods"]

• Edit /opt/pf9/pf9-kube/conf/masterconfig/base/<centos/ubuntu>/master.yamlfile and add below content in the kube-apiserver container section on all master nodes.

- name: "kube-apiserver"

      image: "gcr.io/google_containers/hyperkube:__KUBERNETES_VERSION__"

      command:

...

        - "--audit-policy-file=/etc/kubernetes/audit-policy.yaml"

        - "--audit-log-path=/var/log/kubernetes/audit/audit.log"

​

      volumeMounts:

        - mountPath: "/etc/kubernetes/audit-policy.yaml"

          name: "audit"

          readOnly: true

        - mountPath: "/var/log/kubernetes/audit/"

          name: "audit-log"

          readOnly: false

• Edit /opt/pf9/pf9-kube/conf/masterconfig/base/<centos/ubuntu>/master.yaml file again and add below content in the volume section on all master nodes.

volumes:

    - hostPath:

       path: "/etc/kubernetes/audit-policy.yaml"

      name: "audit"

    - hostPath:

       path: "/var/log/kubernetes/audit"

      name: "audit-log"

• Restart the PMK stack on all master nodes one node at a time.

sudo systemctl stop pf9-hostagent pf9-nodeletd

sudo /opt/pf9/nodelet/nodeletd phases stop

sudo systemctl start pf9-hostagent

• Once the PMK stack is back up, the logs should be created in /var/log/kubernetes/audit directory.

Additional Information

https://kubernetes.io/docs/tasks/debug-application-cluster/audit/