How to Enable Audit Logging in PMK Cluster

Problem

• Enable Audit Logging in PMK Cluster.

Info

The procedures are different depending upon the PMK version and Platform9 Kube versions. Identify the version and follow the procedure the accordingly.

Environment

• Platform9 Managed Kubernetes v5.5 & Platform9 Kube Version 1.21.3-pmk.183 and Higher

• Platform9 Edge Cloud v5.3 LTS Patch #12 v-5.3.0-1911578 & Platform9 Kube Version 1.20.15-pmk.2100 and Higher

Procedure

• For New Clusters, ensure that during cluster creation the API Server Flags parameter is configured with the required Audit logging capability flags.

• For existing Clusters, the flags can be added later by contacting Platform9 Support team.

• Platform9 Edge Cloud users who have access to their own management plane can do the same on their own as well.

Warning

If Auditing is added on existing clusters, for the changes to be applied, a full stack restart will have to be performed on all the master nodes one after the other.

• Ensure that the /var/opt/pf9/kube/apiserver-config/audit-policy.yaml file is populated with required resources that need to be audited on each master node part of the cluster. Also, to enable auditing for other resources and different audit levels, check the official documentation at kubernetes.io-audit .

Example audit-policy.yaml

$ cat /var/opt/pf9/kube/apiserver-config/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  resources:
  - group: "k8s.cni.cncf.io"
    resources: ["network-attachment-definitions"]

• Check if all the required Audit Logging Flags passed to apiServerFlags.

API Call to check Audit Logging Flags passed to apiServerFlags

curl -H "X-Auth-Token: $TOKEN" https://<DU_FQDN>/qbert/v4/<PROJECT_ID>/clusters/<CLUSTER_UUID> | jq .cloudProperties.apiServerFlags

Example API Call showing Audit Logging Flags passed to apiServerFlags

# curl -H "X-Auth-Token: $TOKEN" https://airctl-support-lts.pf9.localnet/qbert/v4/c1e62e73dfb64e12bee58e9e09ddfbc8/clusters/8355678a-8d42-4b2b-9aab-150a9d95fd34 | jq .cloudProperties.apiServerFlags
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3490  100  3490    0     0   9656      0 --:--:-- --:--:-- --:--:--  9667

"\"--audit-policy-file=/var/opt/pf9/kube/apiserver-config/audit-policy.yaml\",\"--audit-log-path=/var/opt/pf9/kube/audit/audit.log\",\"--audit-log-maxage=18\",\"--audit-log-maxsize=18\",\"--audit-log-maxbackup=18\""

• Restart the nodelet phases on all master nodes one node at a time, so that new Audit Logging Flags passed to apiServerFlags will get applied. This step is only required if the audit logging is being enabled on existing clusters.

Full PMK Stack Restart

sudo systemctl stop pf9-hostagent pf9-nodeletd
sudo /opt/pf9/nodelet/nodeletd phases stop
sudo systemctl start pf9-hostagent

Warning

Running the above commands will drain all pods/containers running on the node.

For multi-master clusters, ensure that the PMK stack restart is performed on one master node at a time else ETCD will lose quorum and the cluster will be unreachable.

• Check for audit logs.

Example audit.log

$ less /var/opt/pf9/kube/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4d4a7598-6035-4a98-9b0a-d76c0097dc1b","stage":"RequestReceived","requestURI":"/apis/k8s.cni.cncf.io/v1/namespaces/kube-system/network-attachment-definitions/whereabouts-conf","verb":"delete","user":{"username":"admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["::1"],"userAgent":"kubectl/v1.20.15 (linux/amd64) kubernetes/8f1e5bf","objectRef":{"resource":"network-attachment-definitions","namespace":"kube-system","name":"whereabouts-conf","apiGroup":"k8s.cni.cncf.io","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-14T01:57:34.801014Z","stageTimestamp":"2022-06-14T01:57:34.801014Z"}

Note

If changes are made at a later point to the audit-policy.yaml, for those to get applied, full stack restart on all the master nodes will need to be performed one node at a time.

Environment

• Platform9 Managed Kubernetes v5.5 & Platform9 Kube Version 1.21.3-pmk.111 and Below

• Platform9 Edge Cloud v5.3 LTS Patch #11 v-5.3.0-1806225 & Platform9 Kube Version 1.20.11-pmk.2038 and Below

Note: Below instructions WONT persist post cluster upgrade.

Procedure

• Create a file/etc/kubernetes/audit-policy.yaml with required content the all the master nodes. Also, to enable auditing for other resources and different audit levels check the document at kubernetes.io-audit

Example /etc/kubernetes/audit-policy.yaml

# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  resources:
  - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
    resources: ["pods"]

• Edit /opt/pf9/pf9-kube/conf/masterconfig/base/<centos/ubuntu>/master.yamlfile and add below content in the kube-apiserver container section on all master nodes.

Example master.yaml file showing Section with Changes

- name: "kube-apiserver"
      image: "gcr.io/google_containers/hyperkube:__KUBERNETES_VERSION__"
      command:
...
        - "--audit-policy-file=/etc/kubernetes/audit-policy.yaml"
        - "--audit-log-path=/var/log/kubernetes/audit/audit.log"

      volumeMounts:
        - mountPath: "/etc/kubernetes/audit-policy.yaml"
          name: "audit"
          readOnly: true
        - mountPath: "/var/log/kubernetes/audit/"
          name: "audit-log"
          readOnly: false

• Edit /opt/pf9/pf9-kube/conf/masterconfig/base/<centos/ubuntu>/master.yaml file again and add below content in the volume section on all master nodes.

Example master.yaml file showing Section with Changes

volumes:
    - hostPath:
       path: "/etc/kubernetes/audit-policy.yaml"
      name: "audit"
    - hostPath:
       path: "/var/log/kubernetes/audit"
      name: "audit-log"

• Restart the PMK stack on all master nodes one node at a time.

Full PMK Stack Restart

sudo systemctl stop pf9-hostagent pf9-nodeletd
sudo /opt/pf9/nodelet/nodeletd phases stop
sudo systemctl start pf9-hostagent

Warning

Running the above commands will drain all pods/containers running on the node.

For multi-master clusters, ensure that the PMK stack restart is performed on one master node at a time else ETCD will lose quorum and the cluster will be unreachable.

• Once the PMK stack is back up, the logs should be created in /var/log/kubernetes/audit directory.

Note

If changes are made at a later point to the audit-policy.yaml, for those to get applied, full stack restart on all the master nodes will need to be performed one node at a time.

Additional Information

https://kubernetes.io/docs/tasks/debug-application-cluster/audit/