> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pmk/how-to/how-to-enable-audit-logging-in-pmk-cluster.md). # How to Enable Audit Logging in PMK Cluster ## Problem * Enable Audit Logging in PMK Cluster. {% hint style="info" %} **Info** The procedures are different depending upon the PMK version and Platform9 Kube versions. Identify the version and follow the procedure the accordingly. {% endhint %} ## Environment * Platform9 Managed Kubernetes v5.5 & Platform9 Kube Version 1.21.3-pmk.183 and Higher * Platform9 Edge Cloud v5.3 LTS Patch #12 v-5.3.0-1911578 & Platform9 Kube Version 1.20.15-pmk.2100 and Higher ## Procedure * For New Clusters, ensure that during cluster creation the `API Server Flags` parameter is configured with the required Audit logging capability flags. * For existing Clusters, the flags can be added later by contacting [Platform9 Support](https://support.platform9.com/) team. * Platform9 Edge Cloud users who have access to their own management plane can do the same on their own as well. {% hint style="warning" %} **Warning** If Auditing is added on existing clusters, for the changes to be applied, a full stack restart will have to be performed on all the master nodes one after the other. {% endhint %} * Ensure that the `/var/opt/pf9/kube/apiserver-config/audit-policy.yaml` file is populated with required resources that need to be audited on each master node part of the cluster. Also, to enable auditing for other resources and different audit levels, check the official documentation at [kubernetes.io-audit](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) . {% tabs %} {% tab title="Example audit-policy.yaml" %} ```bash $ cat /var/opt/pf9/kube/apiserver-config/audit-policy.yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata resources: - group: "k8s.cni.cncf.io" resources: ["network-attachment-definitions"] ``` {% endtab %} {% endtabs %} * Check if all the required Audit Logging Flags passed to apiServerFlags. {% tabs %} {% tab title="API Call to check Audit Logging Flags passed to apiServerFlags" %} ```bash curl -H "X-Auth-Token: $TOKEN" https:///qbert/v4//clusters/ | jq .cloudProperties.apiServerFlags ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Example API Call showing Audit Logging Flags passed to apiServerFlags" %} ```bash # curl -H "X-Auth-Token: $TOKEN" https://airctl-support-lts.pf9.localnet/qbert/v4/c1e62e73dfb64e12bee58e9e09ddfbc8/clusters/8355678a-8d42-4b2b-9aab-150a9d95fd34 | jq .cloudProperties.apiServerFlags % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3490 100 3490 0 0 9656 0 --:--:-- --:--:-- --:--:-- 9667 "\"--audit-policy-file=/var/opt/pf9/kube/apiserver-config/audit-policy.yaml\",\"--audit-log-path=/var/opt/pf9/kube/audit/audit.log\",\"--audit-log-maxage=18\",\"--audit-log-maxsize=18\",\"--audit-log-maxbackup=18\"" ``` {% endtab %} {% endtabs %} * Restart the nodelet phases on all master nodes one node at a time, so that new Audit Logging Flags passed to apiServerFlags will get applied. This step is only required if the audit logging is being enabled on existing clusters. {% tabs %} {% tab title="Full PMK Stack Restart" %} ```bash sudo systemctl stop pf9-hostagent pf9-nodeletd sudo /opt/pf9/nodelet/nodeletd phases stop sudo systemctl start pf9-hostagent ``` {% endtab %} {% endtabs %} {% hint style="warning" %} **Warning** Running the above commands will drain all pods/containers running on the node. For multi-master clusters, ensure that the PMK stack restart is performed on **one master node at a time** else ETCD will lose quorum and the cluster will be unreachable. {% endhint %} * Check for audit logs. {% tabs %} {% tab title="Example audit.log" %} ```bash $ less /var/opt/pf9/kube/audit/audit.log {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4d4a7598-6035-4a98-9b0a-d76c0097dc1b","stage":"RequestReceived","requestURI":"/apis/k8s.cni.cncf.io/v1/namespaces/kube-system/network-attachment-definitions/whereabouts-conf","verb":"delete","user":{"username":"admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["::1"],"userAgent":"kubectl/v1.20.15 (linux/amd64) kubernetes/8f1e5bf","objectRef":{"resource":"network-attachment-definitions","namespace":"kube-system","name":"whereabouts-conf","apiGroup":"k8s.cni.cncf.io","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-14T01:57:34.801014Z","stageTimestamp":"2022-06-14T01:57:34.801014Z"} ``` {% endtab %} {% endtabs %} {% hint style="info" %} **Note** If changes are made at a later point to the *audit-policy.yaml*, for those to get applied, full stack restart on all the master nodes will need to be performed one node at a time. {% endhint %} ## Environment * Platform9 Managed Kubernetes v5.5 & Platform9 Kube Version 1.21.3-pmk.111 and Below * Platform9 Edge Cloud v5.3 LTS Patch #11 v-5.3.0-1806225 & Platform9 Kube Version 1.20.11-pmk.2038 and Below **Note:** Below instructions **WONT** persist post cluster upgrade. ## Procedure * Create a file`/etc/kubernetes/audit-policy.yaml` with required content the all the master nodes. Also, to enable auditing for other resources and different audit levels check the document at [kubernetes.io-audit](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) {% tabs %} {% tab title="Example /etc/kubernetes/audit-policy.yaml" %} ```none # Log all requests at the Metadata level. apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata resources: - group: "" # Resource "pods" doesn't match requests to any subresource of pods, # which is consistent with the RBAC policy. resources: ["pods"] ``` {% endtab %} {% endtabs %} * Edit `/opt/pf9/pf9-kube/conf/masterconfig/base//master.yaml`file and add below content in the *`kube-apiserver`* container section on all master nodes. {% tabs %} {% tab title="Example master.yaml file showing Section with Changes" %} ```none - name: "kube-apiserver" image: "gcr.io/google_containers/hyperkube:__KUBERNETES_VERSION__" command: ... - "--audit-policy-file=/etc/kubernetes/audit-policy.yaml" - "--audit-log-path=/var/log/kubernetes/audit/audit.log" volumeMounts: - mountPath: "/etc/kubernetes/audit-policy.yaml" name: "audit" readOnly: true - mountPath: "/var/log/kubernetes/audit/" name: "audit-log" readOnly: false ``` {% endtab %} {% endtabs %} * Edit `/opt/pf9/pf9-kube/conf/masterconfig/base//master.yaml` file again and add below content in the *`volume`* section on all master nodes. {% tabs %} {% tab title="Example master.yaml file showing Section with Changes" %} ```none volumes: - hostPath: path: "/etc/kubernetes/audit-policy.yaml" name: "audit" - hostPath: path: "/var/log/kubernetes/audit" name: "audit-log" ``` {% endtab %} {% endtabs %} * Restart the PMK stack on all master nodes one node at a time. {% tabs %} {% tab title="Full PMK Stack Restart" %} ```none sudo systemctl stop pf9-hostagent pf9-nodeletd sudo /opt/pf9/nodelet/nodeletd phases stop sudo systemctl start pf9-hostagent ``` {% endtab %} {% endtabs %} {% hint style="warning" %} **Warning** Running the above commands will drain all pods/containers running on the node. For multi-master clusters, ensure that the PMK stack restart is performed on **one master node at a time** else ETCD will lose quorum and the cluster will be unreachable. {% endhint %} * Once the PMK stack is back up, the logs should be created in `/var/log/kubernetes/audit` directory. {% hint style="info" %} **Note** If changes are made at a later point to the *audit-policy.yaml*, for those to get applied, full stack restart on all the master nodes will need to be performed one node at a time. {% endhint %} ## Additional Information --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://platform9.com/kb/pmk/how-to/how-to-enable-audit-logging-in-pmk-cluster.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.