How To Rotate Container Logs on The Master and Worker Nodes
Problem
Container logs are utilizing all the disk space on the master and worker nodes.
Environment
- Platform9 Managed Kubernetes - All Versions
- Docker
Cause
The docker service is not configured to rotate the container logs.
Resolution
- Back up the /etc/docker/daemon.json file on the affected node if present. Add the log rotation config on the node in the file /etc/docker/daemon.json file as shown below.
Change the values for max-size and max-file parameter as per your requirement.
$ less /etc/docker/daemon.json{ "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "5" }}Perform the below steps on one node at a time to avoid downtime.
- Stop the Platform9 Managed Kubernetes stack by executing commands in the below sequence.
$ sudo systemctl stop pf9-hostagent pf9-nodeletd$ sudo /opt/pf9/nodelet/nodeletd phases stop- Restart the docker service.
$ systemctl restart docker- Start the Platform9 Managed Kubernetes stack by executing the commands in the below sequence.
$ sudo /opt/pf9/nodelet/nodeletd phases start$ sudo systemctl start pf9-hostagent pf9-nodeletd- Verify the changes took effect on the container by running below command.
$ sudo docker ps | grep -i <conmtainer-name>$ sudo docker inspect <container-id>| grep -A6 -i LogConfig "LogConfig": { "Type": "json-file", "Config": { "max-file": "5", "max-size": "50m" } },Was this page helpful?