How To Rotate Container Logs on The Master and Worker Nodes

Problem

Container logs are utilizing all the disk space on the master and worker nodes.

Environment

• Platform9 Managed Kubernetes - All Versions

• Docker

Cause

The docker service is not configured to rotate the container logs.

Resolution

1. Back up the /etc/docker/daemon.json file on the affected node if present. Add the log rotation config on the node in the file /etc/docker/daemon.json file as shown below.

Info

Change the values for max-size and max-file parameter as per your requirement.

Sample Config

$ less /etc/docker/daemon.json
{
   "log-driver": "json-file",
   "log-opts": {
     "max-size": "50m",   
     "max-file": "5"
   }
}

Warning

Perform the below steps on one node at a time to avoid downtime.

1. Stop the Platform9 Managed Kubernetes stack by executing commands in the below sequence.

Command 1

$ sudo systemctl stop pf9-hostagent pf9-nodeletd

Command 2

$ sudo /opt/pf9/nodelet/nodeletd phases stop

1. Restart the docker service.

Command

$ systemctl restart docker

1. Start the Platform9 Managed Kubernetes stack by executing the commands in the below sequence.

Command 1

$ sudo /opt/pf9/nodelet/nodeletd phases start

Command 2

$ sudo systemctl start pf9-hostagent pf9-nodeletd

1. Verify the changes took effect on the container by running below command.

Command

$ sudo docker ps | grep -i <conmtainer-name>

Command

$ sudo docker inspect <container-id>|  grep -A6 -i LogConfig
           "LogConfig": {
               "Type": "json-file",
               "Config": {
                   "max-file": "5",
                   "max-size": "50m"
               }
           },