How to Restart PMK Stack?

Problem

Restart PMK stack during the discrepancies occurred related to the PMK stack.

Info

PMK (Platform9 Managed Kubernetes) Stack basically refers to the nodelet phases running on the node.

Environment

• Platform9 Managed Kubernetes - v5.3 and Higher.

Procedure

Steps to be followed to restart PMK stack in the required node:

1. Stop pf9-hostagent and pf9-nodeletd services:

In the affected node

sudo systemctl stop pf9-hostagent pf9-nodeletd

1. Stop nodeletd phase:

In the affected node

# sudo /opt/pf9/nodelet/nodeletd phases stop --force
# sudo /opt/pf9/nodelet/nodeletd phases status
  INDEX NUMBER  FILE                                         NAME     PHASE STATUS
  1             Generate certs / Send signing request to CA  stopped
  2             Prepare configuration                        stopped
  3             Configure Container Runtime                  stopped
  4             Start Container Runtime                      stopped
  5             Configure etcd                               stopped
  6             Start etcd                                   stopped
  7             Network configuration                        stopped
  8             Configure CNI plugin                         stopped
  9             Configure and start pf9-bouncer              stopped
  10            Miscellaneous scripts and checks             stopped
  11            Configure and start kubelet                  stopped
  12            Configure and start kube-proxy               stopped
  13            Wait for k8s services and network to be up   stopped
  14            Apply and validate node taints               stopped
  15            Apply dynamic kubelet configuration          stopped
  16            Uncordon node                                stopped
  17            Deploy app catalog                           stopped
  18            Configure and start Keepalived               stopped
  19            Configure and start pf9-addon-operator       stopped
  20            Drain all pods (stop only operation)         stopped
  21            Configure and start monitoring               stopped

1. Start nodeletd phase:

In the affected node

# /opt/pf9/nodelet/nodeletd phases start --verbose
INDEX NUMBER  FILE                                         NAME     PHASE STATUS
  1             Generate certs / Send signing request to CA  running
  2             Prepare configuration                        running
  3             Configure Container Runtime                  running
  4             Start Container Runtime                      running
  5             Configure etcd                               running
  6             Start etcd                                   running
  7             Network configuration                        running
  8             Configure CNI plugin                         running
  9             Configure and start pf9-bouncer              running
  10            Miscellaneous scripts and checks             running
  11            Configure and start kubelet                  running
  12            Configure and start kube-proxy               running
  13            Wait for k8s services and network to be up   running
  14            Apply and validate node taints               running
  15            Apply dynamic kubelet configuration          running
  16            Uncordon node                                running
  17            Deploy app catalog                           running
  18            Configure and start Keepalived               running
  19            Configure and start pf9-addon-operator       running
  20            Drain all pods (stop only operation)         running
  21            Configure and start monitoring               running

1. Start pf9-hostagent and ps9-nodeleted service.

In the affected node

# sudo systemctl start pf9-hostagent pf9-nodeletd

Additional Information

• Starting the pf9-hostagent service will start the PMK stack which is already stopped.

• The nodeletd phases logs are also available at /var/log/pf9/nodeletd.log file on the node.