PMK Stack Execution Fails For the Nodes That Have Hostnames With Uppercase Characters

Problem

• PMK stack is failing to come up after manual stack restart or reboot with the error mentioned below.

STACK RESTART ERROR

F1026 00:22:47.837453   37793 helpers.go:115] Error from server (NotFound): nodes "MASTER-01" not found

Environment

• Platform9 Edge Cloud - All Versions

• All Kubernetes Clusters using hostnames instead of IP address.

Cause

• The reason behind this issue is the way in which the cluster node's hostname gets parsed within different nodeletd phase scripts.

• The 020-gen_certs.sh nodeletd phase script considers node name using the environment variable $NODE_NAME which eventually ends up calling get_node_endpoint function that returns a lower case value.

Node Name after phase 020-gen_certs.sh

[2022-02-28 08:05:04] node IP is master-01
[2022-02-28 08:05:04] node name is master-01

• Whereas in the phase 110-label_and_taint_node.sh if the cluster is configured to use hostnames as the cluster node names then it assigns node's hostname as the node name as it is, without any formatting change.

Node name after phase 110-label_and_taint_node.sh

--- /opt/pf9/pf9-kube/master_scripts/110-label_and_taint_node.sh start at 2022-02-28 08:05:44 ---
[2022-02-28 08:05:44] Node name is MASTER-01

• The above mentioned difference in the Node name value results in the ambiguity and hence the stack execution fails.

Workaround:

• Current workaround for this issue is to reconfigure the node's hostname with all the letters in the lowercase and then restart the PMK stack

Resolution:

• There is an internal bug raised for this issue and Platform9 engineering team is currently working to fix it.