PMK Stack Execution Fails For the Nodes That Have Hostnames With Uppercase Characters

Problem

• PMK stack is failing to come up after manual stack restart or reboot with the error mentioned below.

F1026 00:22:47.837453   37793 helpers.go:115] Error from server (NotFound): nodes "MASTER-01" not found

Environment

• Platform9 Edge Cloud - All Versions
• All Kubernetes Clusters using hostnames instead of IP address.

Cause

• The reason behind this issue is the way in which the cluster node's hostname gets parsed within different nodeletd phase scripts.
• The 020-gen_certs.sh nodeletd phase script considers node name using the environment variable $NODE_NAME which eventually ends up calling get_node_endpoint function that returns a lower case value.

[2022-02-28 08:05:04] node IP is master-01

[2022-02-28 08:05:04] node name is master-01

• Whereas in the phase 110-label_and_taint_node.sh if the cluster is configured to use hostnames as the cluster node names then it assigns node's hostname as the node name as it is, without any formatting change.

--- /opt/pf9/pf9-kube/master_scripts/110-label_and_taint_node.sh start at 2022-02-28 08:05:44 ---

[2022-02-28 08:05:44] Node name is MASTER-01

• The above mentioned difference in the Node name value results in the ambiguity and hence the stack execution fails.

Workaround:

• Current workaround for this issue is to reconfigure the node's hostname with all the letters in the lowercase and then restart the PMK stack

Resolution:

• There is an internal bug raised for this issue and Platform9 engineering team is currently working to fix it.