Management DU cluster creation is failing for LTS3 at "Miscellaneous scripts and checks"

Problem

Nodelet phase restart will be stuck on "Miscellaneous scripts and checks" while management DU Cluster Creation

Environment

• Platform9 SMCP LTS3 version:- 5.9.2-3199093

Cause

When "vm.overcommit_memory" value is set to 0, the nodelet phase restart was failing at "Miscellaneous scripts and checks"

Javascript

Nodelet phase error:

[2024-04-05 14:10:18] + curl --silent --max-time 5 --cacert /etc/pf9/kube.d/certs/authn_webhook/ca.crt https://localhost:9977/healthz
failed to open addon-operator version file /etc/pf9/pf9-addon-operator-image.env . Error: open /etc/pf9/pf9-addon-operator-image.env: no such file or directory
INDEX NUMBER FILE NAME PHASE STATUS
1 Generate certs / Send signing request to CA running
2 Prepare configuration running
3 Configure Container Runtime running
4 Start Container Runtime running
5 Load user images to container runtime running
6 Configure etcd running
7 Start etcd running
8 Network configuration running
9 Configure CNI plugin running
10 Configure and start pf9-bouncer running
11 Miscellaneous scripts and checks failed

Javascript

===========
Nodelet logs Snippet:
* Applying /etc/sysctl.d/99-sysctl.conf ...
net.ipv6.conf.eth0.accept_ra = 2
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory = 0
* Applying /etc/sysctl.d/pf9-kubelet.conf ...
vm.overcommit_memory = 1
kernel.panic = 10
kernel.panic_on_oops = 1
* Applying /etc/sysctl.d/pf9-kubernetes-cri.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
* Applying /etc/sysctl.conf ...
net.ipv6.conf.eth0.accept_ra = 2
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory = 0
===========

Resolution

By default, the system loads the "/etc/sysctl.d/99-sysctl.conf" file, followed by the custom "/etc/sysctl.d/pf9-kubelet.conf" file, and finally the "/etc/sysctl.conf" file, which is a symlink of the default sysctl file "99-sysctl.conf". For making the value of the "vm.overcommit memory" parameter persistent, all these three files should be corrected.