Getting Errors on Reverting MetalLB Addon to v0.13.7 From v0.13.11 Default version in 1.26 K8s

Problem

When performing cluster upgrade from 1.25.11-pmk.115 to 1.26.10-pmk.164 in PMK 5.9.2 management plane version the MetalLB cluster addon is stuck in InstallError state. While troubleshooting we found that the speaker pods running in the ‘metallb-system’ namespace were running, but logging the below error:

Speaker-pod log

{"caller":"speakerlist.go:256","error":"1 error occurred:<br>\t* Failed to join 10.96.4.112:7946: dial tcp 10.96.4.112:7946: connect: connection refused<br>","expected":1,"joined":0,"level":"error","msg":"partial join","op":"memberDiscovery","ts":"2024-03-14T13:13:08Z"}

To solve the issue, the MetalLB addon version was downgraded from 0.13.7 from 0.13.11. After downgrading the version, the issue is resolved.

However, when trying to install the metallb addon in PMK 5.9.2. encountered the below error on 0.13.7 metallb version:

MetalLB cluster addon details in Sunpike pod

{"level":"info","msg":"Failed to read file /addon_templates/metallb/0.13.7/metallb-webhook.yaml","time":"2024-03-14T22:26:24Z"}
{"level":"error","msg":"Failed to apply yaml file: open /addon_templates/metallb/0.13.7/metallb-webhook.yaml: no such file or directory","time":"2024-03-14T22:26:24Z"}
{"level":"error","msg":"Failed to install metallb: open /addon_templates/metallb/0.13.7/metallb-webhook.yaml: no such file or directory","time":"2024-03-14T22:26:24Z"}
{"level":"error","msg":"Error installing addon: open /addon_templates/metallb/0.13.7/metallb-webhook.yaml: no such file or directory","time":"2024-03-14T22:26:24Z"}
{"level":"error","msg":"Unable to process addon: open /addon_templates/metallb/0.13.7/metallb-webhook.yaml: no such file or directory","time":"2024-03-14T22:26:24Z"}

Status:
Message: open /addon_templates/metallb/0.13.7/metallb-webhook.yaml: no such file or directory
Observed Generation: 4
Phase: InstallAddonError
Events: <none>

Environment

• Platform9 Managed Kubernetes - v5.9.2

• Self Managed Cloud Platform (SMCP) 5.9 and Higher

• Kubernetes version- 1.26.10-pmk.164.

Answer

This is a known issue, fixed in PMK release PMK-5.10,2 and higher.

The internal JIRAs used to track are:

• PMK: PMK-6161

• SMCP: AIR-1355