Nodelet Fails at "Configure and start pf9-bouncer" Phase

Problem

Platform9 nodelet phase fails at "Configure and start pf9-bouncer" phase with "bind address already in use" on bouncer logs.

Bash

{"log":"2024/04/10 05:35:40 serve http:listen tcp 127.0.0.1:9977: bind: address already in use<br>","stream":"stderr","time":"2024-04-10T05:35:40.02933616Z"}

Nodelet logs keep showing "Waiting for "authn_webhook_listening" to evaluate to true" message.

Bash

{"L":"INFO","T":"2024-04-10T05:35:39.781Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] Stopping containers 'bouncer'"}
{"L":"INFO","T":"2024-04-10T05:35:39.807Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] No such container: bouncer"}
{"L":"INFO","T":"2024-04-10T05:35:39.810Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] Destroying containers 'bouncer'"}
{"L":"INFO","T":"2024-04-10T05:35:39.836Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] bouncer"}
{"L":"INFO","T":"2024-04-10T05:35:40.030Z","C":"command/command.go:120","M":"[2024-04-10 05:35:40] 90907c098a9b96c6c3ce40605f7d2f869fb4e67d55f86f9b6ddf279a56aba7a4"}
{"L":"INFO","T":"2024-04-10T05:35:40.053Z","C":"command/command.go:120","M":"[2024-04-10 05:35:40] Waiting for \"authn_webhook_listening\" to evaluate to true ..."}
{"L":"INFO","T":"2024-04-10T05:35:45.073Z","C":"command/command.go:120","M":"[2024-04-10 05:35:45] Waiting for \"authn_webhook_listening\" to evaluate to true ..."}
{"L":"INFO","T":"2024-04-10T05:35:50.094Z","C":"command/command.go:120","M":"[2024-04-10 05:35:50] Waiting for \"authn_webhook_listening\" to evaluate to true ..."}

Environment

• SMCP version 5.9.1.

Cause

On the underlying node stale bouncer process keeps holding port 9977.

Resolution

Verify the stale process by running the below command:

Bash

# ps -aux | grep -i "9977"
# netstat -tunlp | grep -i "9977"

Once we get the process ID follow the procedure given below:

1. Stop the pf9-nodeletd and pf9-hostagent service - systemctl stop pf9-hostagent pf9-nodeletd.

2. Force stop the nodelet phases using $ /opt/pf9/nodelet/nodeletd phases stop --force .

3. Kill the stale bouncerd process using $ kill -9 <bouncerd-process-id> .

4. Start nodelet phase using - systemctl start pf9-hostagent.

5. Verify the nodelet phase status using $ /opt/pf9/nodelet/nodeletd phases status .

Additional Information

The above procedure helps to recover the nodelet phase issue due to the bouncer bind address already in use error.