Nodelet Fails at "Configure and start pf9-bouncer" Phase
Problem
Platform9 nodelet phase fails at "Configure and start pf9-bouncer" phase with "bind address already in use" on bouncer logs.
{"log":"2024/04/10 05:35:40 serve http:listen tcp 127.0.0.1:9977: bind: address already in use\n","stream":"stderr","time":"2024-04-10T05:35:40.02933616Z"}Nodelet logs keep showing "Waiting for "authn_webhook_listening" to evaluate to true" message.
{"L":"INFO","T":"2024-04-10T05:35:39.781Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] Stopping containers 'bouncer'"}{"L":"INFO","T":"2024-04-10T05:35:39.807Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] No such container: bouncer"}{"L":"INFO","T":"2024-04-10T05:35:39.810Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] Destroying containers 'bouncer'"}{"L":"INFO","T":"2024-04-10T05:35:39.836Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] bouncer"}{"L":"INFO","T":"2024-04-10T05:35:40.030Z","C":"command/command.go:120","M":"[2024-04-10 05:35:40] 90907c098a9b96c6c3ce40605f7d2f869fb4e67d55f86f9b6ddf279a56aba7a4"}{"L":"INFO","T":"2024-04-10T05:35:40.053Z","C":"command/command.go:120","M":"[2024-04-10 05:35:40] Waiting for \"authn_webhook_listening\" to evaluate to true ..."}{"L":"INFO","T":"2024-04-10T05:35:45.073Z","C":"command/command.go:120","M":"[2024-04-10 05:35:45] Waiting for \"authn_webhook_listening\" to evaluate to true ..."}{"L":"INFO","T":"2024-04-10T05:35:50.094Z","C":"command/command.go:120","M":"[2024-04-10 05:35:50] Waiting for \"authn_webhook_listening\" to evaluate to true ..."}Environment
- SMCP version 5.9.1.
Cause
On the underlying node stale bouncer process keeps holding port 9977.
Resolution
Verify the stale process by running the below command:
# ps -aux | grep -i "9977"# netstat -tunlp | grep -i "9977"Once we get the process ID follow the procedure given below:
- Stop the pf9-nodeletd and pf9-hostagent service -
systemctl stop pf9-hostagent pf9-nodeletd. - Force stop the nodelet phases using
$ /opt/pf9/nodelet/nodeletd phases stop --force. - Kill the stale bouncerd process using
$ kill -9 <bouncerd-process-id>. - Start nodelet phase using -
systemctl start pf9-hostagent. - Verify the nodelet phase status using
$ /opt/pf9/nodelet/nodeletd phases status.
Additional Information
The above procedure helps to recover the nodelet phase issue due to the bouncer bind address already in use error.
Was this page helpful?