Hostagent Fails to Communicate With the Management Plane due to Wrong Credentials

Problem

• Hostagent failing to communicate with the Platform9 Management Plane and hostagent service is logging authentication errors:

Hostagent Logs

2024-10-30 02:17:19,768 - slave.py ERROR - Connection error. Retrying in 10 seconds.
Traceback (most recent call last):
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/slave.py", line 127, in reconnect_loop
    start(config, log, app_db, agent_app_db, app_cache,
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 770, in start
    dual_channel_io_loop(log,
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbcommon/amqp.py", line 245, in dual_channel_io_loop
    conn.ioloop.start()
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 461, in start
    self._poller.start()
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 721, in start
    self.poll()
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 1114, in poll
    self._dispatch_fd_events(fd_event_map)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 831, in _dispatch_fd_events
    handler(fileno, events)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/base_connection.py", line 410, in _handle_events
    self._handle_read()
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/base_connection.py", line 464, in _handle_read
    self._on_data_available(data)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2041, in _on_data_available
    self._process_frame(frame_value)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2162, in _process_frame
    if self._process_callbacks(frame_value):
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2140, in _process_callbacks
    self.callbacks.process(frame_value.channel_number,  # Prefix
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 60, in wrapper
    return function(*tuple(args), **kwargs)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 92, in wrapper
    return function(*args, **kwargs)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 236, in process
    callback(*args, **keywords)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1838, in _on_connection_close
    self._on_terminate(self.closing[0], self.closing[1])
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2119, in _on_terminate
    self.callbacks.process(0,
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 60, in wrapper
    return function(*tuple(args), **kwargs)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 92, in wrapper
    return function(*args, **kwargs)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 236, in process
    callback(*args, **keywords)
  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1856, in _on_connection_error
    raise exceptions.AMQPConnectionError(error_message or
pika.exceptions.AMQPConnectionError: (403, 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.')

Environment

• Platform9 Managed OpenStack - v5.0 and Higher

Cause

• Hostagent uses AMQP credentials to authenticate with the Management Plane. Misconfigured credentials can lead to authentication issues; making host disconnected from the Management Plane.

Resolution

Workaround steps:

1. Retrieve username & password from /etc/pf9/hostagent.conf file present over any healthy host from the same region.

2. Over affected host, modify the credentials for amqp section in /etc/pf9/hostagent.conf file.

/etc/pf9/hostagent.conf

[amqp]
host=example-region.platform9.net
username=<username>
password=<password>

1. Restart pf9-hostagent service over the affected host.

Bash

# systemctl restart pf9-hostagent

Additional Information

• During credential rotation activity (if performed earlier), the OFFLINE hosts would not retrieve updated credentials to authenticate hostagent with AMQP running on Management Plane.

• The legacy bbslave user displayed in the username has been replaced by hostagent user now.