Hostagent Fails to Communicate With the Management Plane due to Wrong Credentials
Problem
- Hostagent failing to communicate with the Platform9 Management Plane and hostagent service is logging authentication errors:
2024-10-30 02:17:19,768 - slave.py ERROR - Connection error. Retrying in 10 seconds.Traceback (most recent call last): File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/slave.py", line 127, in reconnect_loop start(config, log, app_db, agent_app_db, app_cache, File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 770, in start dual_channel_io_loop(log, File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbcommon/amqp.py", line 245, in dual_channel_io_loop conn.ioloop.start() File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 461, in start self._poller.start() File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 721, in start self.poll() File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 1114, in poll self._dispatch_fd_events(fd_event_map) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 831, in _dispatch_fd_events handler(fileno, events) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/base_connection.py", line 410, in _handle_events self._handle_read() File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/base_connection.py", line 464, in _handle_read self._on_data_available(data) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2041, in _on_data_available self._process_frame(frame_value) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2162, in _process_frame if self._process_callbacks(frame_value): File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2140, in _process_callbacks self.callbacks.process(frame_value.channel_number, # Prefix File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 60, in wrapper return function(*tuple(args), **kwargs) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 92, in wrapper return function(*args, **kwargs) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 236, in process callback(*args, **keywords) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1838, in _on_connection_close self._on_terminate(self.closing[0], self.closing[1]) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2119, in _on_terminate self.callbacks.process(0, File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 60, in wrapper return function(*tuple(args), **kwargs) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 92, in wrapper return function(*args, **kwargs) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 236, in process callback(*args, **keywords) File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1856, in _on_connection_error raise exceptions.AMQPConnectionError(error_message orpika.exceptions.AMQPConnectionError: (403, 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.')Environment
- Platform9 Managed OpenStack - v5.0 and Higher
Cause
- Hostagent uses AMQP credentials to authenticate with the Management Plane. Misconfigured credentials can lead to authentication issues; making host disconnected from the Management Plane.
Resolution
Workaround steps:
- Retrieve
username&passwordfrom/etc/pf9/hostagent.conffile present over any healthy host from the same region. - Over affected host, modify the credentials for amqp section in
/etc/pf9/hostagent.conffile.
[amqp]host=example-region.platform9.netusername=<username>password=<password>- Restart
pf9-hostagentservice over the affected host.
# systemctl restart pf9-hostagentAdditional Information
- During credential rotation activity (if performed earlier), the OFFLINE hosts would not retrieve updated credentials to authenticate hostagent with AMQP running on Management Plane.
- The legacy bbslave user displayed in the username has been replaced by hostagent user now.
Was this page helpful?