Resetting RabbitMQ Database Caused Complete Host Disconnection from Management Plane
Problem
All hosts were disconnected from the management plane due to the absence of the bbslave user, which is required to authenticate with the RabbitMQ database.
File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1856, in _on_connection_error raise exceptions.AMQPConnectionError(error_message orpika.exceptions.AMQPConnectionError: (403, 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.') 2025-04-25 05:28:50.371 [error] <0.14706.446> Error on AMQP connection <0.14706.446> (127.0.0.1:37640 -> 127.0.0.1:5672, state: starting):PLAIN login refused: user 'bbslave' - invalid credentials- No users were present on rabbitmq db
[root@testbed rabbitmq]# sudo rabbitmqctl list_usersListing users user tagsguest [administrator]Environment
- Platform9 Managed Edge Cloud - v5.3
Cause
- The following commands performed on rabbitmq db wiped off all the users, which makes all the hosts not able to authenticate with management plane.
sudo rabbitmqctl stop_appsudo rabbitmqctl resetsudo rabbitmqctl start_appsudo rabbitmqctl restartResolution
- The way out from the situation is to perform same version upgrade of the management plane.
- The upgrade process restores default RabbitMQ users and permissions, allowing full recovery.
Additional Information
Do not make any modifications to the databases without consulting Platform9 Support or referring to the official Platform9 documentation.
Was this page helpful?