Resetting RabbitMQ Database Caused Complete Host Disconnection from Management Plane

Problem

• All hosts were disconnected from the management plane due to the absence of the bbslave user, which is required to authenticate with the RabbitMQ database.

Hostagent.logs

File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1856, in _on_connection_error
    raise exceptions.AMQPConnectionError(error_message or
pika.exceptions.AMQPConnectionError: (403, 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.')

Rabbitmq.logs

2025-04-25 05:28:50.371 [error] <0.14706.446> Error on AMQP connection <0.14706.446> (127.0.0.1:37640 -> 127.0.0.1:5672, state: starting):
PLAIN login refused: user 'bbslave' - invalid credentials

• No users were present on rabbitmq db

User listing

[root@testbed rabbitmq]# sudo rabbitmqctl list_users
Listing users ...
user    tags
guest   [administrator]

Environment

• Platform9 Managed Edge Cloud - v5.3

Cause

• The following commands performed on rabbitmq db wiped off all the users, which makes all the hosts not able to authenticate with management plane.

Destructive commands

sudo rabbitmqctl stop_app
sudo rabbitmqctl reset
sudo rabbitmqctl start_app
sudo rabbitmqctl restart

Resolution

• The way out from the situation is to perform same version upgrade of the management plane.

• The upgrade process restores default RabbitMQ users and permissions, allowing full recovery.

Additional Information

Do not make any modifications to the databases without consulting Platform9 Support or referring to the official Platform9 documentation.