Host Fails to Converge With Error: "rpmdb open failed"

Problem

• A host is failing to converge with the following error printed in /var/log/pf9/hostagent.log.

None

246 - session.py INFO - --- Converging ---
...
188 - pf9_app.py INFO - Setting the desired service state
703 - pf9_app_db.py ERROR - Erase command failed : sudo /opt/pf9/hostagent/bin/pf9-yum erase pf9-*. Return code: 1, stdout: , stderr: error: db5 error(11) from dbenv->open: Resource tempo
rarily unavailable
error: cannot open Packages index using db5 - Resource temporarily unavailable (11)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:


Error: rpmdb open failed

Environment

• Platform9 Managed OpenStack - All Versions

• Platform9 Managed Kubernetes - All Versions

• CentOS

Cause

• One or more partitions on the root disk are full.

• The Yum database is corrupted on disk.

Resolution

1. Check the disk utilization on the host.

None

# df -h

1. If one or more partitions are full, identify where space is being utilized and clear the necessary amount of disk space so that the Platform9 components may continue to function.

None

# du -h /partition | sort -h

1. If there is a sufficient amount of disk space available (<97%) on all of the root disk partitions and yum commands fail with the same error, the package database may need to be rebuilt.

None

# rpm --rebuilddb

1. Restart the pf9-hostagent service.

None

# systemctl restart pf9-hostagent

Additional Information

How to Fix Corrupted Package Database for Yum