Host Aggregates Not Showing Availability Status in UI, Instead Displaying "Hosts are Loading"

Troubleshooting High Availability issues in Platform9 Managed OpenStack v5.10.1 and higher This guide explains how to resolve the "Hosts are loading" status in the UI caused by pf9-hamgr service hitti

Problem

The Host Aggregates of the clusters displayed the status "Hosts are loading" instead of the expected availability status in the Platform9 UI. High Availability (HA) functionality appeared to be affected.

Environment

• Platform9 Managed OpenStack - v5.10.1 and Higher

• Component - pf9-hamgr on the control plane

Cause

The pf9-hamgr service hit the file descriptor limit, which caused:

• Failure to read certificate files required for operation

• Broken connectivity to the Nova API

• The HA Manager service becoming unresponsive

• CURL requests to HA Manager endpoint returning 502 errors

This results in the UI being unable to retrieve and display the Host Aggregate status properly.

Info

The following diagnosis, resolution steps involve commands that must be run on the Platform9 control plane. Please contact Platform9 Support for assistance.

Diagnosis

• From the Management Plane, the /var/log/pf9/hamgr/hamgr.log logs showed:

Management Plane

OSError: [Errno 24] Too many open files: '/etc/pf9/hamgr/certs/ca/ca.cert.pem'
Failed to establish a new connection: [Errno 24] Too many open files'

• The curl requests to HA reports below 502 error:

Management Plane

$ curl -kv https://<FQDN>/hamgr/v1/ha
< HTTP/1.1 502 Bad Gateway

• The pf9-hamgr-server process was not listening on the expected port , indicating it had stopped responding:

Management Plane

$ netstat -tulpn | grep <hamgr-process-id>
[no output]

• Too many open files by hamgr process:

Management Plane

$ lsof -p <hamgr-process-id> | wc -l
648

Resolution

Restarting the HA Manager service, resolves the issue:

Management Plane

# systemctl restart pf9-hamgr-server.service

Post-restart:

• The service resumed listening on port

• Open file descriptors dropped to normal operating levels

• Error logs stopped appearing

Validation

From the Management Plane:

After the restart:

• Verified that pf9-hamgr-server is listening on port :

Management Plane

$ netstat -tulpn | grep <hamgr-process-id>
tcp        0      0 0.0.0.0:[PORT-NUMBER]           0.0.0.0:*               LISTEN      [HAMGR-PROCESS-ID]/python

• Check the number of open files:

Management Plane

$ lsof -p <hamgr-process-id> | wc -l
110

From the UI:

In the UI, the user can verify the resolution by navigating to the Infrastructure Aggregates. The aggregates should display the status "Available" instead of "Hosts are loading".