VM Console Access Failures Related to noVNC Service

Problem

VM console access fails with a “site can’t be reached” error when attempting to open the console. Attempts to connect to the console proxy port (6080), including telnet or direct connection tests, also time out.

Environment

• Private Cloud Director Virtualization - v2025.4 and Higher
• Private Cloud Director Kubernetes – v2025.4 and Higher
• Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
• Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher
• Component - novncproxy service

Cause

The issue is often caused by the novncproxy service on the affected hypervisor getting into a bad state due to:

• Stale TCP connections in the CLOSE_WAIT state, indicating the service did not properly close old client sessions.
• Possible stale processes or token mappings, leading to connection failures.
• This condition prevents the service from establishing new console connections for VMs on that hypervisor.

Diagnostics

When VM console access fails (e.g., “site can’t be reached” or "timeout"), perform the following checks:

1. Check service status from affected hosts:

$ systemctl status pf9-novncproxy.service

Ensure the service is active and running without any errors.

2. Validate time synchronisation

$ timedatectl status

Look for “System clock synchronized: yes” → means NTP/chrony is active.

Confirm the local time shown in the timedatectl status output matches the expected current time. If the host clock is out of sync, it can cause authentication token mismatches during VM console access. Correcting the time and ensuring synchronization with NTP/chrony is essential before retrying console access.

3. Test local connectivity from affected host to console port:

$ curl -vk http://127.0.0.1:6080/

$ curl -vk https://<HOST_IP>:6080/

If you are seeing a timeout here, it indicates the console proxy service is not responding to connections. This could be due to firewall rules, routing issues, SSL problems, or the service being in a stale state. In this case, continue with reviewing the novncproxy logs and other diagnostics to narrow down the cause.

4. Review novncproxy logs:

$ less /var/log/pf9/novncproxy.log

Review the noVNC logs to confirm whether new entries are being written when you attempt console access. If the logs stop updating, this can indicate the service is stuck in a stale state. In this case, continue by checking for stale or blocked connections.

5. Check for stale or blocked connections:

$ lsof -i :6080

If multiple CLOSE_WAIT entries are observed for pf9-novncproxy, it indicates that remote connections were closed but the service did not release those sockets. Such stale connections can accumulate and eventually block new console sessions.

Resolution

Depending on the findings in the Diagnostics section, use the corresponding resolution:

1. If service is in-active or unhealthy

$ sudo systemctl restart pf9-novncproxy.service

Restarts the noVNC proxy and re-establishes connections.

2. If stale TCP connections (CLOSE_WAIT) are found. Then restart the novncproxy service (as above) to clear the stale sessions.
3. If time drift is detected as, correct the time using your NTP/chrony configuration and restart the service:

$ sudo systemctl restart chronyd   # or ntpd / timesyncd depending on setup

$ timedatectl status

$ ntpq -p 2>/dev/null || chronyc sources 

$ sudo systemctl restart pf9-novncproxy.service

These commands restart the system’s NTP/chrony service to re-sync the clock, check the current time status and offset, verify synchronization sources, and then restart the noVNC proxy service to ensure fresh tokens are generated using the corrected system time.

Additional Information

Additionally, reviewing system logs (e.g., /var/log/syslog or /var/log/messages) and /var/log/pf9/pf9-ostackhost.log can provide deeper insight into underlying issues affecting console connectivity. These logs may reveal host-level errors, service crashes, or dependency failures that are not visible in the noVNC logs, and can be valuable for further analysis before taking corrective action.