Host Unable to Communicate With Management Plane: "ETIMEDOUT"

Problem

• A Platform9 managed host shows as offline in Clarity UI.

• On the host, the following traceback and subsequent error are observed in /var/log/pf9/hostagent.log.

None

session.py WARNING - Not sending status message because channel is closed
session.py INFO - Using the default virtual host '/' on the AMQP broker localhost
slave.py ERROR - Connection error. Retrying in 10 seconds.
Traceback (most recent call last):
File "/opt/pf9/hostagent/lib/python2.7/site-packages/bbslave/slave.py", line 83, in reconnect_loop
channel_retry_period=retry_period)
File "/opt/pf9/hostagent/lib/python2.7/site-packages/bbslave/session.py", line 712, in start
socket_timeout=socket_timeout) # in secs
File "/opt/pf9/hostagent/lib/python2.7/site-packages/bbcommon/amqp.py", line 238, in dual_channel_io_loop
conn.ioloop.start()
File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 354, in start
self.poll()
File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 602, in poll
self._process_fd_events(fd_event_map, write_only)
File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 443, in _process_fd_events
handler(fileno, events, write_only=write_only)
File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adaptersbase_connection.py", line 364, in _handle_events
self._handle_read()
File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 412, in _handle_read
return self._handle_disconnect()
File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 288, in _handle_disconnect
self._adapter_disconnect()
File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 95, in _adapter_disconnect
super(SelectConnection, self)._adapter_disconnect()
File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 154, in _adapter_disconnect
self._check_state_on_disconnect()
File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 169, in _check_state_on_disconnect
raise exceptions.IncompatibleProtocolError
IncompatibleProtocolError

• Additionally, in /var/log/pf9/comms.log, an 'ETIMEDOUT' error is observed for the IPv4 address of the Management Plane.

None

[ERROR] sni-broker-openstack-v1-5673-3 - Server socket for client 32003 error: { [Error: connect ETIMEDOUT<IPv4_addrress>:443]
code: 'ETIMEDOUT',
errno: 'ETIMEDOUT',
syscall: 'connect',
address: '<IPv4_addrress>',
port: 443 }
Error: connect ETIMEDOUT<IPv4_addrress>:443
at Object.exports._errnoException (util.js:907:11)
at exports._exceptionWithHostPort (util.js:930:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1081:14)
[DEBUG] sni-broker-openstack-v1-5673-3 - Server socket for client 32003 closed with error ... remaining: 17
[WARN] sni-broker-v1-5672-2 - Forcefully destroying server socket for client 32025
[DEBUG] sni-broker-v1-5672-2 - Server socket for client 32025 closed normally ... remaining: 3
[WARN] sni-broker-openstack-v1-5673-3 - Forcefully destroying client 32003
[DEBUG] sni-broker-openstack-v1-5673-3 - Client 32003 socket closed normally ... remaining: 17
[ERROR] sni-broker-openstack-v1-5673-3 - Server socket for client 32005 error: { [Error: connect ETIMEDOUT<IPv4_addrress>:443]
code: 'ETIMEDOUT',
errno: 'ETIMEDOUT',
syscall: 'connect',
address: '<IPv4_addrress>',
port: 443 }

Environment

• Platform9 Managed OpenStack - All Versions

• Platform9 Managed Kubernetes - All Versions

Cause

Bi-directional communication is required on port 443 between the host and Platform9 Management Plane in order to orchestrate operations necessary to provision cloud resources.

Resolution

1. Disable any firewalls temporarily.

2. Otherwise, ensure bi-directional traffic between the host and Management Plane is allowed on port 443.

Additional Information

Platform9 Managed OpenStack Prerequisites for Linux KVM