Host disconnected due to pf9-comms proxy issue

Problem

• Host is disconnected from the management plane, even though all the pf9 services [hostagent, comms and sidekick] are actively running.
• Host is able to reach management plane over 443 port yet it is showing disconnected.
• Pf9-hostagent logs

2024-03-10 16:49:39,257 - slave.py ERROR - Connection error. Retrying in 10 seconds.
Traceback (most recent call last):
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/bbslave/slave.py", line 129, in reconnect_loop
    channel_retry_period=retry_period)
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/bbslave/session.py", line 796, in start
    socket_timeout=socket_timeout)  # in secs
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/bbcommon/amqp.py", line 240, in dual_channel_io_loop
    conn.ioloop.start()
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 354, in start
    self.poll()
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 602, in poll
    self._process_fd_events(fd_event_map, write_only)
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 443, in _process_fd_events
    handler(fileno, events, write_only=write_only)
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 364, in _handle_events
    self._handle_read()
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 412, in _handle_read
    return self._handle_disconnect()
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 288, in _handle_disconnect
    self._adapter_disconnect()
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 95, in _adapter_disconnect
    super(SelectConnection, self)._adapter_disconnect()
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 154, in _adapter_disconnect
    self._check_state_on_disconnect()
  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 169, in _check_state_on_disconnect
    raise exceptions.IncompatibleProtocolError
pika.exceptions.IncompatibleProtocolError

• Pf9-sidekick logs

C

[2024-03-10 16:49:16.647] [ERROR] client - request or response processing failed: RequestError: Error: read ECONNRESET
    at new RequestError (/opt/pf9/sidekick/node_modules/request-promise/lib/errors.js:11:15)
    at Request.RP$callback [as _callback] (/opt/pf9/sidekick/node_modules/request-promise/lib/rp.js:60:32)
    at self.callback (/opt/pf9/sidekick/node_modules/request/request.js:185:22)
    at Request.emit (events.js:400:28)
    at Request.onRequestError (/opt/pf9/sidekick/node_modules/request/request.js:877:8)
    at ClientRequest.emit (events.js:412:35)
    at TLSSocket.socketErrorListener (_http_client.js:475:9)
    at TLSSocket.emit (events.js:400:28)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)

Environment

• Platform9 Managed OpenStack - All versions
• Platform9 Managed Kubernetes - All versions

Cause

• Service pf9-comms failing to establish communication with sni-broker and sni-vouch specifically. Rest of the tunnels are intact.

C

[2024-03-10 16:34:23.625] [ERROR] sni-broker-openstack.v4.<DU-FQDN>-127.0.0.1-5673-3 - TLS socket for client 118749 error: Error: read ECONNRESET
[2024-03-10 16:34:23.626] [ERROR] sni-broker-openstack.v4.<DU-FQDN>-127.0.0.1-5673-3 - TLS socket for client 118750 error: Error: read ECONNRESET
[2024-03-10 16:35:13.927] [ERROR] sni-vouch-private.v4.<DU-FQDN>-127.0.0.1-5673-3 - TLS socket for client 118735 error: Error: read ECONNRESET

Resolution

• Proxy for pf9-comms service needs to be enabled as per KB; that allows the host to connect to Platform9 Management Plane via proxy server instead of direct connectivity.