Host disconnected due to pf9-comms proxy issue
Problem
- Host is disconnected from the management plane, even though all the pf9 services [hostagent, comms and sidekick] are actively running.
- Host is able to reach management plane over 443 port yet it is showing disconnected.
Pf9-hostagent logs
2024-03-10 16:49:39,257 - slave.py ERROR - Connection error. Retrying in 10 seconds.Traceback (most recent call last): File "/opt/pf9/hostagent/lib/python3.6/site-packages/bbslave/slave.py", line 129, in reconnect_loop channel_retry_period=retry_period) File "/opt/pf9/hostagent/lib/python3.6/site-packages/bbslave/session.py", line 796, in start socket_timeout=socket_timeout) # in secs File "/opt/pf9/hostagent/lib/python3.6/site-packages/bbcommon/amqp.py", line 240, in dual_channel_io_loop conn.ioloop.start() File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 354, in start self.poll() File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 602, in poll self._process_fd_events(fd_event_map, write_only) File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 443, in _process_fd_events handler(fileno, events, write_only=write_only) File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 364, in _handle_events self._handle_read() File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 412, in _handle_read return self._handle_disconnect() File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 288, in _handle_disconnect self._adapter_disconnect() File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 95, in _adapter_disconnect super(SelectConnection, self)._adapter_disconnect() File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 154, in _adapter_disconnect self._check_state_on_disconnect() File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 169, in _check_state_on_disconnect raise exceptions.IncompatibleProtocolErrorpika.exceptions.IncompatibleProtocolErrorPf9-sidekick logs
[2024-03-10 16:49:16.647] [ERROR] client - request or response processing failed: RequestError: Error: read ECONNRESET at new RequestError (/opt/pf9/sidekick/node_modules/request-promise/lib/errors.js:11:15) at Request.RP$callback [as _callback] (/opt/pf9/sidekick/node_modules/request-promise/lib/rp.js:60:32) at self.callback (/opt/pf9/sidekick/node_modules/request/request.js:185:22) at Request.emit (events.js:400:28) at Request.onRequestError (/opt/pf9/sidekick/node_modules/request/request.js:877:8) at ClientRequest.emit (events.js:412:35) at TLSSocket.socketErrorListener (_http_client.js:475:9) at TLSSocket.emit (events.js:400:28) at emitErrorNT (internal/streams/destroy.js:106:8) at emitErrorCloseNT (internal/streams/destroy.js:74:3) at processTicksAndRejections (internal/process/task_queues.js:82:21)Environment
- Platform9 Managed OpenStack - All versions
- Platform9 Managed Kubernetes - All versions
Cause
- Service
pf9-commsfailing to establish communication withsni-brokerandsni-vouchspecifically. Rest of the tunnels are intact.
[2024-03-10 16:34:23.625] [ERROR] sni-broker-openstack.v4.<DU-FQDN>-127.0.0.1-5673-3 - TLS socket for client 118749 error: Error: read ECONNRESET[2024-03-10 16:34:23.626] [ERROR] sni-broker-openstack.v4.<DU-FQDN>-127.0.0.1-5673-3 - TLS socket for client 118750 error: Error: read ECONNRESET[2024-03-10 16:35:13.927] [ERROR] sni-vouch-private.v4.<DU-FQDN>-127.0.0.1-5673-3 - TLS socket for client 118735 error: Error: read ECONNRESETResolution
- Proxy for
pf9-commsservice needs to be enabled as per KB; that allows the host to connect to Platform9 Management Plane via proxy server instead of direct connectivity.
Was this page helpful?