pf9-ostackhost Service Fails to Connect to Libvirt

Problem

• The host is reported offline in Clarity UI.
• /var/log/pf9/hostagent.log showsthat pf9-ostackhost service has failed to start.

None

815 - pf9_app.py INFO - Setting service state pf9-ostackhost.3.5.0-3246. Command: sudo /etc/init.d/pf9-ostackhost start

• /var/log/pf9/ostackhost.log contains the message "Failed to connect socket to /var/run/libvirt/libvirt-sock".

None

310 ESC[00;36mINFO nova.service [-ESC[00;36m] ESC[00;36mStarting compute node (version 2013.2.1)322 ESC[00;36mINFO nova.virt.libvirt.driver [-ESC[00;36m] ESC[00;36mConnection event '0' reason 'Failed to connect to libvirt'783 ESC[01;31mERROR nova.virt.libvirt.host [req-e5803ac8-b08f-4d3f-a6c9-93f36f37d0e0 None NoneESC[01;31m] ESC[01;31mConnection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

• /var/log/libvirt/libvirtd.log contains the message "Unable to bind to port".

None

794+0000: 20652: info : libvirt version: 1.2.2794+0000: 20652: error : virNetSocketNewListenTCP:311 : Unable to bind to port: Address already in use

• In /var/log/syslog.log, it is observed that the libvirt service has stopped and is not able to restart.

None

kernel: [69549.764040] init: libvirt-bin main process (20652) terminated with status 6kernel: [69549.764068] init: libvirt-bin respawning too fast, stoppedkernel: [69549.766659] init: libvirt-bin post-start process (20663) terminated with status 1​

None

# service libvirt-bin status  libvirt-bin stop/waiting

Environment

• Platform9 Managed OpenStack - v3.5.0 and Higher
• Ubuntu 14.04

Cause

A stale libvirt process is holding the lock on the service port.

Resolution

1. Kill the stale libvirtd daemon.

None

# kill [pid]

1. Start the libvirt-bin service.

None

# service libvirt-bin start

1. Start the pf9-ostackhost service.

None

# service pf9-ostackhost start