Hostagent Fails to Install Packages During Host Upgrade

Problem

• During a host upgrade, the pf9-hostagent service fails to install its updated package, throwing the following error in /var/log/pf9/hostagent.log.

None

pf9_app.py INFO - Updating pf9-hostagent.3.9.0-2453.2b93f79
    session.py ERROR - Updating the pf9 host agent failed
    Traceback (most recent call last):
      File "/opt/pf9/hostagent/lib/python2.7/site-packages/bbslave/session.py", line 423, in update_agent
        agent_app_class, log)
      File "/opt/pf9/hostagent/lib/python2.7/site-packages/pf9app/algorithms.py", line 207, in process_agent_update
        new_agent.update()
      File "/opt/pf9/hostagent/lib/python2.7/site-packages/pf9app/pf9_app.py", line 426, in update
        self.app_db.update_package(local_path)
      File "/opt/pf9/hostagent/lib/python2.7/site-packages/pf9app/pf9_app_db.py", line 329, in update_package
        self.pkgmgr.update_from_file(path)
      File "/opt/pf9/hostagent/lib/python2.7/site-packages/pf9app/pf9_app_db.py", line 117, in update_from_file
        self.install_from_file(pkg_path)
      File "/opt/pf9/hostagent/lib/python2.7/site-packages/pf9app/pf9_app_db.py", line 107, in install_from_file
        raise InstallOperationFailed()
    InstallOperationFailed

• For other packages, it throws the following error.

None

pf9_app.py INFO - Installing pf9-ostackhost.3.9.0-3442
    session.py ERROR - Exception during apps processing: [class 'pf9app.exceptions.InstallOperationFailed']
    session.py INFO - Converge failed

• An attempt to manually install the package fails with the following error.

None

$ sudo /opt/pf9/hostagent/bin/pf9-apt install /var/cache/pf9apps/pf9-hostagent/3.9.0-2453.2b93f79/pf9-hostagent-3.9.0-2453.2b93f79.x86_64.deb
    Setting up libvirt-bin (1.3.1-1ubuntu10.25) ...
    Job for libvirt-bin.service failed because the control process exited with error code. See "systemctl status libvirt-bin.service" and "journalctl -xe" for details.
    invoke-rc.d: initscript libvirt-bin, action "restart" failed.
    ● libvirt-bin.service - Virtualization daemon
       Loaded: loaded (/lib/systemd/system/libvirt-bin.service; enabled; vendor preset: enabled)
       Active: activating (auto-restart) (Result: exit-code) since Mon 2019-03-18 20:50:40 PDT; 8ms ago
         Docs: man:libvirtd(8)
               http://libvirt.org
      Process: 161154 ExecStart=/usr/sbin/libvirtd $libvirtd_opts (code=exited, status=1/FAILURE)
     Main PID: 161154 (code=exited, status=1/FAILURE)
        Tasks: 633
       Memory: 5.3G
          CPU: 371ms
       CGroup: /system.slice/libvirt-bin.service
               ├─  3457 /usr/sbin/libvirtd -d -l
               ├─ 13235 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
               ├─ 13236 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
    host systemd[1]: Failed to start Virtualization daemon.
    host systemd[1]: libvirt-bin.service: Unit entered failed state.
    host systemd[1]: libvirt-bin.service: Failed with result 'exit-code'.
    dpkg: error processing package libvirt-bin (--configure):
    subprocess installed post-installation script returned error exit status 1
    Errors were encountered while processing:
     libvirt-bin
    E:Sub-process /usr/bin/dpkg returned an error code (1)
    Failed to install package: /var/cache/pf9apps/pf9-hostagent/3.9.0-2453.2b93f79/pf9-hostagent-3.9.0-2453.2b93f79.x86_64.debTraceback (most recent call last):
      File "/opt/pf9/hostagent/bin/pf9-apt", line 90, in
        _install_package(package, cache)
      File "/opt/pf9/hostagent/bin/pf9-apt", line 68, in _install_package
        cache.commit()
      File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 552, in commit
        raise SystemError("installArchives() failed")
    SystemError: installArchives() failed
    Traceback (most recent call last):
      File "/opt/pf9/hostagent/bin/pf9-apt", line 90, in
        _install_package(package, cache)
      File "/opt/pf9/hostagent/bin/pf9-apt", line 68, in _install_package
        cache.commit()
      File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 552, in commit
        raise SystemError("installArchives() failed")
    SystemError: installArchives() failed

Environment

• Platform9 Managed OpenStack - v3.3.0 and Higher
• Ubuntu 16.04

Cause

• Even though the libvirt-bin package is installed, it is half-configured, as shown below.

None

$ dpkg -l | grep libvirt-bin
    iF  libvirt-bin                      1.3.1-1ubuntu10.25                         amd64        programs for the libvirt library
    $

• The package post-install script to start the libvirt-bin service fails.
• Systemd reports a failure because the '-d' (daemon) flag was passed to the binary. This isn't compatible with systemd as it forces the libvirtd process to fork, thus reporting the service as failed, even when the forked process is running in the background.
• The drawback is that the forked process cannot be managed with systemd, and must be killed manually.

Resolution

1. Remove '-d' option from the parameter libvirtd_opts in /etc/default/libvirt-bin.

None

# sed -i 's/^libvirtd_opt.*/libvirtd_opts="-l"/g' /etc/default/libvirt-bin

1. Kill the libvirtd daemon with the following command.

None

# pkill libvirtd

1. Fix the half configured libvirt-bin package.

None

# apt install -f

1. Restart the libvirt-bin service.

None

# systemctl restart libvirt-bin

1. Contact Platform9 Support to re-run the host upgrades.

Additional Information

• [Ubuntu] #1567272 - systemd claims libvirt-bin is dead on xenial
• Libvirt Change Log:

None

libvirt (0.9.13~rc1-2) experimental; urgency=low

      * [628a05a] Add empty systemd dir on kFreeBSD to make dh_install pass
      * [e5b4fbf] Remove -d from libvirtd_opts
        so we can use the same options in the systemd unit file without having
        to fork there too.