Host Upgrade Install Fails When Attempting To Download hostagent.conf From Management Plane When Pro

Problem

Host upgrade install fails when attempting to download hostagent.conf from management plane. The proxy settings are in place. Post this failure, /var/opt/pf9/hostagent/XXXXX/desired_apps.json.part file and the /var/opt/pf9/hostagent/ directory itself is not seen. Restarting pf9-hostagent service recreates this directory back and then on a subsequent retry, host upgrade succeeds.

Hostagent logs

2023-05-05 18:26:36,376 - pf9_app_cache.py INFO - Using previously cached file /var/cache/pf9apps/pf9-hostagent/5.6.4-1078.9f795fe/pf9-hostagent-5.6.4-1078.9f795fe.x86_64.deb
2023-05-05 18:26:36,377 - pf9_app.py INFO - Updating pf9-hostagent.5.6.4-1078.9f795fe
2023-05-05 18:27:45,607 - pf9_app_db.py ERROR - Install command failed: sudo /opt/pf9/hostagent/bin/pf9-apt install /var/cache/pf9apps/pf9-hostagent/5.6.4-1078.9f795fe/pf9-hostagent-5.6.4-1078.9f795fe.x86_64.deb. Return code: 1, stdout: (Reading database ... 178992 files and directories currently installed.)
Preparing to unpack .../pf9-hostagent-5.6.4-1078.9f795fe.x86_64.deb ...
Downloading hostagent.conf from PF9
Failed to download hostagent.conf from PF9
, stderr: dpkg: error processing archive /var/cache/pf9apps/pf9-hostagent/5.6.4-1078.9f795fe/pf9-hostagent-5.6.4-1078.9f795fe.x86_64.deb (--install):
new pf9-hostagent package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
/var/cache/pf9apps/pf9-hostagent/5.6.4-1078.9f795fe/pf9-hostagent-5.6.4-1078.9f795fe.x86_64.deb
Failed to install package: /var/cache/pf9apps/pf9-hostagent/5.6.4-1078.9f795fe/pf9-hostagent-5.6.4-1078.9f795fe.x86_64.debTraceback (most recent call last):
Traceback (most recent call last):
File "/opt/pf9/hostagent/bin/pf9-apt", line 94, in <module>_install_package(package, cache)
File "/opt/pf9/hostagent/bin/pf9-apt", line 72, in _install_package package_path
Exception: Errors encountered while installing : /var/cache/pf9apps/pf9-hostagent/5.6.4-1078.9f795fe/pf9-hostagent-5.6.4-1078.9f795fe.x86_64.deb
2023-05-05 18:27:45,608 - session.py ERROR - Updating the pf9 host agent failed
Traceback (most recent call last):
File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 441, in update_agent
process_agent_update(agent_info, agent_app_db, app_cache,
File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/algorithms.py", line 216, in process_agent_update new_agent.update()
File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/pf9_app.py", line 437, in update
self.app_db.update_package(local_path)
File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/pf9_app_db.py",line 342, in update_package
self.pkgmgr.update_from_file(path)
File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/pf9_app_db.py",line 116, in update_from_file
self.install_from_file(pkg_path)
File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/pf9_app_db.py",line 106, in install_from_file
raise InstallOperationFailed()
pf9app.exceptions.InstallOperationFailed
2023-05-05 18:27:47,767 - session.py ERROR - Failed to save desired configuration: [Errno 2] No such file or directory: '/var/opt/pf9/hostagent/04755078-1580-4f3e-b886-d2e756d27ff0/desired_apps.json.part'
2023-05-05 18:27:47,769 - session.py INFO - Already converged. Idling...
2023-05-05 18:27:50,378 - session.py ERROR - timeout 120 /opt/pf9/hostagent/extensions/fetch_pod_info.py command failed: Command '['timeout', '120', '/opt/pf9/hostagent/extensions/fetch_pod_info.py']' returned non-zero exit status 1.

Environment

• Platform9 Managed Kubernetes - v5.5

Answer

• This is a known issue and Platform9 Engineering team is currently working on the CORE-1288.

Workaround

1. Try removing proxy environment variables from /etc/environment for worker nodes and added localhost to NO_PROXY variable on master nodes. Post doing this, host upgrades succeeds for most nodes.

2. Restart pf9-hostagent service.

3. Manually copy the .deb package from the active node and save it in the same location in the affected. Example: /var/cache/pf9apps/pf9-hostagent/5.6.4-1078.9f795fe/pf9-hostagent-5.6.4-1078.9f795fe.x86_64.deb

4. Restart the Platform9 services related to Hostagent, Comms and Sidekick.

Additional Information

To track the progress of the fix for this issue, open a support ticket mentioning the Jira CORE-1288.