Management Plane Upgrade Fails with "Connection timed out" Error.

Problem:

• Management Upgrade failed with below mentioned error in the airctl-logs/pf9deploy-prepare-restore.log file.

Error

2023-06-13T05:36:25.811Z info INFO:pf9deploy.server.util.shell:| fatal: [airctl-1.pf9.localnet]: UNREACHABLE! => {"changed": false, "msg": "Connection timed out.", "unreachable": true}
 2023-06-13T05:36:25.811Z info INFO:pf9deploy.server.util.shell:| to retry, use: --limit @/ansible-stack/mysql-configure-only.retry
 2023-06-13T05:36:25.811Z info INFO:pf9deploy.server.util.shell:| 
 2023-06-13T05:36:25.811Z info INFO:pf9deploy.server.util.shell:| PLAY RECAP *********************************************************************
 2023-06-13T05:36:25.811Z info INFO:pf9deploy.server.util.shell:| airctl-1.pf9.localnet      : ok=0    changed=0    unreachable=1    failed=0

Environment:

• Platform9 Edge Cloud - v5.3 and Higher

• Airctl

Troubleshooting:

• Try to ssh to the deployment host using FQDN and confirm if it fails with Operation not permitted error.

SSH error

$ ssh airctl-1.pf9.localnet
Warning: Permanently added 'airctl-1.pf9.localnet,10.10.13.133' (ECDSA) to the list of known hosts.
-bash: ulimit: open files: cannot modify limit: Operation not permitted

Resolution:

• Please perform below mentioned changes on the Deployment Host.

• Add below mentioned entries in file /etc/security/limits.conf

Javascript

* soft     nproc          1048576
* hard     nproc          1048576
* soft     nofile         1048576
* hard     nofile         1048576
root soft     nproc          1048576
root hard     nproc          1048576
root soft     nofile         1048576
root hard     nofile         1048576

• Add below mentioned line in /etc/pam.d/sshd file.

Javascript

session    required   pam_limits.so