Migrations Failing After Host Reprovisioned

Problem

Migrations of instances are failing to a specific host after the host was re-provisioned.

Log /var/log/pf9/ostackhost.log

2022-02-14 07:40:05.963 ERROR oslo_messaging.rpc.server [req-8a10be95-3555-4c99-807e-e8ddbef6d01f Geraldo Macedo transition1] Exception during message handling: nova.exception.ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.

Command: ssh -o BatchMode=yes <Destination Host IP>  mkdir -p /pf9/instances/<Instance_uuid>

Exit code: 255

Stdout: ''

Stderr: 'No ECDSA host key is known for <Destination Host IP> and you have requested strict checking.\r<br>Host key verification failed.\r<br>'

Environment

• Platform9 Managed OpenStack - All Versions

• Nova

Cause

During a migration attempt, the pf9-ostackhost service is responsible to perform an ssh on the destination node and create a directory for the instance to store the disk. The Error message is observed when the source host has strict checking enabled in the sshd service configuration file and the key for the remote host is not correct.

Resolution

1. As a workaround, if the destination host is verified and correct then disable the strict key check policy in sshd service and restart the service on the source host.

ssh config

$ grep -v "^#" .ssh/config
Host *
  User pf9
  StrictHostKeyChecking yes <<< change this value to "no"
  IdentityFile /opt/pf9/home/.ssh/id_rsa
  PasswordAuthentication no

ssh config

# systemctl restart sshd

Additional Information

https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html