Live Migration Fails With Error: "Too many open files

Problem

• Live migration of an instance fails with error "Too many open files" as see in /var/log/nova/nova-conductor.log.

None

04 3437 WARNING nova.scheduler.utils [req-a34cc0ab-5cc2-4332-b2e4-9c2b16c934d8 9f39d094aa9649da8fdd01f84d3a6a04-57a5eabb1acb4f8a90931897bafb19fc - - -] Failed to compute_task_migrate_server: [Errno [Errno 24] Too many open files

• Below errors are seen in /var/log/pf9/ostackhost.log.

None

442 TRACE nova.compute.manager [instance: 31879d2c-6096-4db6-aadd-7d0201226e9d] ConnectFailure: Unable to establish connection to http://localhost:8158/keystone/v3/auth/tokens: HTTPConnectionPool(host='localhost', port=8158): Max retries exceeded with url: /keystone/v3/auth/tokens (Caused by NewConnectionError('[requests.packages.urllib3.connection.HTTPConnection object at 0x7f20b45c5550]: Failed to establish a new connection: [Errno 24] Too many open files',))

Environment

• Platform9 Managed OpenStack - v3.6.0 and Higher

Cause

For the majority of operating systems, 1024 is the maximum number of open files per process. But for the pf9-ostackhost process, we can see that the threshold has been crossed and there are 1158 files opened.

None

$ systemctl status pf9-ostackhost● pf9-ostackhost.service - Platform9 OpenStack Compute service Loaded: loaded (/lib/systemd/system/pf9-ostackhost.service; disabled; vendor preset: enabled) Active: active (running) since Thu XXXX-XX-XX YY:YY:YY PDT; 11 months 25 days ago Main PID: 4579 (pf9-ostackhost) Tasks: 24 Memory: 433.9M CPU: 4month 2d 10h 41min 57.056s CGroup: /system.slice/pf9-ostackhost.service ├─ 4579 /opt/pf9/venv/bin/python /opt/pf9/venv/bin/pf9-ostackhost --config-dir /opt/pf9/etc/nova/conf.d/ --log-file /var/log/pf9/ostackhost.log ├─ 5468 /opt/pf9/venv/bin/python /opt/pf9/venv/bin/privsep-helper --config-dir /opt/pf9/etc/nova/conf.d --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpawAsR0/privs └─418015 /opt/pf9/venv/bin/python /opt/pf9/venv/bin/privsep-helper --config-dir /opt/pf9/etc/nova/conf.d --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpS_M48Q/privsep

None

$ cat /proc/4579/limitsLimit Soft Limit Hard Limit UnitsMax open files 1024 4096 files

None

$ ulimit -a | grep openopen files (-n) 1024

None

$ lsof -p 4579 | wc -l1158

This indicates that pf9-ostackhost service has 1158 open files and has crossed the limit of 1024.

Resolution

Restart pf9-ostackhost service on the affected host.