Glance: Failed to Upload Image Data Due to HTTP Error

Problem

• Uploading an image to Glance or creating a snapshot of an instance fails silently.

• The following exception is seen in /var/log/pf9/ostackhost.log on the host where the instance resides.

None

682 INFO nova.image.pf9_glance [req-4941d293-b910-4c01-9406-ecd62f791127 [e-mail] Platform9] Exception http://x.x.x.x:9292 calling 'upload' with args (u'99358bc4-a28f-4a9b-9371-5f57a4aa4a30', [open file '/opt/pf9/data/instances//snapshots/tmpXzQvoR/1adc831bbf5f47cb8cf50de291685e2b', mode 'r' at 0x7f66645e4810]), {}: 503 Service Unavailable

• On the Glance node, the Glance API logs in /var/log/glance/api.log logs the following error.

None

121 196167 ERROR glance.api.v2.image_data [req-5ed23d37-6a16-4bdb-ac30-3e7844d65775 fe3228bb597f49d4b9931c7f5a52e2e6 55c66611af2543bba534d89dedf21a1d - default default] Failed to upload image data due to HTTP error 277 196167 INFO eventlet.wsgi.server [req-5ed23d37-6a16-4bdb-ac30-3e7844d65775 fe3228bb597f49d4b9931c7f5a52e2e6 55c66611af2543bba534d89dedf21a1d - default default] 10.4.10.111 - - [26/Dec/2018 18:26:06] "PUT /v2/images/99358bc4-a28f-4a9b-9371-5f57a4aa4a30/file HTTP/1.1" 503 351 0.638097

Environment

• Platform9 Managed OpenStack - v3.7.0 and Higher

Cause

The directory on the Glance nodes where the snapshots/images are uploaded have incorrect permissions.

Resolution

1. The directory where the Glance images are uploaded should have the following permissions.

• Ownership: pf9:pf9group

• Permission: 755

Additional Information

• In this scenario, enabling the debug mode for Glance won't help.

• To identify the issue, capture the strace of the Glance process when uploading an image or taking a snapshot.

None

# strace -xvfttT -s 2048 -o /tmp/glance_strace.txt -p [pid of glance api process]

• To stop the strace capture, press Ctrl+C.