Image Creation is Failing for Volume Size More than 100GB

Problem

When converting large volumes (100GB–400GB or more) from the block device service into images in the image service, the operation may fail or result in an image being created with zero size.

Environment

• Private Cloud Director Virtualization - v2025.10 to v2026.1.427
• Self-Hosted Private Cloud Director Virtualization - v2025.10 to v2026.1.427
• Component - Image Service

Cause

The issue arises from a combination of buffering behavior and timeout limitations during the volume-to-image conversion process. When a large upload is initiated, the web server first buffers the incoming data temporarily before forwarding it to the image service. The image service, in turn, writes this data to a staging area before finally committing it to backend storage. Although moving the temporary buffering path to a network file system helps reduce local disk exhaustion, it does not eliminate the dependency on internal timeout limits for handling large uploads.

One of the primary causes is that the default timeout configured between the block device service and the image service, typically around 600 seconds, is not sufficient for larger data transfers. As a result, the client connection may get interrupted during the upload process, leading to errors such as a disconnection while sending data. This ultimately results in incomplete uploads, where the image may be marked as successfully created but ends up having zero size due to the partial transfer.

Repeated errors in cindervolume-base.log:

/var/log/pf9/cindervolume-base.log

INFO cinder.image.pf9_glance [req-[REQ_UUID] None [TENANT]] Exception https://[IMAGE_HOST_IP]:9494 calling 'upload' with args ('[IMAGE_UUID]', <_io.BufferedReader name='/opt/pf9/pf9-cindervolume-base/state/mnt/[UUID]/volume-[VOLUME_UUID]'>), {}: Error communicating with https://[IMAGE_HOST_IP]:9494/v2/images/[IMAGE_UUID]/file: HTTPSConnectionPool(host='[IMAGE_HOST_IP]', port=9494): Read timed out. (read timeout=600.0)

Resolution

The engineering team has worked on an enhancement that is available starting v2026.4.89 (April Release) and above versions, which was tracked via PCD-5923.

If an upgrade is not a viable option, proceed with the Workaround

Workaround

Workaround changes will NOT persist across upgrades since they are overwritten during package updates and will need to be reapplied until upgraded to v2026.4.89 version

Update the web server proxy timeout settings to align with the block device service and image service timeout values, ensuring that the entire request flow can support longer upload durations without interruption.

Recommended Timeout Values by Volume Size

Volume Size	Expected Upload Time*	Recommended Timeout	Safety Buffer
10GB	5-10 minutes	1800s (30 min)	3x buffer
50GB	15-25 minutes	3600s (1 hour)	2.5x buffer
100GB	30-45 minutes	7200s (2 hours)	2.5x buffer
200GB	60-90 minutes	14400s (4 hours)	2.5x buffer
250GB	75-110 minutes	14400s (4 hours)	2x buffer
500GB	150-220 minutes	21600s (6 hours)	2x buffer

*Based on 100 MB/s network speed with 15% overhead

1

Backup the existing config

Take Backup of existing configurations from Image Library Host

Image Library Host

$ cp /etc/nginx/conf.d/glance-ssl.conf /etc/nginx/conf.d/glance-ssl.conf.bkp

Take Backup of existing configuration from Persistent Storage Host

Persistent Storage Host

$ cp /opt/pf9/pf9-cindervolume-base/lib/python3.9/site-packages/cinder/image/pf9_glance.py \
   /opt/pf9/pf9-cindervolume-base/lib/python3.9/site-packages/cinder/image/pf9_glance.py.bkp

2

Increase image upload timeout

On the Persistent storage host, check if a cinder_override.conf is present. If present update the file to add the timeout. If not create a new file and add the timeout values.

/opt/pf9/etc/pf9-cindervolume-base/conf.d/cinder_override.conf

# cinder_override.conf should have ownership set to pf9:pf9group
[DEFAULT]
glance_request_timeout = 14400

3

Redirect Nginx temporary buffer to shared storage

On the host with Image library role, update

/etc/nginx/conf.d/glance-ssl.conf

location / {
    client_body_temp_path /var/lib/glance/os_glance_staging_store 1 2;
    client_max_body_size 0;
    proxy_connect_timeout 300;     # 5 minutes
    proxy_send_timeout 14400;      # 4 hours (match Cinder) <--Add the timeout same as above
    proxy_read_timeout 14400;      # 4 hours (match Cinder) <-
    send_timeout 14400;            # 4 hours (match Cinder) <-
    proxy_pass http://127.0.0.1:9393;
}

4

Add timeout parameter in pf9_glance.py

Update pf9_glance.py file on Persistent Storage host. Apply this change on one Persistent Storage host first, confirm the service is healthy, then repeat on the remaining hosts

sudo python3 - <<'PYEOF'
p = "/opt/pf9/pf9-cindervolume-base/lib/python3.9/site-packages/cinder/image/pf9_glance.py"
lines = open(p).read().split("\n")
out, done, i = [], False, 0
while i < len(lines):
    s = lines[i]
    if not done and s.rstrip().endswith("glanceclient.Client(str(version), endpoint,"):
        ind = s[:len(s) - len(s.lstrip(" "))]      # exact leading spaces of the original line
        out += [
            ind + "client_kwargs = {",
            ind + "    'token': str(context.auth_token),",
            ind + "    'insecure': CONF.glance_api_insecure",
            ind + "}",
            ind + "timeout = getattr(CONF, 'glance_request_timeout', None)",
            ind + "if timeout is not None:",
            ind + "    client_kwargs['timeout'] = timeout",
            "",
            ind + "client = glanceclient.Client(str(version), endpoint, **client_kwargs)",
        ]
        i += 2          # skip the original 2-line statement
        done = True
        continue
    out.append(s)
    i += 1
assert done, "original client line not found - file may differ, no change made"
open(p, "w").write("\n".join(out))
print("patched")
PYEOF

Post applying above changes, restart the service

Persistent Storage Host

$ systemctl restart pf9-cindervolume-base.service
$ systemctl reload nginx

Additional Information

Nginx temporary buffering requires sufficient disk space, and it is recommended to use an NFS-backed path to prevent exhaustion of the local filesystem during large uploads. The image service staging area will continue to generate temporary files throughout the upload process, which is expected behavior, particularly for large volume-to-image operations.