How to Migrate the NFS Filer Backing an NFS Backend

Problem

When the storage team has replicated NFS exports to a new filer and the goal is to redirect the existing PCD control plane and hypervisors to the new network path — without re-creating volumes, re-attaching disks, or rebuilding VMs — the Cinder NFS backend, hypervisor mounts, and underlying databases must all be updated in a coordinated sequence. This article describes that process for migrating from one NFS filer (including non-NetApp servers) to another.

Environment

• Private Cloud Director Virtualization - v2025.10 and Higher
• Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher
• Component: Storage Service

Procedure

Complete all steps in order during a scheduled maintenance window. VM downtime is required.

Prerequisites

• A maintenance window with planned VM downtime.
• A freeze on new VM creation and VM power-on is in place for the hypervisors being modified during the window.
• The storage team has already replicated the NFS exports to the new filer and the new exports are accessible from all hypervisors.
• The following placeholder values are known and ready to substitute before running any command:

Placeholder	Meaning	Example
<OLD_FILER_FQDN>	FQDN of the current NFS filer	oldfiler.example.com
<NEW_FILER_FQDN>	FQDN of the target NFS filer	newfiler.example.com
<EXPORT_1>, <EXPORT_2>	NFS export paths used by Cinder	/nfs_pcd01, /nfs_pcd02
<BACKEND_NAME>	Name of the Cinder volume backend in the cluster blueprint	site_netapp
<HV_HOST_N>	Hostnames of the hypervisors mounting the backend	hv0001, hv0002

1

Shut Down All VMs on the Affected Hypervisors

VMs must be cleanly powered off before the storage path is changed. From the PCD UI, select the VMs and stop them in batches (preferred for large clusters).

Wait until every VM reaches SHUTOFF:

Controller / OpenStack CLI

$ openstack server list --all-projects --status ACTIVE

Confirm at the libvirt layer on each hypervisor — the output must be empty:

Hypervisor Node

$ sudo virsh list

2

Update the PCD Cluster Blueprint

In the PCD UI, navigate to:

Infrastructure → Cluster Blueprint → Volume Backend Configuration → <BACKEND_NAME>

Update the following fields:

• nfs_mount_points — change from <OLD_FILER_FQDN>:<EXPORT_1> and <OLD_FILER_FQDN>:<EXPORT_2> to <NEW_FILER_FQDN>:<EXPORT_1> and <NEW_FILER_FQDN>:<EXPORT_2>.
• nfs_mount_options — leave unchanged unless the new filer requires different options. A typical value is vers=3,proto=tcp,lookupcache=pos,nolock,noacl.

Click Apply.

Verify on each hypervisor that the new mounts are active:

Hypervisor Node

$ mount | grep -i pcd

Sample Output

[NEW_FILER_FQDN]:/nfs_pcd01 on /opt/pf9/... type nfs (...)
[NEW_FILER_FQDN]:/nfs_pcd02 on /opt/pf9/... type nfs (...)

The new <NEW_FILER_FQDN> mounts should appear. Old <OLD_FILER_FQDN> entries may still appear transiently and will be cleaned up in Step 5.

3

Update the Cinder and Nova Databases

These database updates rewrite the embedded NFS network path on every existing volume and block device mapping so that volume operations and VM boot continue to find their backing files on the new filer.

Self-Hosted only: Self-hosted customers can run these updates against their own control-plane database. SaaS customers cannot perform this step independently — contact Platform9 Support to have this completed on their behalf.

Use the following command from Control Plane nodes to login to Cinder DB

Control Plane Node

$ DB_PASS=$(kubectl exec -i -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c \
    "DB=\$(consul-dump-yaml --start-key customers/<CUSTOMER_ID>/regions/<REGION_UUID>/db | awk '/dbserver:/ {print \$2}'); \
     consul-dump-yaml --start-key customers/<CUSTOMER_ID>/dbservers/\$DB | awk '/admin_pass:/ {print \$2}'")

$ IP=$(kubectl get pods -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=mysql -o jsonpath="{.items[0].metadata.name}") \
    --template '{{.status.podIP}}')

$ kubectl exec -it -n "<REGION_NS>" \
    $(kubectl get po -n "<REGION_NS>" -l du-app=resmgr -o jsonpath="{.items[0].metadata.name}") \
    -c resmgr -- /bin/bash -c "mysql -u root -h $IP -p$DB_PASS -D cinder"

Run each block independently. Capture the Rows matched output from each UPDATE statement for the record.

3a — Cinder provider_location

Control Plane Node — MySQL - Cinder

USE cinder; 

SELECT COUNT(*) AS to_update
  FROM volumes
 WHERE provider_location LIKE '<OLD_FILER_FQDN>:%'
   AND deleted = 0;

UPDATE volumes
   SET provider_location = REPLACE(provider_location,
       '<OLD_FILER_FQDN>:', '<NEW_FILER_FQDN>:')
 WHERE provider_location LIKE '<OLD_FILER_FQDN>:%'
   AND deleted = 0;

SELECT COUNT(*) AS still_old
  FROM volumes
 WHERE provider_location LIKE '<OLD_FILER_FQDN>:%'
   AND deleted = 0;

Sample Output

+----------+
| to_update|
+----------+
|       42 |
+----------+

Query OK, 42 rows affected

+----------+
| still_old|
+----------+
|        0 |   <-- must return 0
+----------+

3b — Cinder host

Control Plane Node — MySQL

USE cinder;

UPDATE volumes
   SET host = REPLACE(host,
       '<OLD_FILER_FQDN>:', '<NEW_FILER_FQDN>:')
 WHERE host LIKE '%<OLD_FILER_FQDN>:%'
   AND deleted = 0;

3c — Nova block_device_mapping.connection_info

Control Plane Node — MySQL

USE nova;

UPDATE block_device_mapping
   SET connection_info = REPLACE(connection_info,
       '<OLD_FILER_FQDN>:', '<NEW_FILER_FQDN>:')
 WHERE connection_info LIKE '%<OLD_FILER_FQDN>:%'
   AND deleted = 0;

Why these updates matter:

The Cinder updates ensure all existing volumes report the new network path so subsequent volume operations (snapshot, clone, extend, detach/attach) target the new filer. The Nova block device mapping update is required for libvirt to mount each VM's volumes from the new path when VMs are powered back on in Step 6.

4

Update /etc/fstab on Each Hypervisor

Run the following sequence on every hypervisor that mounts the backend:

Hypervisor Node

$ sudo systemctl stop pf9-nova-compute libvirtd
$ mount | egrep "images|instances"

$ sudo umount /var/lib/glance/images
$ sudo umount /opt/data/instances

$ sudo sed -i 's|<OLD_FILER_FQDN>|<NEW_FILER_FQDN>|g' /etc/fstab

$ sudo mount -a
$ mount | egrep "images|instances"

Sample Output

[NEW_FILER_FQDN]:/nfs_pcd01 on /var/lib/glance/images type nfs (...)
[NEW_FILER_FQDN]:/nfs_pcd01 on /opt/data/instances type nfs (...)

The output must show only <NEW_FILER_FQDN> entries — no <OLD_FILER_FQDN> entries.

Bring services back up:

Hypervisor Node

$ sudo systemctl start libvirtd pf9-ostackhost

Confirm each hypervisor returns to a healthy state in the PCD UI before proceeding to the next hypervisor.

5

Clean Up Orphan Old Mounts

If any <OLD_FILER_FQDN> mounts are still present after Step 4, clean them up on each affected hypervisor.

First, confirm no processes have open file handles on the old filer:

Hypervisor Node

$ sudo lsof | grep <OLD_FILER_FQDN>

Sample Output

(empty — must return no results)

Unmount the orphan paths:

Hypervisor Node

$ sudo umount /opt/pf9/etc/pf9-cindervolume-base/volumes/<OLD-HASH-EXPORT-1>
$ sudo umount /opt/pf9/etc/pf9-cindervolume-base/volumes/<OLD-HASH-EXPORT-2>
$ sudo umount /opt/pf9/data/state/mnt/<OLD-HASH-EXPORT-1>
$ sudo umount /opt/pf9/data/state/mnt/<OLD-HASH-EXPORT-2>

Verify no old mounts remain:

Hypervisor Node

$ mount | grep <OLD_FILER_FQDN>

Sample Output

(empty — must return no results)

Checkpoint: Every mount on every hypervisor should now reference <NEW_FILER_FQDN>. None should reference <OLD_FILER_FQDN>.

6

Power VMs Back On

Start the VMs in batches from the PCD UI.

Spot-check a representative sample of VMs after each batch:

Controller / OpenStack CLI

$ openstack server show <VM_NAME_OR_UUID> -c status -f value

Sample Output

ACTIVE

Verify disk paths reference the new filer mount on the hypervisor:

Hypervisor Node

$ sudo virsh domblklist instance-<VM_UUID>

Sample Output

 Target   Source
---------------------------------------------------------
 vda      /opt/pf9/data/state/mnt/[NEW_HASH_EXPORT]/[volume-file]

The source path must reference the new filer mount hash — not the old one.

Monitor the cluster for 10–15 minutes after the last VM is powered on to confirm no late-failing volume operations or NFS timeouts.

Validation

After all VMs are back online, run the following end-to-end checks against the new filer:

1. Create a test volume and confirm it is created on <NEW_FILER_FQDN>.
2. Attach the test volume to a running VM
3. Snapshot the test volume.
4. Extend the test volume.
5. Boot a new test VM and confirm it comes up using the new filer path.