Stateful Control Plane Pods Fail To Start Due To NetApp Storage Volume Corruption

Problem

One or more stateful control-plane pods on a single node fail to start. airctl status reports the region as Not Ready and the Networking Service shows as unhealthy. The affected pods sit in Init:0/1 (the volume will not mount) or CrashLoopBackOff (the volume mounts but the data cannot be read), and the affected pods are concentrated on one control plane node.

Sample Output

NAMESPACE    NAME                           READY  STATUS            NODE
default      decco-consul-consul-server-2   0/1    Init:0/1          [NODE_NAME]
[REGION_NS]  ovn-ovsdb-sb-0                 0/1    CrashLoopBackOff  [NODE_NAME]
[REGION_NS]  percona-db-pxc-db-pxc-0        0/3    Init:0/1          [NODE_NAME]

Environment

• Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher
• Component:Consul, OVN OVSDB databases, Percona XtraDB Cluster
• Storage: NetApp Trident CSI over Fibre Channel, storage class pcd-sc, provisioner csi.trident.netapp.io

Cause

Stateful Private Cloud Director workloads store data on persistent volumes provisioned by NetApp Trident over Fibre Channel. Each volume is presented to the host as a LUN, assembled by Linux multipath into a device under /dev/mapper/, formatted ext4, and mounted into the pod. A pod can only start once the persistent volume mounts and the filesystem on the volume is readable.

After an extended storage degradation or Fibre Channel path-flapping event, the LUN is re-presented at a different size or the contents are damaged, and the filesystem becomes unreadable. The host cannot mount or open the volume, so the dependent pod cannot start. Because every stateful pod scheduled on one node shares the same host storage path layer, a single storage event affects several pods at once. A pod restart cannot repair data on disk, so the affected pods never progress.

Two related corruption signatures appear in the host kernel log. The first is a geometry mismatch: the ext4 superblock records a filesystem larger than the LUN now presented to the host.

Sample Output

EXT4-fs (dm-9): bad geometry: block count 1310720 exceeds size of device (262144 blocks)

A block count of 1310720 at 4K equals 5GiB (the size the filesystem expects), while 262144 blocks equals 1GiB (the actual device). The device is physically smaller than the filesystem the superblock describes. The second signature is metadata corruption: the LUN is the correct size but the directory or inode structures are damaged.

Sample Output

EXT4-fs error (device dm-11): __ext4_find_entry: inode #2: comm ovsdb-server: checksumming directory block 0
EXT4-fs error (device dm-13): htree_dirblock_to_tree: inode #2: Directory block failed checksum

The failure then cascades into the clustered services that depend on the affected volumes. When the OVN Southbound member on the corrupted node restarts onto an unreadable volume, that member can abandon the original cluster and bootstrap a new, empty, single-node cluster, while the surviving members stay in the original cluster and cannot elect a leader. The Diagnostics section shows how to capture and confirm this split-brain from the Southbound cluster status. A surviving Southbound member logs repeated elections at a higher term and is terminated by the startup watchdog before quorum can form. A Consul member on the same node drops out of the Raft cluster, leaving Consul at reduced quorum.

Diagnostics

1

List the stuck pods and the node those pods share

Control Plane Node

kubectl get pods -A -o wide | egrep -v 'Running|Completed'

Sample Output

default      decco-consul-consul-server-2   0/1  Init:0/1          [NODE_NAME]
[REGION_NS]  ovn-ovsdb-sb-0                 0/1  CrashLoopBackOff  [NODE_NAME]
[REGION_NS]  percona-db-pxc-db-pxc-0        0/3  Init:0/1          [NODE_NAME]

2

Read the mount failure reported by the kubelet

Control Plane Node

kubectl describe pod <POD_NAME> -n <NAMESPACE> | grep -A4 FailedMount

Sample Output

Warning  FailedMount  (x2288 over 3d5h)  kubelet  MountVolume.SetUp failed for volume
"[PVC_ID]": mount failed: exit status 32, output: mount: wrong fs type, bad option,
bad superblock on /dev/mapper/[DEVICE_WWID], ...

3

Confirm the corruption at the host kernel layer

Control Plane Node

dmesg -T | egrep -i 'EXT4-fs|bad geometry|failed checksum|I/O error'

Sample Output

EXT4-fs (dm-9): bad geometry: block count 1310720 exceeds size of device (262144 blocks)
EXT4-fs error (device dm-11): htree_dirblock_to_tree: inode #2: Directory block failed checksum

4

Map the failing device to the persistent volume claim

Control Plane Node

lsblk | grep -A1 mpath
df -h | grep /dev/mapper

5

Confirm the filesystem is not repairable

For a geometry mismatch, e2fsck aborts rather than risk writing past the end of the device. Run the check with -n (answers no to all prompts) so the check stays read-only. This output is the evidence for the storage team.

Control Plane Node

e2fsck -n /dev/mapper/<DEVICE_WWID>

Sample Output

The filesystem size (according to the superblock) is 1310720 blocks
The physical size of the device is 262144 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort? yes

6

Check the OVN Southbound cluster for a split-brain

Query the Southbound cluster status. A split-brain shows one member reporting a different Cluster ID at Term 1 with only that member in the server list, while the other members sit at a much higher term and cannot elect a leader.

Control Plane Node

kubectl exec -n <REGION_NS> ovn-ovsdb-sb-0 -c ovsdb -- ovs-appctl cluster/status OVN_Southbound

Sample Output

Cluster ID: [NEW_CLUSTER_ID]     <- a NEW cluster id, different from the surviving members
Role: leader
Term: 1                          <- the surviving members are on a much higher term
Servers:
    [SERVER_ID] (self)           <- only this server; a one-member cluster

7

Confirm the surviving cluster members are healthy

Confirm the surviving Consul servers and OVN members are Running, and that at least one Percona member reports Synced to act as the donor for a rebuild.

Control Plane Node

kubectl get pods -n default | grep consul-server
kubectl get pods -n <REGION_NS> | grep ovn-ovsdb
kubectl get pxc -n <REGION_NS>

If the surviving members are not healthy, stop. Regenerating a volume while the surviving members are unhealthy can cause permanent data loss. Contact Platform9 Support.

If all checks above match, proceed to the Workaround.

Resolution

The permanent fix must be applied at the storage layer, as the corruption originates entirely at the NetApp storage array and the Fibre Channel path layer beneath the Kubernetes cluster. The storage team / OS administration team must identify and correct the underlying condition that re-presented the LUN at an incorrect size or corrupted its contents (such as an unannounced resize, remap, clone, snapshot restore, or severe path loss during the disconnect window). They must also confirm that the LUN WWIDs and physical capacities presented to the host exactly match what Trident originally provisioned. Until the root storage condition is corrected, this corruption can recur. Use the Workaround below to restore service in the meantime.

Provide the storage team with the following evidence, captured during Diagnostics:

• The dmesg output showing the geometry mismatch and directory-checksum errors
• The e2fsck abort output showing the filesystem-size versus device-size mismatch
• The affected /dev/mapper/ device identifiers and the matching persistent volume claim names

Workaround

The affected workloads rebuild from a surviving copy, so regenerating the corrupted volumes restores service. A Consul member re-syncs from the cluster leader, a Percona member copies the database from a healthy member through a State Snapshot Transfer, and the OVN Southbound database is recomputed by ovn-northd from the Northbound database.

Deleting a persistent volume claim destroys the data on the volume. This workaround is safe only because Consul, Percona, and the OVN Southbound database rebuild from a surviving copy. Do not apply this workaround to any workload that holds the only copy of the data. If the surviving cluster members are not healthy (Consul below quorum, no Synced Percona donor, or the OVN Northbound database also broken), restore from backup or contact Platform9 Support instead.

1

Regenerate the Consul server volume

Recover Consul first: Because the affected Consul member is stuck in an Init:0/1 state with its volume locked by the host OS, temporarily scaling down the cluster releases this lock and allows us to safely delete the corrupted PVC. Re-scaling back to 3 replicas forces the array to provision a fresh, healthy LUN, which validates that the backend storage plane can successfully attach clean volumes before we attempt the higher-risk OVN Southbound regeneration.

Control Plane Node

kubectl scale sts decco-consul-consul-server -n default --replicas=2
kubectl delete pvc data-default-decco-consul-consul-server-<ORDINAL> -n default
kubectl scale sts decco-consul-consul-server -n default --replicas=3

NetApp Trident provisions a fresh, correctly sized LUN. The new pod joins the cluster and re-syncs state from the leader. Confirm the regenerated member returns to Running.

Control Plane Node

kubectl get pods -n default | grep consul-server

2

Regenerate the Percona member volume

When an affected Percona member is stuck in an Init:0/1 state with its volume locked or corrupted at the host layer, deleting the persistent volume claim (PVC) clears the broken hardware mapping and forces the Percona Operator to spin up a fresh, empty drive. A Percona XtraDB Cluster member safely rejoins empty by automatically copying the entire running database from a healthy, Synced donor node through a State Snapshot Transfer (SST)—provided that at least one of the other cluster survivors was verified as healthy during your initial diagnostics check.

Control Plane Node

kubectl delete pvc datadir-percona-db-pxc-db-pxc-<ORDINAL> -n <REGION_NS>

The operator recreates the member, which automatically performs a State Snapshot Transfer (SST) from a Synced donor. Confirm the cluster returns to the full member count and reports ready:

Control Plane Node

kubectl get pxc -n <REGION_NS>
kubectl get pods -n <REGION_NS> | grep percona-db-pxc-db-pxc

3

Rebuild the OVN Southbound cluster

When the Southbound database is split-brain - with one member isolated in a new single-node cluster and the surviving members unable to elect a leader - the safest recovery is to clear all three Southbound members and let the cluster re-form empty. This is entirely safe because the Southbound database holds no unique data; the ovn-northd process will completely recompute the Southbound database from the Northbound database.

Do not scale the Southbound StatefulSet to a partial replica count to keep one member. kubectl scale sts ... --replicas=1 always retains ordinal 0, which may be the corrupted or isolated member, and the following volume deletion can then destroy the volume of the only healthy member. Because the Southbound database holds no unique data, clear all three members instead. This step requires a healthy Northbound database, confirmed in Diagnostics.

Control Plane Node

# capture the StatefulSet definition first as a rollback reference
kubectl get sts ovn-ovsdb-sb -n <REGION_NS> -o yaml > ovn-ovsdb-sb-bak.yaml

# scale the split cluster to zero and clear all three member volumes
kubectl scale sts ovn-ovsdb-sb -n <REGION_NS> --replicas=0
kubectl delete pvc data-ovn-ovsdb-sb-0 data-ovn-ovsdb-sb-1 data-ovn-ovsdb-sb-2 -n <REGION_NS>

# scale back up; the members form one fresh cluster and ovn-northd repopulates the Southbound database from the Northbound database
kubectl scale sts ovn-ovsdb-sb -n <REGION_NS> --replicas=3
kubectl rollout status sts ovn-ovsdb-sb -n <REGION_NS> --timeout=300s

The rebuilt Southbound database starts empty and depends on ovn-northd to repopulate the Southbound database from the Northbound database. Confirm ovn-northd is running after the rebuild. Three Running pods serving an empty Southbound database appear healthy but leave the Networking Service broken.

4

Verify the recovered state

Control Plane Node

# one consistent Southbound cluster, one leader, three members
for p in 0 1 2; do kubectl exec -n <REGION_NS> ovn-ovsdb-sb-$p -c ovsdb -- \
  ovs-appctl cluster/status OVN_Southbound | egrep -i 'Cluster ID|Role|Leader|Servers'; done

# ovn-northd and ovn-ovsdb-relay are running, and the Southbound database is populated, not empty
kubectl get pods -n <REGION_NS> | egrep 'ovn-northd|ovn-ovsdb-relay|ovn-ovsdb-sb'
kubectl exec -n <REGION_NS> ovn-ovsdb-sb-0 -c ovsdb -- ovn-sbctl lflow-list | wc -l

# the region reports healthy
airctl status

Recovery is confirmed when the Southbound cluster reports a single leader with three members, the Southbound flow count is large rather than near zero, and airctl status reports the region as Ready. Existing virtual machines continue forwarding network traffic uninterrupted during the Southbound rebuild, because packet forwarding runs on the hypervisors.

Additional Information

Regenerating the volumes restores service but does not prevent recurrence. The storage-side condition described in Resolution must be corrected by the storage team, or the corruption can return on the same node after the next storage event.