airctl restore Fails with TypeError: Legacy Consul Placeholder Keys

Problem

airctl restore fails with TypeError: 'unicode' object does not support item assignment inside deccaxon's restore.py during an upgrade or nodelet-to-K3s migration. The error occurs when the Consul backup archive contains legacy empty-value placeholder keys at parent namespace paths that also have real child keys under the same path. The DU services never come up even though the management cluster itself is healthy.

Environment

• Self-Hosted Private Cloud Director: All versions — affects DUs originally deployed with an older version of deccaxon.
• Typical trigger: upgrading from v-2026.1.0 (January) or earlier to a newer airctl version, or performing a nodelet → K3s cluster migration.

Procedure

Important: Run the pre-flight check (Step 1) before starting airctl restore. If conflicts are detected, delete them as soon as Consul pods reach Running/Ready — do not wait for deccaxon to fail.

Step 1 — Pre-flight check: detect conflicting keys in the backup archive

Extract the backup and run the following script to identify conflicting keys:

tar -xzf <backup>.tar.gz

python3 -c "
import json, glob
for fname in glob.glob('consul_*.json'):
    with open(fname) as f:
        data = json.load(f)
    keys = [d['Key'] for d in data]
    conflicts = [
        k for k in keys
        if any(k2.startswith(k + '/') for k2 in keys if k2 != k)
        and not k.endswith('server_key')
    ]
    if conflicts:
        print(fname, '- conflicts:', conflicts)
"

Sample output when the issue is present:

consul_infra.json - conflicts: ['customers/a1b2c3d4-.../regions/infra/rabbit_broker/users', 'customers/a1b2c3d4-.../regions/infra/roles']
consul_tcx.json   - conflicts: ['customers/a1b2c3d4-.../regions/tcx/rabbit_broker/users', 'customers/a1b2c3d4-.../regions/tcx/roles']

If the output is empty, no action is needed — proceed with airctl restore normally.

Step 2 — Get the Consul ACL token

CONSUL_TOKEN=$(cat .airctl/state.yaml | grep -i consultoken | awk '{print $2}')

Step 3 — Delete conflicting placeholder keys from live Consul

Wait until Consul pods are Running/Ready during restore. Using the exact paths from Step 1, delete each conflicting key. Repeat for every region:

kubectl exec -n default decco-consul-consul-server-0 -c consul -- \
  sh -c "CONSUL_HTTP_TOKEN=$CONSUL_TOKEN consul kv delete \
  'customers/<uuid>/regions/<region>/rabbit_broker/users'"

kubectl exec -n default decco-consul-consul-server-0 -c consul -- \
  sh -c "CONSUL_HTTP_TOKEN=$CONSUL_TOKEN consul kv delete \
  'customers/<uuid>/regions/<region>/roles'"

Step 4 — Verify deletion

kubectl exec -n default decco-consul-consul-server-0 -c consul -- \
  sh -c "CONSUL_HTTP_TOKEN=$CONSUL_TOKEN consul kv get \
  'customers/<uuid>/regions/<region>/rabbit_broker/users'"
# Expected: Error! No key exists at that path

Once all conflicting keys are removed, deccaxon will proceed past the TypeError and complete the restore successfully.

Additional Information

Root cause: Long-lived DUs carry stale empty-value placeholder keys in Consul written by an older version of deccaxon. These keys exist at parent namespace paths (rabbit_broker/users, roles) while real child keys also exist under the same path. deccaxon's restore.py assigns the empty string value first, then tries to nest child keys under it — Python raises TypeError because string objects do not support item assignment.

Do not delete server_key: The key customers/<uuid>/regions/<region>/services/vouch/vault/server_key holds a real value required by the vouch service. If accidentally deleted, restore it with:

kubectl exec -n default decco-consul-consul-server-0 -c consul -- \
  sh -c "CONSUL_HTTP_TOKEN=$CONSUL_TOKEN consul kv put \
  'customers/<uuid>/regions/<region>/services/vouch/vault/server_key' \
  'customers/<uuid>/vault_servers/dev'"

Permanent fix: PCD-8806 tracks a code fix to restore.py and backup sanitization. Until this ships, apply the manual workaround above.

Reference incident: ECS-407