> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/self-hosted/airctl-restore-fails-typeerror-legacy-consul-placeholder-keys.md).

# airctl restore Fails with TypeError: Legacy Consul Placeholder Keys

## Problem

`airctl restore` fails with `TypeError: 'unicode' object does not support item assignment` inside deccaxon's `restore.py` during an upgrade or nodelet-to-K3s migration. The error occurs when the Consul backup archive contains legacy empty-value placeholder keys at parent namespace paths that also have real child keys under the same path. The DU services never come up even though the management cluster itself is healthy.

## Environment

* Self-Hosted Private Cloud Director: All versions — affects DUs originally deployed with an older version of deccaxon.
* Typical trigger: upgrading from v-2026.1.0 (January) or earlier to a newer airctl version, or performing a nodelet → K3s cluster migration.

## Procedure

> **Important:** Run the pre-flight check (Step 1) before starting `airctl restore`. If conflicts are detected, delete them as soon as Consul pods reach `Running/Ready` — do not wait for deccaxon to fail.

### Step 1 — Pre-flight check: detect conflicting keys in the backup archive

Extract the backup and run the following script to identify conflicting keys:

```bash
tar -xzf <backup>.tar.gz

python3 -c "
import json, glob
for fname in glob.glob('consul_*.json'):
    with open(fname) as f:
        data = json.load(f)
    keys = [d['Key'] for d in data]
    conflicts = [
        k for k in keys
        if any(k2.startswith(k + '/') for k2 in keys if k2 != k)
        and not k.endswith('server_key')
    ]
    if conflicts:
        print(fname, '- conflicts:', conflicts)
"
```

Sample output when the issue is present:

```
consul_infra.json - conflicts: ['customers/a1b2c3d4-.../regions/infra/rabbit_broker/users', 'customers/a1b2c3d4-.../regions/infra/roles']
consul_tcx.json   - conflicts: ['customers/a1b2c3d4-.../regions/tcx/rabbit_broker/users', 'customers/a1b2c3d4-.../regions/tcx/roles']
```

If the output is empty, no action is needed — proceed with `airctl restore` normally.

### Step 2 — Get the Consul ACL token

```bash
CONSUL_TOKEN=$(cat .airctl/state.yaml | grep -i consultoken | awk '{print $2}')
```

### Step 3 — Delete conflicting placeholder keys from live Consul

Wait until Consul pods are `Running/Ready` during restore. Using the exact paths from Step 1, delete each conflicting key. Repeat for every region:

```bash
kubectl exec -n default decco-consul-consul-server-0 -c consul -- \
  sh -c "CONSUL_HTTP_TOKEN=$CONSUL_TOKEN consul kv delete \
  'customers/<uuid>/regions/<region>/rabbit_broker/users'"

kubectl exec -n default decco-consul-consul-server-0 -c consul -- \
  sh -c "CONSUL_HTTP_TOKEN=$CONSUL_TOKEN consul kv delete \
  'customers/<uuid>/regions/<region>/roles'"
```

### Step 4 — Verify deletion

```bash
kubectl exec -n default decco-consul-consul-server-0 -c consul -- \
  sh -c "CONSUL_HTTP_TOKEN=$CONSUL_TOKEN consul kv get \
  'customers/<uuid>/regions/<region>/rabbit_broker/users'"
# Expected: Error! No key exists at that path
```

Once all conflicting keys are removed, deccaxon will proceed past the TypeError and complete the restore successfully.

## Additional Information

**Root cause:** Long-lived DUs carry stale empty-value placeholder keys in Consul written by an older version of deccaxon. These keys exist at parent namespace paths (`rabbit_broker/users`, `roles`) while real child keys also exist under the same path. deccaxon's `restore.py` assigns the empty string value first, then tries to nest child keys under it — Python raises TypeError because string objects do not support item assignment.

**Do not delete `server_key`:** The key `customers/<uuid>/regions/<region>/services/vouch/vault/server_key` holds a real value required by the vouch service. If accidentally deleted, restore it with:

```bash
kubectl exec -n default decco-consul-consul-server-0 -c consul -- \
  sh -c "CONSUL_HTTP_TOKEN=$CONSUL_TOKEN consul kv put \
  'customers/<uuid>/regions/<region>/services/vouch/vault/server_key' \
  'customers/<uuid>/vault_servers/dev'"
```

**Permanent fix:** [PCD-8806](https://platform9.atlassian.net/browse/PCD-8806) tracks a code fix to `restore.py` and backup sanitization. Until this ships, apply the manual workaround above.

**Reference incident:** [ECS-407](https://platform9.atlassian.net/browse/ECS-407)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pcd/self-hosted/airctl-restore-fails-typeerror-legacy-consul-placeholder-keys.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.