Vouch-Noauth And Vouch-Keystone Pods Are Not Ready Due To Token Expiry
Problem
The Vouch-Noauth and Vouch-Keystone pods are not in a ready state in both Infra and Workload regions. This situation is preventing the environments from being fully operational and has resulted in the upgrade being stalled.
Environment
Self-Hosted Private Cloud Director Virtualization - v2025.2 to v2025.6
Cause
Vouch token stored in consul has expired, and it weren't renewed automatically by the
vouch-renew-tokencronjob.The issue has been reported as a bug, and the Platform Engineering team tracked it under the ID PCD-1468 and the fix has been released in July[v2025.7-47] and above release.
Diagnostics
vouch-keystoneandvouch-noauthpods become not ready.
$ kubectl get pods --all-namespaces | grep vouch
[INFRA_NS] vouch-keystone-POD 1/2 Running 0 3h
[INFRA_NS] vouch-noauth-POD 2/3 Running 0 3h
[WORKLOAD_NS] vouch-keystone-POD 1/2 Running 0 3h
[WORKLOAD_NS] vouch-noauth-POD 2/3 Running 0 3hMethod 1: cURL Test
Exec into the vouch-keystone container of vouch-keystone pod and get the vault token from vouch-keystone.conf
Run the cURL command after replacing the actual token from above output
If the token has expired, the output will indicate "Permission denied." as shown above.
Method 2: Verify Vault token using Consul
Read the token from Consul (use when the vouch-keystone container is in CrashLoopBackOff). This step does not require exec into the failing vouch pod.
Get the Consul ACL token from the airctl state file on the control plane node:
Open a shell inside the Consul server pod:
Inside the Consul shell, export the ACL token (replace <CONSUL_TOKEN>) and read the host_signing_token:
The output is a string starting with hvs.. Copy it, then type exit to leave the Consul pod.
Open a shell inside the Vault pod (replace <VAULT_POD> with the name from kubectl get pods -n default | grep vault):
Inside the Vault shell, set the address and token (replace <HOST_SIGNING_TOKEN>) and run the lookup:
If the token has expired, you will see:
Resolution
Upgrade to Self-hosted Private Cloud Director [v2025.7-47] or above version.
Workaround
Manually renew the expired token so that vouch pods can communicate with consul.
Steps:
Get the
CONSUL_HTTP_TOKENfrom Airctl host [The host with airctl state file is present.]
Exec into
decco-consul-consul-serverpod in the default namespace
Export the COSUL_HTTP_TOKEN from step 1 in
decco-consul-consul-serverpod
The following commands generate several outputs that correspond to the total number of regions present in the environment.
Retrieve region UUIDs.
The
<REGION_UUID>serves a crucial role in distinguishing between multiple regions. This unique identifier ensures that each region can be clearly identified and managed effectively within your environment.
Retrieve existing tokens
Delete the existing Token for the specified affected region(s).
Exit from the decco-consul-consul-server pod
Manually run the
vouch-renew-tokenJob
Repeat this step for all affected regions by changing the <AFFECTED_NS>
Check if the
Vouch-keystone and Vouch-noauthback healthy
If these steps prove insufficient to resolve the issue, reach out to the Platform9 Support Team for additional assistance.
Last updated
