# Vouch-Noauth And Vouch-Keystone Pods Are Not Ready Due To Token Expiry

## Problem

The `Vouch-Noauth` and `Vouch-Keystone` pods are not in a ready state in both Infra and Workload regions. This situation is preventing the environments from being fully operational and has resulted in the upgrade being stalled.

## Environment

* Self-Hosted Private Cloud Director Virtualization - **v2025.2 to v2025.6**

## Cause

* Vouch token stored in consul has expired, and it weren't renewed automatically by the `vouch-renew-token` cronjob.
* The issue has been reported as a bug, and the Platform Engineering team tracked it under the ID **PCD-1468** and the fix has been released in [July\[v2025.7-47\] and above release.](https://docs.platform9.com/release-notes/july-2025-release)

## Diagnostics

1. `vouch-keystone` and `vouch-noauth` pods become not ready.

{% code title="Control Plane Node" %}

```bash
$ kubectl get pods --all-namespaces | grep vouch                                                                                  
[INFRA_NS]        vouch-keystone-POD     1/2      Running             0  3h
[INFRA_NS]        vouch-noauth-POD       2/3      Running             0  3h

[WORKLOAD_NS]     vouch-keystone-POD     1/2      Running             0  3h
[WORKLOAD_NS]     vouch-noauth-POD       2/3      Running             0  3h
```

{% endcode %}

### **Method 1: cURL Test**

{% stepper %}
{% step %}
Exec into the `vouch-keystone` container of `vouch-keystone` pod and get the vault token from `vouch-keystone.conf`

{% code title="Control Plane Node" %}

```bash
$ kubectl exec -it -n <AFFECTED_NS> <VOUCH_KEYSTONE_POD>  -- bash
[vouch-keystone-POD>]$ grep vault_token /etc/vouch/vouch-keystone.conf | awk '{ print $2 }'
```

{% endcode %}
{% endstep %}

{% step %}
Run the cURL command after replacing the actual token from above output

{% code title="Vouch-Keystone" %}

```bash
$ curl --header "X-Vault-Token: <TOKEN>" "http://decco-vault-active.default.svc.cluster.local:8200/v1/auth/token/lookup-self" -v
```

{% endcode %}

<pre class="language-bash" data-title="Sample Output"><code class="lang-bash">$ curl --header "X-Vault-Token: &#x3C;TOKEN>" "http://decco-vault-active.default.svc.cluster.local:8200/v1/auth/token/lookup-self" -v

Host decco-vault-active.default.svc.cluster.local:8200 was resolved.
...
...

{"errors":["permission denied"]}

<strong>* Connection #0 to host decco-vault-active.default.svc.cluster.local left intact
</strong></code></pre>

{% endstep %}

{% step %}
If the token has expired, the output will indicate "*Permission denied.*" as shown above.
{% endstep %}
{% endstepper %}

### **Method 2:** Verify Vault token using Consul

{% stepper %}
{% step %}
Read the token from Consul (use when the vouch-keystone container is in CrashLoopBackOff). This step does not require exec into the failing vouch pod.

Get the Consul ACL token from the `airctl` state file on the control plane node:

{% code title="Control Plane Node" %}

```bash
$ grep consulToken ${HOME}/.airctl/state.yaml | cut -d' ' -f2
```

{% endcode %}
{% endstep %}

{% step %}
Open a shell inside the Consul server pod:

{% code title="Control Plane Node" %}

```bash
$ kubectl exec -it -n default decco-consul-consul-server-0 -- sh
```

{% endcode %}
{% endstep %}

{% step %}
Inside the Consul shell, export the ACL token (replace `<CONSUL_TOKEN>`) and read the `host_signing_token`:

{% code title="Consul Server Pod" %}

```bash
$ export CONSUL_HTTP_TOKEN="<CONSUL_TOKEN>"
$ consul kv get customers/<CUSTOMER_ID>/regions/<REGION_UUID>/services/vouch/vault/host_signing_token
```

{% endcode %}

The output is a string starting with `hvs.`. Copy it, then type `exit` to leave the Consul pod.
{% endstep %}

{% step %}
Open a shell inside the Vault pod (replace `<VAULT_POD>` with the name from `kubectl get pods -n default | grep vault`):

{% code title="Control Plane Node" %}

```bash
$ kubectl exec -it -n default <VAULT_POD> -- sh
```

{% endcode %}
{% endstep %}

{% step %}
Inside the Vault shell, set the address and token (replace `<HOST_SIGNING_TOKEN>`) and run the lookup:

{% code title="Vault Pod" %}

```bash
$ export VAULT_ADDR=http://localhost:8200
$ export VAULT_TOKEN="<HOST_SIGNING_TOKEN>"
$ vault token lookup
```

{% endcode %}

If the token has expired, you will see:

```
Code: 403. Errors:
* permission denied
```

{% endstep %}
{% endstepper %}

## Resolution

* Upgrade to Self-hosted Private Cloud Director [\[v2025.7-47\]](https://docs.platform9.com/release-notes/july-2025-release) **or above version.**

## Workaround

* Manually renew the expired token so that vouch pods can communicate with consul.

**Steps:**

1. Get the `CONSUL_HTTP_TOKEN` from Airctl host \[The host with airctl state file is present.]

{% code title="Control Plane Node" %}

```bash
$ grep consulToken ${HOME}/.airctl/state.yaml | cut -d' ' -f2
```

{% endcode %}

2. Exec into `decco-consul-consul-server` pod in the *default* namespace

{% code title="Control Plane Node" %}

```bash
$ kubectl exec -it decco-consul-consul-server-0 -- sh -n default
```

{% endcode %}

3. Export the COSUL\_HTTP\_TOKEN from step 1 in `decco-consul-consul-server` pod

{% code title="Consul Pod" %}

```bash
[decco-consul-consul-server-0]$ export CONSUL_HTTP_TOKEN="<TOKEN>"
```

{% endcode %}

The following commands generate several outputs that correspond to the total number of regions present in the environment.

4. Retrieve region UUIDs.

{% code title="Consul Pod" %}

```bash
[decco-consul-consul-server-0]$ consul kv get -recurse | grep <REGION_UUID>
```

{% endcode %}

* The `<REGION_UUID>` serves a crucial role in distinguishing between multiple regions. This unique identifier ensures that each region can be clearly identified and managed effectively within your environment.

```bash
region_fqdns/example-infra.platform9.localnet/region_uuid:<REGION_UUID>
region_fqdns/example-workload.platform9.localnet/region_uuid:<REGION_UUID>
```

5. Retrieve existing tokens

{% code title="Consul Pod" %}

```bash
[decco-consul-consul-server-0]$ consul kv get -recurse | grep host_signing_token
```

{% endcode %}

```bash
customers/<CUSTOMER_ID>/regions/REGION/UUID/services/vouch/vault/host_signing_token:hvs.<TOKEN>
```

6. Delete the existing Token for the specified affected region(s).

{% code title="Consul Pod" %}

```bash
[decco-consul-consul-server-0]$ consul kv delete customers/<CUSTOMER_ID>/regions/REGION_UUID/services/vouch/vault/host_signing_token
```

{% endcode %}

{% code title="Sample Output" %}

```bash
Success! Deleted key: customers/[CUSTOMER_ID]/regions/REGION_UUID/services/vouch/vault/host_signing_token
```

{% endcode %}

Exit from the `decco-consul-consul-server` pod

7. Manually run the `vouch-renew-token`Job

Repeat this step for all affected regions by changing the \<AFFECTED\_NS>

{% code title="Control Plane Node" %}

```bash
$ kubectl create job --from=cronjob/vouch-renew-token vouch-renew-token-manual -n <AFFECTED_NS>
```

{% endcode %}

{% hint style="warning" %}
If the above job fails and the logs indicate Vault rejecting a policy-creation request, this suggests that this may require renewal of admin tokens, then refer [Vouch-Renew-Token job Fails with Vault Permission Denied – Renewing Expired Vault admin\_token](/kb/pcd/self-hosted/vouch-renew-token-job-fails-with-vault-permission-denied-renewing-expired-vault-admin_token.md)
{% endhint %}

8. Check if the`Vouch-keystone and Vouch-noauth` back healthy

{% code title="Control Plane Node" %}

```bash
$ kubectl get pods --all-namespaces | grep vouch
```

{% endcode %}

* If these steps prove insufficient to resolve the issue, reach out to the [Platform9 Support Team](https://support.platform9.com/hc/en-us) for additional assistance.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://platform9.com/kb/pcd/self-hosted/vouch-noauth-and-vouch-keystone-pods-are-not-ready-due-to-token-expiry.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.