> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pcd/generic/multiple-pcd-services-down-and-hosts-unresponsive-due-to-sealed-decco-vault.md).

# Multiple PCD Services Down and Hosts Unresponsive Due to Sealed Decco Vault

## Problem

* Hosts become unresponsive and report as offline.
* Critical Platform9 services on the host, such as `pf9-ostackhost` and `pf9-cindervolume-base` , `pf9-neutron-ovn-metadata-agententer` , `pf9-novncproxy` entered a failed state.

## Environment

* Self-Hosted Private Cloud Director Virtualization

## Cause

Following pods on the control plane were stuck in the *Initializing* phase:

* **Decco Vault** in the `default` namespace
* **Vouch Keystone** in the corresponding region namespace
* **Vouch NoAuth** in the corresponding region namespace

The **Decco Vault** was found to be in a **sealed** state. A sealed state is a security posture where the vault's data is encrypted and inaccessible.

## Diagnostics

Most of the service logs report connectivity errors.

{% code title="ostackhost logs:" %}

```bash
ERROR oslo.messaging._drivers.impl_rabbit [REQ_ID] AMQP server on 127.0.0.1:5673 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
ERROR oslo.messaging._drivers.impl_rabbit [REQ_ID] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: ConnectionRefusedError: [Errno 111] ECONNREFUSED
```

{% endcode %}

{% code title="cindervolume-base logs" overflow="wrap" %}

```bash
ERROR oslo.messaging._drivers.impl_rabbit [-] [REQ_ID] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: ConnectionRefusedError: [Errno 111] ECONNREFUSEDEADDRINUSE encountered during SNI client creation ... retrying in 20000 milliseconds/ERRORnn:qlscd ..lsvim novG/ERROR:qvim cinvG/ERRORmodel server went away: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')n
```

{% endcode %}

## Resolution

1. Cordoned the node where the **Decco Vault,Vouch Keystone,Vouch NoAuth** Pods are scheduled.

```bash
$ kubectl cordon <NODE_NAME>
```

{% hint style="info" %}
If all three pods are located on the same node, only need to cordon that single node. However, if the pods are distributed across different nodes, cordon each node and restart the pods one by one. The primary goal here is to reschedule the pods from their current node.
{% endhint %}

2. Performed a rollout restart of the following deployments in the same sequence

* **Decco Vault** in the `default` namespace
* **Vouch Keystone** in the corresponding region namespace
* **Vouch NoAuth** in the corresponding region namespace

```bash
$ kubectl rollout restart deployment <DEPLOYMENT_NAME> -n <NAMESPACE>
```

3. Restarted the pf9-hostagent service on the affected host.

```none
# systemctl restart pf9-hostagent
```

## Validation

1. The hypervisor returned to a **healthy** state
2. All Platform9 services were observed to be running normally on the affected node.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://platform9.com/kb/pcd/generic/multiple-pcd-services-down-and-hosts-unresponsive-due-to-sealed-decco-vault.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.