> For the complete documentation index, see [llms.txt](https://platform9.com/kb/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://platform9.com/kb/pmk/solutions/cluster-op-fail-etcdserver-mvcc-database-space-exceed.md).

# Cluster Operations Fail With Error - etcdserver: mvcc: database space exceeded

## Problem

We have noticed below logs while performing any operation on the cluster.

{% tabs %}
{% tab title="None" %}

```none
etcdserver: mvcc: database space exceeded
```

{% endtab %}
{% endtabs %}

Requests are failing with *no space* warning:

{% tabs %}
{% tab title="Logs" %}

```java
etcdserver: failed to apply request,took 2.429[C2][B5]s,request header:[ID:1920634987875929770 ] txn:[compare:<target:MOD key:"/registry/services/endpoints/kube-system/kube-controller-manager" mod_revision:287319046 ] success:[request_put:<key:"/registry/services/endpoints/kube-system/kube-controller-manager" value_size:473 ]> failure:[]>,resp ,err is etcdserver: no space
```

{% endtab %}
{% endtabs %}

## Environment

* Platform9 Managed Kubernetes - All Versions
* ETCD

## Cause

The ETCD database size on master node reached 2.1 GB so ETCD stopped serving new read/write requests due to database size saturation.

## Resolution

Follow the below steps to increase the ETCD space to accommodate new write requests. Make sure you have the ETCD backup before proceeding with the steps.

{% hint style="warning" %}
**For Multi-Master Setup**

Scale down the master nodes to a single master and perform the steps mentioned below.

If scale down of the master nodes is not possible, stop the PMK stack and update the etcd DB size and disable alarm, on all master nodes and later start the PMK stack on all the master nodes.

Once the PMK stack is up, perform the *Step 5* and *Step 6* in rolling fashion starting with the non-leader master nodes and at last on the active master node. It is important to note that the defrag action is blocking. ***The member will not respond until the defrag is complete***. For this reason, defrag should be a rolling action.
{% endhint %}

**Note:** In this example, the ETCD database space to increased to 6 GB.

1. Stop the PMK stack.

{% tabs %}
{% tab title="Command" %}

```none
systemctl stop pf9-hostagent pf9-nodeletd

/opt/pf9/nodelet/nodeletd phases stop
```

{% endtab %}
{% endtabs %}

2. For **PMK v5.1** and **K8s v1.20** and **above**, append the `ETCD_QUOTA BACKEND_BYTES` and `ETCD_SNAPSHOT_COUNT` values to */etc/pf9/kube\_override.env* file.

```bash
echo "export ETCD_SNAPSHOT_COUNT=10000" >> /etc/pf9/kube_override.env
echo "export ETCD_QUOTA_BACKEND_BYTES=6442450944" >> /etc/pf9/kube_override.env
```

**Note**: The variable `ETCD_QUOTA_BACKEND_BYTES` is used to increase disk space and `ETCD_SNAPSHOT_COUNT` to reduce the snapshot count in memory.

{% hint style="info" %}
**Note**

For **PMK v5.1** and **K8s v1.19** and **below**, modify `/opt/pf9/pf9-kube/master_utils.sh file’s ensure_etcd_running()` function by adding environment variable `"-e ETCD_QUOTA_BACKEND_BYTES=6442450944"`and `"-e ETCD_SNAPSHOT_COUNT=10000"`

These changes won't persist post cluster upgrade.
{% endhint %}

3. Start the PMK stack by running command:

{% tabs %}
{% tab title="Command" %}

```bash
/opt/pf9/nodelet/nodeletd phases start
```

{% endtab %}
{% endtabs %}

4. Once the PMK stack is up, start the *pf9-hostagent* service.

{% tabs %}
{% tab title="Command" %}

```none
# systemctl start pf9-hostagent
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Info**

Once ETCD container is up, the new variable values can be confirmed using: `# docker inspect etcd | egrep -i "snapshot|quota"`
{% endhint %}

5. Run the compaction on ETCD database by logging into the ETCD container.

{% tabs %}
{% tab title="Docker" %}

```python
docker exec -it etcd /bin/sh
# rev=$(ETCDCTL_API=3 etcdctl --endpoints=:2379 endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')

# ETCDCTL_API=3 etcdctl compact $rev
compacted revision 163674
{% endtab %}
{% tab language="bash" title="Containerd" %}
# rev=$(/opt/pf9/pf9-kube/bin/etcdctl  --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key endpoint status --write-out="json" |  egrep -o '"revision":[0-9]*' |  egrep -o '[0-9].*')

# /opt/pf9/pf9-kube/bin/etcdctl  --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key compact $rev
```

{% endtab %}
{% endtabs %}

{% hint style="warning" %}
**Warning**

Ensure the ETCD storage is **using SSD** that provides lower write latency, instead of HDD.
{% endhint %}

6. Perform defragmentation on ETCD to reclaim space.

{% tabs %}
{% tab title="Docker" %}

```none
# ETCDCTL_API=3 etcdctl defrag
{% endtab %}
{% tab language="bash" title="Containerd" %}
/opt/pf9/pf9-kube/bin/etcdctl  --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key defrag
```

{% endtab %}
{% endtabs %}

7. Disable the space exceeded alarms for the etcd container.

{% tabs %}
{% tab title="Docker" %}

```none
# ETCDCTL_API=3 etcdctl alarm disarm
{% endtab %}
{% tab language="bash" title="Containerd" %}
/opt/pf9/pf9-kube/bin/etcdctl  --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key alarm disarm
```

{% endtab %}
{% endtabs %}

## Additional Information

If post compaction and defragmentation the ETCD DB size is getting filled within days, enabling `ETCD_AUTO_COMPACTION` is a better solution. Reference: [etcd documentation](https://etcd.io/docs/v3.4/op-guide/maintenance/)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://platform9.com/kb/pmk/solutions/cluster-op-fail-etcdserver-mvcc-database-space-exceed.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.