etcd Secrets Encryption

As of Platform9 Managed Kubernetes v5.2, Kubernetes clusters running version 1.20 and later can be configured to encrypt secrets that are stored in etcd.

To setup a cluster to use Secrets Encryption, the encryption provider config YAML needs to be created on each master node prior to creating the cluster and must reside under /var/opt/pf9/kube/apiserver-config directory and the same absolute path must be provided when creating the cluster.

• "apiServerFlags": "--encryption-provider-config=/var/opt/pf9/kube/apiserver-config/encryption-provider.yaml"

To enable secrets encryption add the following custom API Server Flag:

"apiServerFlags": "--encryption-provider-config=/var/opt/pf9/kube/apiserver-config/encryption-provider.yaml"

Cluster creation

To create a cluster with encryption each Kubernetes master node must contain the encryption provider configuration prior to creating the cluster.

To create the file and directory run the commands below.

mkdir -p /var/opt/pf9/kube/apiserver-config # This directory may be missing in case the node has not been preped using pf9ctl

touch /var/opt/pf9/kube/apiserver-config/encryption-provider.yaml

chown pf9:pf9group /var/opt/pf9/kube/apiserver-config/encryption-provider.yaml

Refer to Encrypting Secret Data at Rest for contents of this file. Replace the key name and secret with your valid configuration.

A sample config:

apiVersion: apiserver.config.k8s.io/v1

kind: EncryptionConfiguration

resources:

  - resources:

    - secrets

    providers:

    - aescbc:

        keys:

        - name: key1

          secret: ZnfoLrhTJFRq3pl4cPrlboH0lsHS33A3axWDl1HKWj8=

    - identity: {}

Create a cluster with secret encryptions using the Qbert API

To create a cluster with secrets encryption using the Qbert API an additional field is required in the clusters POST command. The field "apiServerFlags"needs to be provided with the location of the encryption provider yaml.

URL: qbert/v4/<project ID>/clusters

Method: POST

Headers: X-Auth-Token, Content-Type

Body: 

{

  "name": <>,

  "allowWorkloadsOnMaster": true,

  "containersCidr": "10.20.0.0/22",

  "servicesCidr": "10.21.0.0/22",

  "mtuSize": 1440,

  "privileged": true,

  "appCatalogEnabled": false,

  "nodePoolUuid": <>,

  "kubeRoleVersion": "1.20.5-pmk.1774",

  "calicoIpIpMode": "Always",

  "calicoNatOutgoing": true,

  "calicoV4BlockSize": "26",

  "calicoIPv4DetectionMethod": "first-found",

  "networkPlugin": "calico",

  "runtimeConfig": "",

  "etcdBackup": {

    "storageType": "local",

    "isEtcdBackupEnabled": 1,

    "storageProperties": {

      "localPath": "/etc/pf9/etcd-backup"

    },

    "intervalInMins": 1440

  },

  "apiServerFlags": "--encryption-provider-config=/var/opt/pf9/kube/apiserver-config/encryption-provider.yaml"

}

Response:

200 OK

{

    "uuid": "885ba8fb-50cf-48a7-bb38-d973cece8abf"

}

Attach the master nodes using Qbert API

If you are using the API to create the cluster you will need to attach each master node using the Qbert API.

uuid: The UUID can be found on the Infrastructure dashboard Nodes tab by adding UUID to the table.

URL: /qbert/v3/<project ID>/clusters/<cluster UUID from step 2>/attach

Method: POST

Headers: X-Auth-Token, Content-Type

Body: 

[{

  "uuid": "d454dac1-7e1b-417d-af2f-367efc91cf89",

  "isMaster": true

}]

Response:

200 OK

OK

Rotating a decryption key

To rotate the keys currently in use the encryption-provider.yaml file needs to be updated with the new key on each master node and the Kubernetes services must be restarted.

To update the encryption key a new key must first be generated in your Key Management System (KMS) and then added to encryption-provider.yamlas the second key entry for the current provider on all master nodes.

Once the encryption-provider.yaml has been updated on each master node stop the Platform9 services on each master node. To avoid a cluster outage progress through each node one by one ensuring that the master node converges and returns into the cluster before moving to the next.

systemctl stop pf9-nodeletd pf9-hostagent

Once each node has been restarted the Kubernetes API Server must be restarted to ensure that all nodes can decrypt using the new key. This may cause an outage if all nodes are restarted simultaneously,

Platform9 starts the apiserver as a container, to stop the kube-apiserver you will need to restart the entire stack. (Work is in inprogress to enable light-touch restarts.)

/opt/pf9/nodelet/nodeletd phases stop

​

/opt/pf9/nodelet/nodeletd phases start

Once the Kubernetes API server has started edit encryption-provider.yaml to promote the new key to be the first entry in the keys array. Promoting the new key to the first position in the array ensures that it is used for encryption.

Once the new key is promoted the Kubernetes API Server will need to be restarted for a second time.

Restart all kube-apiserver processes to ensure each server now encrypts using the new key.

/opt/pf9/nodelet/nodeletd phases stop

​

/opt/pf9/nodelet/nodeletd phases start

Once the Kubernetes API Server has restarted runkubectl get secrets --all-namespaces -o json | kubectl replace -f - to encrypt all existing secrets with the new key.

To complete the process backup etcd and then remove the old decryption key from the config.

Decrypting all data

To disable encryption at rest move the identity provider entry to be read as the first entry in the config:

apiVersion: apiserver.config.k8s.io/v1

kind: EncryptionConfiguration

resources:

  - resources:

    - secrets

    providers:

    - identity: {}

    - aescbc:

        keys:

        - name: key1

          secret: <BASE 64 ENCODED SECRET>

Once the encryption-provider.yaml has been updated restart all kube-apiserver processes.

Platform9 starts the apiserver as a container, to stop the kube-apiserver we need to restart the entire stack.

To force all the secrets to be decrypted run the command below.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -