Rook Ceph CSI

Rook is an open source cloud-native storage orchestrator, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments.

Rook turns storage software into self-managing, self-scaling, and self-healing storage services. It does this by automating deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the facilities provided by the underlying cloud-native container management, scheduling and orchestration platform to perform its duties.

Rook integrates deeply into cloud native environments leveraging extension points and providing a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, and user experience.

The full Rook documentation can be found here: https://rook.io/docs/rook/v1.5/ceph-quickstart.html the Rook project status can be found here: project status

Minimum Requirements

For Rook with Ceph as the storage provider to install and operate each Kubernetes node must meet the following requirements:

Cluster Access & Github

To install Rook your client machine needs to be able to clone the Rook GitHub repository and have Kubectl installed with connectivity to the cluster.

Nodes

For a production deployment it is recommended to have at least 3 Kubernetes worker nodes within the cluster.

Storage Devices

One of the following storage options must be available:

Raw devices (no partitions or formatted filesystems)
Raw partitions (no formatted filesystem)
PVs available from a storage class in block mode

OS LVM package

Rook Ceph OSDs have a dependency on LVM OS package in the following scenarios:

OSDs are created on raw devices or partitions
If encryption is enabled (encryptedDevice: true in the cluster CR)
A metadata device is specified

To install LVM run the OS specific command below:

Ubuntu
CentOS
    
 
sudo apt-get install -y lvm2
Copy

Minimal Installation

Rook can be installed on a single node cluster or a single master and single worker cluster to provide storage for application deployments. Please note this configuration is not recommended for production workloads. Using the "Rook Ceph Test" configuration a small cluster can be configured with minimal physical storage and less that 3 nodes, follow the steps below to configure Rook for a minimal installation.

Requirements:

Each node in the cluster requires a mounted unformatted volume, and LVM must be installed.
Allow workloads on Masters: Enabled
CNI: Calico

Step 1 - Install LVM

To avoid any issues in setting up Ceph on raw devices install LVM by running the command below

Ubuntu
CentOS
    
 
sudo apt-get install -y lvm2
Copy

Step 2 - Clone Rook Github Repo

On your client machine clone the Rook Github repository into an empty directory using the command below.

Bash
    
 
mkdir rook-single-nodecd rook-single-nodegit clone --single-branch --branch release-1.5 https://github.com/rook/rook.git
Copy

This guide is using Rook release 1.5, you can find the latest release here: https://github.com/rook/rook/tree/master

Git Clone Output

Bash
    
 
Cloning into 'rook'...remote: Enumerating objects: 211, done.remote: Counting objects: 100% (211/211), done.remote: Compressing objects: 100% (146/146), done.remote: Total 58570 (delta 97), reused 114 (delta 60), pack-reused 58359Receiving objects: 100% (58570/58570), 36.20 MiB | 850.00 KiB/s, done.Resolving deltas: 100% (40766/40766), done.
Copy

Step 3 - Install Rook

Installing Rook is a two step process, first the CRDs and Operator are installed and then the Rook Ceph Cluster is created.

To install the CRDs and Operator run the command below:

Bash
    
 
cd rook/cluster/examples/kubernetes/cephkubectl create -f crds.yaml -f common.yaml -f operator.yaml
Copy

CRD and Operator Example Output.

Bash
    
 
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/volumes.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io createdcustomresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io creatednamespace/rook-ceph createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket createdserviceaccount/rook-ceph-admission-controller createdclusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding createdclusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt createdrole.rbac.authorization.k8s.io/rook-ceph-system createdclusterrole.rbac.authorization.k8s.io/rook-ceph-global createdclusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster createdclusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket createdserviceaccount/rook-ceph-system createdrolebinding.rbac.authorization.k8s.io/rook-ceph-system createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global createdserviceaccount/rook-ceph-osd createdserviceaccount/rook-ceph-mgr createdserviceaccount/rook-ceph-cmd-reporter createdrole.rbac.authorization.k8s.io/rook-ceph-osd createdclusterrole.rbac.authorization.k8s.io/rook-ceph-osd createdclusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system createdrole.rbac.authorization.k8s.io/rook-ceph-mgr createdrole.rbac.authorization.k8s.io/rook-ceph-cmd-reporter createdrolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt createdrolebinding.rbac.authorization.k8s.io/rook-ceph-osd createdrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr createdrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd createdrolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter createdpodsecuritypolicy.policy/00-rook-privileged createdclusterrole.rbac.authorization.k8s.io/psp:rook createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp createdrolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp createdrolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp createdrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp createdrolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp createdserviceaccount/rook-csi-cephfs-plugin-sa createdserviceaccount/rook-csi-cephfs-provisioner-sa createdrole.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg createdrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg createdclusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin createdclusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner createdclusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp createdclusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp createdclusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin createdclusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role createdserviceaccount/rook-csi-rbd-plugin-sa createdserviceaccount/rook-csi-rbd-provisioner-sa createdrole.rbac.authorization.k8s.io/rbd-external-provisioner-cfg createdrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg createdclusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin createdclusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner createdclusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp createdclusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp createdclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin createdclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role createdconfigmap/rook-ceph-operator-config createddeployment.apps/rook-ceph-operator created
Copy

Once the Operator and CRDs have been installed the cluster can be created by running the command below:

Bash
    
 
kubectl create -f cluster-test.yaml
Copy

Example Output

Bash
    
 
configmap/rook-config-override createdcephcluster.ceph.rook.io/my-cluster created
Copy

Step 4 - Cluster Status

To validate the installation run the command below;

Bash
    
 
kubectl -n rook-ceph get pod
Copy

The following output will be displayed:

Bash
    
 
NAME                                            READY   STATUS      RESTARTS   AGEcsi-cephfsplugin-provisioner-7cbcfdc5b9-58f82   6/6     Running     0          10mcsi-cephfsplugin-provisioner-7cbcfdc5b9-fpxqm   0/6     Pending     0          10mcsi-cephfsplugin-qqwx5                          3/3     Running     0          10mcsi-rbdplugin-277ft                             3/3     Running     0          10mcsi-rbdplugin-provisioner-7675f97656-4864t      6/6     Running     0          10mrook-ceph-mgr-a-5fc6c648b9-swxv7                1/1     Running     0          3m39srook-ceph-mon-a-54fd5c8c54-ln6wt                1/1     Running     0          3m52srook-ceph-operator-547cd645bc-xmkrs             1/1     Running     3          21mrook-ceph-osd-prepare-10.128.130.14-2cvpv       0/1     Completed   0          3m38s
Copy

Storage Class

To use Rook a storage class can be created in the cluster. To create a storage class for a minimal installation use the storageclass-test.yamlfound in the Rook Github repository rook/cluster/examples/kubernetes/ceph/csi/rbd

Example Storage Class for a minimal installation

Bash
    
 
kubectl apply -f storageclass-test.yaml
Copy

storageclass-test.yaml
    
​x
 
apiVersion: ceph.rook.io/v1kind: CephBlockPoolmetadata:  name: replicapool  namespace: rook-ceph # namespace:clusterspec:  failureDomain: host  replicated:    size: 1    # Disallow setting pool with replica 1, this could lead to data loss without recovery.    # Make sure you're *ABSOLUTELY CERTAIN* that is what you want    requireSafeReplicaSize: false    # gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool    # for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size    #targetSizeRatio: .5---apiVersion: storage.k8s.io/v1kind: StorageClassmetadata:   name: rook-ceph-block# Change "rook-ceph" provisioner prefix to match the operator namespace if neededprovisioner: rook-ceph.rbd.csi.ceph.com  # driver:namespace:operatorparameters:    # clusterID is the namespace where the rook cluster is running    # If you change this namespace, also change the namespace below where the secret namespaces are defined    clusterID: rook-ceph # namespace:cluster​    # If you want to use erasure coded pool with RBD, you need to create    # two pools. one erasure coded and one replicated.    # You need to specify the replicated pool here in the `pool` parameter, it is    # used for the metadata of the images.    # The erasure coded pool must be set as the `dataPool` parameter below.    #dataPool: ec-data-pool    pool: replicapool​    # RBD image format. Defaults to "2".    imageFormat: "2"​    # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.    imageFeatures: layering​    # The secrets contain Ceph admin credentials. These are generated automatically by the operator    # in the same namespace as the cluster.    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph # namespace:cluster    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner    csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph # namespace:cluster    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # namespace:cluster    # Specify the filesystem type of the volume. If not specified, csi-provisioner    # will set default as `ext4`.    csi.storage.k8s.io/fstype: ext4# uncomment the following to use rbd-nbd as mounter on supported nodes#mounter: rbd-nbdallowVolumeExpansion: truereclaimPolicy: Delete
Copy

Bash
    
 
cjones@pf9-0108 ceph % kubectl -n rook-ceph get podNAME                                  READY   STATUS             RESTARTS   AGErook-ceph-operator-547cd645bc-xmkrs   0/1     ImagePullBackOff   0          105s
Copy

Then run describe pod to view the pod events

Bash
    
 
kubectl describe pod rook-ceph-operator-547cd645bc-xmkr -n rook-ceph
Copy

The Events section will document the likely cause of the issue.

Bash
    
Events:  Type     Reason     Age                  From                    Message  ----     ------     ----                 ----                    -------  Normal   Scheduled  <unknown>            default-scheduler       Successfully assigned rook-ceph/rook-ceph-operator-547cd645bc-xmkrs to 10.128.130.14  Normal   Pulling    75s (x4 over 2m47s)  kubelet, 10.128.130.14  Pulling image "rook/ceph:v1.5.9"  Warning  Failed     73s (x4 over 2m46s)  kubelet, 10.128.130.14  Failed to pull image "rook/ceph:v1.5.9": rpc error: code = Unknown desc = Error response from daemon: manifest for rook/ceph:v1.5.9 not found: manifest unknown: manifest unknown  Warning  Failed     73s (x4 over 2m46s)  kubelet, 10.128.130.14  Error: ErrImagePull  Normal   BackOff    58s (x6 over 2m46s)  kubelet, 10.128.130.14  Back-off pulling image "rook/ceph:v1.5.9"  Warning  Failed     44s (x7 over 2m46s)  kubelet, 10.128.130.14  Error: ImagePullBackOff
Copy

Last updated on

Was this page helpful?

Rook Ceph CSI

Minimum Requirements

Minimal Installation

Requirements:

Step 1 - Install LVM

Step 2 - Clone Rook Github Repo

Step 3 - Install Rook

Step 4 - Cluster Status

Storage Class

Example Storage Class for a minimal installation

Known Error Failed to pull image

Resolution

How to Diagnose