# Determine disk usage by volumes on a ceph storage cluster **URL:** \ **Category:** How do I?\ **Tags:** kurl, ceph, rook\ **Created:** [January 26, 2024, 2:40pm UTC](https://community.replicated.com/t/determine-disk-usage-by-volumes-on-a-ceph-storage-cluster/1354 "2024-01-26T14:40:03Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Evans\_Mungai](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/evans_mungai/32/546_2.png) [@Evans\_Mungai](https://community.replicated.com/u/Evans_Mungai)\ **Post date:** [January 26, 2024, 2:40pm UTC](https://community.replicated.com/t/determine-disk-usage-by-volumes-on-a-ceph-storage-cluster/1354/1 "2024-01-26T14:40:03Z")
When running a kubernetes cluster backed by rook/ceph, such as [kURL](https://kurl.sh/), you may want to know how much actual data is stored in ceph on the mounted PVs. To do this you can run these commands NOTE: These commands assume a kURL cluster with the [rook/ceph addon](https://kurl.sh/docs/add-ons/rook) - Get the rook/ceph tools pod name. We’ll run ceph commands on this pod ```sh kubectl get pods -l app=rook-ceph-tools -n rook-ceph NAME READY STATUS RESTARTS AGE rook-ceph-tools-6465749568-jp4jc 1/1 Running 0 48m ``` - Check the overall usage of ceph storage. Here you can see how the object store (`rook-ceph-store`), ceph pools (`replicapool`) and the shared file system (`rook-shared-fs`) utilise the available space. In my test cluster, `replicapool` ceph pool is used to provide storage space for kubernetes PVs. ```sh kubectl exec -n rook-ceph rook-ceph-tools-6465749568-jp4jc -- ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 200 GiB 197 GiB 2.6 GiB 2.6 GiB 1.30 TOTAL 200 GiB 197 GiB 2.6 GiB 2.6 GiB 1.30 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 577 KiB 2 580 KiB 0 187 GiB replicapool 2 32 2.5 GiB 705 2.5 GiB 1.33 187 GiB rook-ceph-store.rgw.control 3 8 0 B 8 0 B 0 187 GiB rook-shared-fs-metadata 4 16 3.3 KiB 19 32 KiB 0 187 GiB rook-ceph-store.rgw.meta 5 8 2.8 KiB 14 48 KiB 0 187 GiB rook-shared-fs-data0 6 32 0 B 0 0 B 0 187 GiB rook-ceph-store.rgw.log 7 8 23 KiB 308 652 KiB 0 187 GiB rook-ceph-store.rgw.buckets.index 8 8 0 B 33 0 B 0 187 GiB rook-ceph-store.rgw.buckets.non-ec 9 8 0 B 0 0 B 0 187 GiB rook-ceph-store.rgw.otp 10 8 0 B 0 0 B 0 187 GiB .rgw.root 11 8 4.8 KiB 16 60 KiB 0 187 GiB rook-ceph-store.rgw.buckets.data 12 32 807 KiB 5 816 KiB 0 187 GiB ``` - Check your PVs. Note that the capacity values are reservation of space in the storage engine (ceph). Think of this as the size of your disk in a VM. ```sh kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-49d1c4af-d50b-40b5-ab93-a370709206af 1Gi RWO Delete Bound default/kotsadm-rqlite-kotsadm-rqlite-0 default 30m pvc-ade284ee-1917-4cb3-a278-d246241a8065 10Gi RWO Delete Bound default/nginx-claim default 5m37s pvc-bc75a782-f9d0-4fd1-bf58-79b29b27a879 10Gi RWO Delete Bound minio/minio-pv-claim default 32m ``` - Find out how much space is used in your volume. We’ll inspect `default/nginx-claim` which has a capacity of `10Gi`. - Get the `imageName` value from the PV object. This maps to a RADOS Block Device (RBD) image which is a ceph block storage primitive. ```sh kubectl describe pv pvc-ade284ee-1917-4cb3-a278-d246241a8065 Name: pvc-ade284ee-1917-4cb3-a278-d246241a8065 Labels: Annotations: pv.kubernetes.io/provisioned-by: rook-ceph.rbd.csi.ceph.com volume.kubernetes.io/provisioner-deletion-secret-name: rook-csi-rbd-provisioner volume.kubernetes.io/provisioner-deletion-secret-namespace: rook-ceph Finalizers: [kubernetes.io/pv-protection] StorageClass: default Status: Bound Claim: default/nginx-claim Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 10Gi Node Affinity: Message: Source: Type: CSI (a Container Storage Interface (CSI) volume source) Driver: rook-ceph.rbd.csi.ceph.com FSType: ext4 VolumeHandle: 0001-0009-rook-ceph-0000000000000002-7499c92d-213d-4ea6-94e0-35f27ca677b1 ReadOnly: false VolumeAttributes: clusterID=rook-ceph imageFeatures=layering imageFormat=2 imageName=csi-vol-7499c92d-213d-4ea6-94e0-35f27ca677b1 journalPool=replicapool pool=replicapool storage.kubernetes.io/csiProvisionerIdentity=1706275215978-8081-rook-ceph.rbd.csi.ceph.com Events: ``` - Now lets check how much space of the provisioned volume is in use. Note that `csi-vol-7499c92d-213d-4ea6-94e0-35f27ca677b1` has `2.6GiB` of the `10GiB` in use ```sh kubectl exec -n rook-ceph rook-ceph-tools-6465749568-jp4jc -- rbd du --pool=replicapool NAME PROVISIONED USED csi-vol-51fb01a2-54d2-472c-8149-d8856acd7ecc 10 GiB 100 MiB csi-vol-7499c92d-213d-4ea6-94e0-35f27ca677b1 10 GiB 2.6 GiB csi-vol-8501040b-c714-48e1-9421-24262ce6f0c8 1 GiB 40 MiB 21 GiB 2.7 GiB ``` - You can also run `radosgw-admin bucket stats` command to check how the object store is utilised. The command can be verbose but you can see number of buckets and usage stats of each bucket. Here is an example truncated output ```sh kubectl exec -n rook-ceph rook-ceph-tools-6465749568-jp4jc -- radosgw-admin bucket stats [ { "bucket": "kotsadm", "num_shards": 11, "tenant": "", "zonegroup": "218b132c-988e-4b6c-b44d-f6c558c6312f", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "14f0b1c9-2337-4094-8d03-8f03a3d65df9.4589.2", "marker": "14f0b1c9-2337-4094-8d03-8f03a3d65df9.4589.2", "index_type": "Normal", "owner": "kurl", "ver": "0#4,1#1,2#1,3#1,4#1,5#2,6#2,7#1,8#1,9#1,10#2", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", "mtime": "0.000000", "creation_time": "2024-01-26T13:26:48.467580Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", "usage": { "rgw.main": { "size": 826056, "size_actual": 835584, "size_utilized": 826056, "size_kb": 807, "size_kb_actual": 816, "size_kb_utilized": 807, "num_objects": 5 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } }, ... ... ] ```