When running a kubernetes cluster backed by rook/ceph, such as kURL, you may want to know how much actual data is stored in ceph on the mounted PVs. To do this you can run these commands
NOTE: These commands assume a kURL cluster with the rook/ceph addon
- Get the rook/ceph tools pod name. We’ll run ceph commands on this pod
kubectl get pods -l app=rook-ceph-tools -n rook-ceph
NAME READY STATUS RESTARTS AGE
rook-ceph-tools-6465749568-jp4jc 1/1 Running 0 48m
- Check the overall usage of ceph storage. Here you can see how the object store (
rook-ceph-store), ceph pools (replicapool) and the shared file system (rook-shared-fs) utilise the available space. In my test cluster,replicapoolceph pool is used to provide storage space for kubernetes PVs.
kubectl exec -n rook-ceph rook-ceph-tools-6465749568-jp4jc -- ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 200 GiB 197 GiB 2.6 GiB 2.6 GiB 1.30
TOTAL 200 GiB 197 GiB 2.6 GiB 2.6 GiB 1.30
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.mgr 1 1 577 KiB 2 580 KiB 0 187 GiB
replicapool 2 32 2.5 GiB 705 2.5 GiB 1.33 187 GiB
rook-ceph-store.rgw.control 3 8 0 B 8 0 B 0 187 GiB
rook-shared-fs-metadata 4 16 3.3 KiB 19 32 KiB 0 187 GiB
rook-ceph-store.rgw.meta 5 8 2.8 KiB 14 48 KiB 0 187 GiB
rook-shared-fs-data0 6 32 0 B 0 0 B 0 187 GiB
rook-ceph-store.rgw.log 7 8 23 KiB 308 652 KiB 0 187 GiB
rook-ceph-store.rgw.buckets.index 8 8 0 B 33 0 B 0 187 GiB
rook-ceph-store.rgw.buckets.non-ec 9 8 0 B 0 0 B 0 187 GiB
rook-ceph-store.rgw.otp 10 8 0 B 0 0 B 0 187 GiB
.rgw.root 11 8 4.8 KiB 16 60 KiB 0 187 GiB
rook-ceph-store.rgw.buckets.data 12 32 807 KiB 5 816 KiB 0 187 GiB
- Check your PVs. Note that the capacity values are reservation of space in the storage engine (ceph). Think of this as the size of your disk in a VM.
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-49d1c4af-d50b-40b5-ab93-a370709206af 1Gi RWO Delete Bound default/kotsadm-rqlite-kotsadm-rqlite-0 default 30m
pvc-ade284ee-1917-4cb3-a278-d246241a8065 10Gi RWO Delete Bound default/nginx-claim default 5m37s
pvc-bc75a782-f9d0-4fd1-bf58-79b29b27a879 10Gi RWO Delete Bound minio/minio-pv-claim default 32m
- Find out how much space is used in your volume. We’ll inspect
default/nginx-claimwhich has a capacity of10Gi.- Get the
imageNamevalue from the PV object. This maps to a RADOS Block Device (RBD) image which is a ceph block storage primitive.
- Get the
kubectl describe pv pvc-ade284ee-1917-4cb3-a278-d246241a8065
Name: pvc-ade284ee-1917-4cb3-a278-d246241a8065
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: rook-ceph.rbd.csi.ceph.com
volume.kubernetes.io/provisioner-deletion-secret-name: rook-csi-rbd-provisioner
volume.kubernetes.io/provisioner-deletion-secret-namespace: rook-ceph
Finalizers: [kubernetes.io/pv-protection]
StorageClass: default
Status: Bound
Claim: default/nginx-claim
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: rook-ceph.rbd.csi.ceph.com
FSType: ext4
VolumeHandle: 0001-0009-rook-ceph-0000000000000002-7499c92d-213d-4ea6-94e0-35f27ca677b1
ReadOnly: false
VolumeAttributes: clusterID=rook-ceph
imageFeatures=layering
imageFormat=2
imageName=csi-vol-7499c92d-213d-4ea6-94e0-35f27ca677b1
journalPool=replicapool
pool=replicapool
storage.kubernetes.io/csiProvisionerIdentity=1706275215978-8081-rook-ceph.rbd.csi.ceph.com
Events: <none>
- Now lets check how much space of the provisioned volume is in use. Note that
csi-vol-7499c92d-213d-4ea6-94e0-35f27ca677b1has2.6GiBof the10GiBin use
kubectl exec -n rook-ceph rook-ceph-tools-6465749568-jp4jc -- rbd du --pool=replicapool
NAME PROVISIONED USED
csi-vol-51fb01a2-54d2-472c-8149-d8856acd7ecc 10 GiB 100 MiB
csi-vol-7499c92d-213d-4ea6-94e0-35f27ca677b1 10 GiB 2.6 GiB
csi-vol-8501040b-c714-48e1-9421-24262ce6f0c8 1 GiB 40 MiB
<TOTAL> 21 GiB 2.7 GiB
- You can also run
radosgw-admin bucket statscommand to check how the object store is utilised. The command can be verbose but you can see number of buckets and usage stats of each bucket. Here is an example truncated output
kubectl exec -n rook-ceph rook-ceph-tools-6465749568-jp4jc -- radosgw-admin bucket stats
[
{
"bucket": "kotsadm",
"num_shards": 11,
"tenant": "",
"zonegroup": "218b132c-988e-4b6c-b44d-f6c558c6312f",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "14f0b1c9-2337-4094-8d03-8f03a3d65df9.4589.2",
"marker": "14f0b1c9-2337-4094-8d03-8f03a3d65df9.4589.2",
"index_type": "Normal",
"owner": "kurl",
"ver": "0#4,1#1,2#1,3#1,4#1,5#2,6#2,7#1,8#1,9#1,10#2",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "0.000000",
"creation_time": "2024-01-26T13:26:48.467580Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {
"rgw.main": {
"size": 826056,
"size_actual": 835584,
"size_utilized": 826056,
"size_kb": 807,
"size_kb_actual": 816,
"size_kb_utilized": 807,
"num_objects": 5
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
},
...
<truncated>
...
]