Ceph RBD as KVM VM Storage with libvirt

A single local disk is a single point of failure. By backing your KVM virtual machines with Ceph RBD (RADOS Block Device), each VM disk becomes a replicated, self-healing object spread across your storage cluster — the VM survives a dead drive or even a dead host, and you can live-migrate it between hypervisors because the storage is shared. This is exactly how OpenStack Nova and Cinder back instances in production.

This tutorial connects compute and storage: you will wire a KVM/libvirt host to an existing Ceph cluster, create an RBD image, register it as a libvirt storage pool, and attach a resilient block device to a VM. It assumes you have already created a Ceph cluster with cephadm and are comfortable with virsh basics.

Prerequisites

You need a healthy Ceph cluster with an rbd pool, a KVM/libvirt host on Ubuntu 24.04 with qemu-kvm and libvirt-daemon-system installed, network reachability from the hypervisor to the Ceph monitors (port 6789/3300), and admin access to both. The examples use a monitor at 10.10.10.11.

Step 1: Install the Ceph client on the hypervisor

The libvirt/QEMU host needs the Ceph client libraries and the RBD tooling:

sudo apt update
sudo apt install -y ceph-common

Copy /etc/ceph/ceph.conf from a cluster node so the client knows the monitors:

sudo scp root@ceph-1:/etc/ceph/ceph.conf /etc/ceph/ceph.conf

Step 2: Create a dedicated Ceph user for libvirt

Never hand libvirt the admin key. Create a scoped client.libvirt user that can only touch the rbd pool. On a Ceph node:

sudo cephadm shell -- ceph auth get-or-create client.libvirt \
  mon 'profile rbd' \
  osd 'profile rbd pool=rbd'

Print the key — you will feed it to libvirt as a secret in the next step:

sudo cephadm shell -- ceph auth get-key client.libvirt

Step 3: Define the Ceph auth secret in libvirt

libvirt stores the Ceph key as a secret object. Create a definition file using single-quoted attributes:

cat > secret.xml <<'EOF'
<secret ephemeral='no' private='no'>
  <usage type='ceph'>
    <name>client.libvirt</name>
  </usage>
</secret>
EOF
virsh secret-define --file secret.xml

That prints a UUID. Bind the actual Ceph key to it (substitute your key and UUID):

virsh secret-set-value \
  --secret 2a1b... \
  --base64 AQD...your-key...==

Step 4: Create an RBD image for the VM

Provision a 20 GiB block image in the rbd pool. Disabling newer features keeps it compatible with the QEMU RBD driver:

rbd create rbd/vm-disk1 --size 20480 \
  --image-feature layering
rbd info rbd/vm-disk1
rbd ls rbd

Step 5: Register an RBD-backed libvirt storage pool

Define a libvirt pool that maps onto the Ceph rbd pool, referencing the secret UUID from Step 3:

cat > rbd-pool.xml <<'EOF'
<pool type='rbd'>
  <name>rbd-libvirt</name>
  <source>
    <name>rbd</name>
    <host name='10.10.10.11' port='6789'/>
    <auth username='libvirt' type='ceph'>
      <secret uuid='2a1b...'/>
    </auth>
  </source>
</pool>
EOF
virsh pool-define rbd-pool.xml
virsh pool-start rbd-libvirt
virsh pool-autostart rbd-libvirt

Confirm libvirt can see the images in the pool:

virsh vol-list rbd-libvirt

Step 6: Attach the RBD device to a VM

Describe the network disk and attach it to a running domain. The auth block points at the same secret:

cat > disk.xml <<'EOF'
<disk type='network' device='disk'>
  <driver name='qemu' type='raw'/>
  <source protocol='rbd' name='rbd/vm-disk1'>
    <host name='10.10.10.11' port='6789'/>
  </source>
  <auth username='libvirt'>
    <secret type='ceph' uuid='2a1b...'/>
  </auth>
  <target dev='vdb' bus='virtio'/>
</disk>
EOF
virsh attach-device ubuntu-test disk.xml --persistent

Step 7: Verify inside the guest

Inside the VM, the new device appears as /dev/vdb. Format and mount it:

lsblk
sudo mkfs.ext4 /dev/vdb
sudo mkdir -p /mnt/ceph
sudo mount /dev/vdb /mnt/ceph
df -h /mnt/ceph

Troubleshooting and pitfalls

• ‘failed to connect to the RADOS monitor’ — the hypervisor cannot reach the monitor IP/port, or /etc/ceph/ceph.conf is missing.
• Authentication errors — the base64 key in the libvirt secret does not match client.libvirt, or the secret UUID in the XML is wrong.
• ‘feature set mismatch’ — the RBD image enables features the client kernel/QEMU does not support; recreate with only --image-feature layering.
• Disk not visible in guest — you attached without --persistent and rebooted, or the bus type is unsupported; use virtio.

Where to go next

You have now joined hypervisor and storage cluster — the foundation of any private cloud. The same RBD pools transparently back OpenStack Cinder volumes, and you manage these disks day-to-day with the virsh commands covered earlier.

Conclusion

Backing KVM VMs with Ceph RBD gives you replication, live migration, and storage that outlives any single disk or host — without buying a proprietary SAN. This compute-plus-storage pattern is exactly how clouditiv backs every instance with resilient storage. Assembling and operating Ceph, libvirt, secrets, and networking yourself is real work; if you would rather just launch resilient VMs, clouditiv runs a sovereign, ISO 27001 / BSI C5 private cloud on Ubuntu 24.04 + OpenStack 2025.2 with Ceph-backed storage and your data in Germany. Explore our on-premise cloud solution.