Ceph Basics for Cluster Wizard

Cluster-Wizard requires a ceph.conf and a keyring file for ceph integration.

Obtaining ceph.conf file

To obtain a ceph.conf file connect to a ceph monitor and run:

sudo ceph config generate-minimal-conf -o ceph.conf

A file ceph.conf contains the information needed for a connection to a ceph cluster and will look like:

# minimal ceph.conf for 8108d676-8848-11aa-8e88-35b7aac3aa46
[global]
	fsid = 8108d676-8848-11aa-8e88-35b7aac3aa46
	mon_host = [v2:172.17.3.1:3300/0,v1:172.17.3.1:6789/0] [v2:172.17.3.2:3300/0,v1:172.17.3.2:6789/0] [v2:172.17.3.3:3300/0,v1:172.17.3.3:6789/0]

warning

ceph.conf must contain "keyring = /cluster_wizard/libvirt.keyring" for Cluster-Wizard.

note

The IP addresses of mon_host. In this case there are 3 hosts with the IPs of 172.17.3.1, 172.17.3.2 and 172.17.3.3. The number of hosts will be ceph cluster dependent. This information will be needed to create a virsh pool.

Creating a Ceph RBD pool

Create the libvirt-pool RBD pool on your Ceph cluster for Cluster-Wizard by running the following commands on a Ceph monitor:

sudo ceph osd pool create libvirt-pool  
sudo rbd pool init libvirt-pool

Create a Ceph user for RBD pool access

Create the client.libvirt user for Cluster-Wizard to access the libvirt-pool RBD pool by running the following command on a Ceph monitor::

sudo ceph auth get-or-create client.libvirt \
  mon 'profile rbd' \
  osd 'profile rbd pool=libvirt-pool' \
  mgr 'profile rbd pool=libvirt-pool' \
  -o libvirt.keyring

The libvirt.keyring file contains the credentials for RBD pool access and looks like this:

[client.libvirt]
	key = CQC+j3xl/bcROhCCBd0aGSfiTIie9gRJEewYRw==

Create a virsh pool from a RBD pool

Install ceph-common and libvirt-daemon-driver-storage-rbd on all KVM hosts that will use the RBD pool for VM image storage

On an Ubuntu systems, run:

sudo apt install ceph-common  
sudo apt install libvirt-daemon-driver-storage-rbd

Generate a UUID with uuidgen

uuidgen
70b3294b-7c5b-4e5c-b3f5-0cc561969df3

Create an xml file using the uuid and ceph user created above. libvirt-secret.xml:

<secret ephemeral='no' private='no'>
  <uuid>70b3294b-7c5b-4e5c-b3f5-0cc561969df3</uuid>
    <usage type='ceph'>
    <name>client.libvirt secret</name>
  </usage>
</secret>

Next, we need to define our secret and set it using our uuid and the libvirt.keyring key value.

virsh secret-define --file libvirt-secret.xml  
virsh secret-list
#virsh secret-set-value --secret <uuid> --base64 <key>
virsh secret-set-value \
  --secret "70b3294b-7c5b-4e5c-b3f5-0cc561969df3" \
  --base64 "CQC+j3xl/bcROhCCBd0aGSfiTIie9gRJEewYRw=="

Create an xml file with the information needed to define a virsh pool with RBD access. The file ceph.conf will contain the host name/IP information. The username and pool name will be from the commands above.

libvirt-rbd-pool.xml:

<pool type="rbd">
  <name>libvirt-pool</name>
  <source>
    <name>libvirt-pool</name>
    <host name='172.17.3.1' port='6789'/>
    <host name='172.17.3.2' port='6789'/>
    <host name='172.17.3.3' port='6789'/>
    <auth username='libvirt' type='ceph'>
      <secret uuid='70b3294b-7c5b-4e5c-b3f5-0cc561969df3'/>
    </auth>
  </source>
</pool>

Define the pool, enable it, autostart start it, and verify its status. Once done, restart libvirtd.

virsh pool-define libvirt-rbd-pool.xml     
virsh pool-autostart libvirt-pool  
virsh pool-start  libvirt-pool

#Checking
virsh pool-list --all        
 Name                State    Autostart
-----------------------------------------
 ...
 libvirt-rbd-pool    active   yes
 ...

sudo systemctl restart libvirtd

Run the following command to verify everything is OK by listing volumes in libvirt-pool.

virsh vol-list --pool libvirt-rbd-pool