前提條件

一個可以正常執行的Ceph叢集，該叢集採用ceph-deploy建立，dataprovider為管理節點，包括三個MON,四個MON 以及Openstack叢集，其中Openstack叢集假設Cinder,Glance都是安裝在叫做controllernode的節點上，computernode,networknode分別為計算以及網路節點

建立Pool

在dataprovider上建立幾個Pool

ceph osd pool create volumes 32
ceph osd pool create images 32
ceph osd pool create backups 32
ceph osd pool create vms 32
 

配置Ceph Client

在glance節點執行下面的命令

sudo yum install python-ceph

在nova-compute,cinder-backup,cinder-volume節點,執行下面的命令:

sudo yum install ceph

在controller,computenode上建立leadorceph使用者

sudo useradd -d /home/leadorceph -m leadorceph
sudo passwd leadorceph
echo "leadorceph  ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/leadorceph
sudo chmod 0440 /etc/sudoers.d/leadorceph
 

使用leadorceph使用者的身份執行sudo visudo命令，然後修改Defaults requiretty 為Defaults:ceph !requiretty

在dataprovider 上設定無密碼登陸

ssh-copy-id [email protected]
ssh-copy-id [email protected]

配置授權

在dataprovider節點上執行下面的命令

ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'
ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'

ceph auth get-or-create client.glance | ssh controllernode sudo tee /etc/ceph/ceph.client.glance.keyring
ssh controllernode sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring
ceph auth get-or-create client.cinder | ssh controllernode  sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh controllernode  sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder-backup | ssh controllernode  sudo tee /etc/ceph/ceph.client.cinder-backup.keyring
ssh controllernode  sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring

ceph auth get-key client.cinder | ssh computenode tee client.cinder.key
 

在computenode上執行下面的操作

切換使用者su leadorceph

uuidgen
上面命令的輸出為78f475b1-846f-47ba-8145-9f305de5c516

cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
  <uuid>78f475b1-846f-47ba-8145-9f305de5c516</uuid>
  <usage type='ceph'>
    <name>client.cinder secret</name>
  </usage>
</secret>
EOF
sudo virsh secret-define --file secret.xml
sudo virsh secret-set-value --secret 78f475b1-846f-47ba-8145-9f305de5c516 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml

配置glance

在controller節點，切換到root使用者

編輯 /etc/glance/glance-api.conf 檔案中的 [DEFAULT]:

default_store = rbd
rbd_store_user = glance
rbd_store_pool = images
rbd_store_chunk_size = 8

配置cinder

在/etc/cinder/cinder.conf中新增下面的內容:

volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2

rbd_user = cinder


rbd_secret_uuid = 78f475b1-846f-47ba-8145-9f305de5c516

配置cinder backup

backup_driver = cinder.backup.drivers.ceph
backup_ceph_conf = /etc/ceph/ceph.conf
backup_ceph_user = cinder-backup
backup_ceph_chunk_size = 134217728
backup_ceph_pool = backups
backup_ceph_stripe_unit = 0
backup_ceph_stripe_count = 0
restore_discard_excess_bytes = true

配置VM掛載的許可權

在 /etc/nova/nova.conf中新增以下內容

rbd_user = cinder

rbd_secret_uuid = 78f475b1-846f-47ba-8145-9f305de5c516

編輯computenode 上的ceph配置

[client]
    rbd cache = true
    rbd cache writethrough until flush = true
    admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok

H以及I版本的改動

computernode 的/etc/nova/nova.conf 檔案新增下面的內容:

libvirt_images_type = rbd
libvirt_images_rbd_pool = vms
libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = 78f475b1-846f-47ba-8145-9f305de5c516
libvirt_inject_password = false
libvirt_inject_key = false
libvirt_inject_partition = -2
libvirt_live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST"

重啟服務

sudo service openstack-glance-api restart
sudo service openstack-nova-compute restart
sudo service openstack-cinder-volume restart
sudo service openstack-cinder-backup restart

驗證

建立硬碟，映象後，在dadaprovider節點檢視rados pool的狀況

rados -p images ls
rados -p volumes ls