Blog

Configuring multiple block storage backends in OpenStack Cinder

Configuring multiple block storage backends in OpenStack Cinder

If you're an administrator of virtualized environments you have definetly ran into IO performance issues. IO is the first bottleneck that one hits. Luckily persistant storage has evolved troughout the years and lately we see the high performant SSDs at a reasonable price, still far higher to allow organizations to fully migrate to SSD. The hybrid environments become more and more popular as they combine the low costs for traditional HDD with the high performant Solid state drives. One of the key features of the Cinder storage back- end component of Openstack is the flexibility that allows us to have more than one storage backend on our storage node. This gives us the flexibility to diferentiate the IO heavy applications from the more compute-oriented ones that are more heavy on cpu usage. Typical example is to configure a database VM to run from a SSD drive and the application server cluster to be on a normal storage that is heavily read only during startup.

Here is how to achive that with OpenStack and Cinder.

In our environment we use the NFS volume driver which allows for easy to configure distributed volumes that can be mounted across all compute nodes. It's a different debate wether this is the most appropriate and stable solution giving the fact there is a wide variaty of storage backends - ceph, glusterfs, iSCSI based ones. You can apply the differentiation of SSD and HDD with any one of them.

Our first task will be to have different mountpoints on our server residing on LVM or directly a physical partition (not recommended) on both our SSD storage array or disk and our plain old HDDs. We need to define those mountpoints as different exports in /etc/exports

/exporthdd 192.168.0.0/24(rw,no_subtree_check,sync,no_root_squash)  # this is a mountpoint of plain hard disk physical devices
/exportssd 192.168.0.0/24(rw,no_subtree_check,sync,no_root_squash)  # this is a mountpoint for the SSD physical disks
Make sure NFS exports are loaded with the exportsfs command.

As described in the RedHat documentation of Openstack, we have the option to define multiple configuration blocks in the cinder.conf file. However it leaves the impression that they are for defining different storage drivers. While in fact this is the most typical use case, cinder doesn't limit you to define multiple blocks with the same driver.

[NAME]
volume_group=GROUP
volume_driver=DRIVER
volume_backend_name=BACKEND

Now edit the main configuration file:

/etc/cinder/cinder.conf

Add the newly defined block below the existing one [nfs] and name it [nfsssd]

[nfsssd]
nfs_used_ratio=0.95
nfs_oversub_ratio=1.0
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config=/etc/cinder/nfs_shares_ssd.conf
volume_backend_name=nfsssd

 It's very importnant to define a volume_backend_name. This will allow this backend to appear in the Horizon GUI and also to be selectable when creating a volume via command line. To enable this backend we also have to specify it in the same configuration file /etc/cinder/cinder.conf in the top section in a comma separated list.

enabled_backends=XXX,XXX,nfs,nfsssd

As seen in the nfsssd section we define a different nfs shares config file which describes the mountpoints and paths to the

To allow users to specify which backend their volumes are created on rather than relying on the scheduler to select automatically a volume type must be defined in the database. Once the services are restarted and running this can be done using these commands:

cinder type-create nfsssd
cinder type-key <the id returned from the create command> set volume_backend_name='nfsssd'

This is the relation to the volume backend name specified in the configuration and will allow us to chose when creating a volume what type it should be or in other words on which backend it should reside.

To test out the create you can use the command line tool specifying the volume type.

cinder create --display-name test1 --volume-type nfsssd 2

Now this volume can be mounted to any of our virtual machines.

I hope this article helped you, I would appreciate if you leave comments on how to optimize more the the ssd backend, wether it's better for performance if it's not on LVM but directly on disk.

Also I would like to hear your stories, when did you need to use different volume backends and what would you preffer instad of NFS.