Miroslav Suchý

How Openstack handle RootDisk of VMs

RootDisk of VM is persistent and I always thought that it is stored together with all other volumes. In Cinder. I was soo wrong. Curious too?

Let start with terminology. Openstack knows these storages:

• Images - this is stored in Glance service. When VM start RootDisk is created from content of some image.
• Volumes - this is stored in Cinder service. You can attach Volume to VM and its life is independent from VM.
• Ephemeral - this is handled by Nova service and it is created in /var/lib/nova/instances on Compute node. If you shut down, reset or move VM, then this device is cleared and you have to partition/format it again. On the other hand this device is local on Compute node and therefore usually faster.
• Swap - this is just Ephemeral storage, but cloud-init will mkswap and mount it as swap for you.
• RootDisk - this is ... what? It is persistent across reboots. But it is not stored in Cinder service. Who handle it? And how?

RootDisk is basicaly Ephemeral storage with special handling. It is created by Nova service in /var/lib/nova/instances and is stored only on Compute node where VM was started. If you shutdown instance and then again start it up, VM will always stay on the same Compute node, because RootDisk is only there. You must initiate Migration to move it to another Compute node. Then Nova copy RootDisk (via SSH) to another node (other Ephemeral storages and swap is not migrated).

If your Compute nodes use some cheap storage then you should know that when Compute node dies, it will take some RootDisks to the grave.

One solution can be /var/lib/nova/instances located on some better storage and then e.g. NFS exported to Compute nodes (this way you get shared storage and Live migration will start working). On the other hand Ephemeral devices then does not have too much sense as they will likely consume the same storage as normal Volumes.