In Copr we use Mock as underlying tool for building packages. After recent changes in Copr I discovered that Mock is quite slow. In fact the whole VM was quite slow. I quickly discovered that the issue is disk read/write bandwidth.
Copr spin up a new VM for every task. Right now we have more that 20 VM running in parallel. That is approximately 4-5 VM per one hypervisor. And building package is mostly disk intensive task. Those days when you have to compile everything are gone. Usually it is installing build requirements, unpacking tar file, moving packages to build roots, creating rpm package itself... I discovered that Mock can easily generate disk bandwidth more than 20 MB/s. Times five VM and you have 100 MB/s which too much for the RAID disks in a Compute node.
Our VM are created with 2 VCPU, 50 GB root disk and 5 GB RAM. The root disk is technically created on Compute node in /var/lib/nova/instances as QCOW2 image.
I realized that Mock have TmpFS plugin and we could build package in memory. Our OpenStack instance is quite beefy and have 690 GB RAM in total. However if I want to keep 50 GB for package maintainer, that would give space just for 13 machines. But then I realized that TmpFS can be larger than available memory. When you store there more than is available, it will be just swapped to swap disk. So I can keep 5 GB RAM, just have big big swap disk.
Swap disks are just ephemeral storage, which are treated similar as RootFS. They are stored in /var/lib/nova/instances as QCOW2 instances, but OpenStack will automatically format and mount them for you. So when I create VM with 20 GB root disk, 5 GB RAM and 30 GB swap disk, enable TmpFS plugin, then I am consuming exactly the same resources as previously. However the build task will be done in RAM and off-load the disk. Lets do the some tests with this setup.
Building of mock.src.rpm takes 3 minutes and 2 seconds. When I enable TmpFS it takes 28 seconds. And swap was not used at all. This task was completely done in a memory. Now something more heavy - kernel.src.rpm. It takes 231 minutes and 46 seconds in our VM to build kernel. And when I enable TmpFS it take just 161 minutes and 49 seconds to finish. At some point the TmpFS of mock's chroot grew to 15 GB so some data was writen to swap (I can imagine /usr/share/doc files landed there) however most tasks were still done in memory.
For small packages the improvement is huge (reduced to 16 % of original time), less for huge packages but still significant (reduced to 70 % of original time).
Because we have plenty of space in /var/lib/nova mount point in our Compute nodes (3TB on each node) I decided to use 5 GB RAM, 20 GB RootFS and 50 GB swap. Therefore you will have even more space if needed and the build will be still faster. I already prepared instance images and next week I will put this setup in Copr production.
For the record this is the settings I am using (in /etc/mock/site-defaults.cfg):
config_opts['plugin_conf']['tmpfs_enable'] = True
config_opts['plugin_conf']['tmpfs_opts'] = {}
config_opts['plugin_conf']['tmpfs_opts']['required_ram_mb'] = 1024
config_opts['plugin_conf']['tmpfs_opts']['max_fs_size'] = '50g'
config_opts['plugin_conf']['tmpfs_opts']['mode'] = '0755'
config_opts['plugin_conf']['tmpfs_opts']['keep_mounted'] = False