I have several computers where I have discs in RAID1 configuration (aka mirroring). All of them are magnetics HDD. I am thinking about replacing them with SSD. However to replace all of them, would cost a lot of money.Hmm - what if I only replace one (of two) disc in that array. What will be the performance. Do I get the best from HDD and SSD? Or the worst? There is a lot of benchmarks of HDD alone, a lot of benchmarks of SSD alone. But the mix in RAID is unknow.
Recently I got one spare SSD and some HDD so I was able to do some benchmarks.
First reading tests (write benchmarking is in second part of this blogpost). I used iops test.
The throughput of my SSD is between 61.6 Mbit/s and 2.4 Gbit/s. Raw data:
# python iops-2011-02-11 /dev/sdb
/dev/sdb, 120.03 GB, 32 threads:
512 B blocks: 15033.9 IO/s, 7.3 MiB/s ( 61.6 Mbit/s)
1 KiB blocks: 14838.3 IO/s, 14.5 MiB/s (121.6 Mbit/s)
2 KiB blocks: 13785.3 IO/s, 26.9 MiB/s (225.9 Mbit/s)
4 KiB blocks: 11490.4 IO/s, 44.9 MiB/s (376.5 Mbit/s)
8 KiB blocks: 11273.0 IO/s, 88.1 MiB/s (738.8 Mbit/s)
16 KiB blocks: 7147.7 IO/s, 111.7 MiB/s (936.9 Mbit/s)
32 KiB blocks: 4037.1 IO/s, 126.2 MiB/s ( 1.1 Gbit/s)
64 KiB blocks: 2177.5 IO/s, 136.1 MiB/s ( 1.1 Gbit/s)
128 KiB blocks: 1139.4 IO/s, 142.4 MiB/s ( 1.2 Gbit/s)
256 KiB blocks: 712.4 IO/s, 178.1 MiB/s ( 1.5 Gbit/s)
512 KiB blocks: 434.5 IO/s, 217.3 MiB/s ( 1.8 Gbit/s)
1 MiB blocks: 241.6 IO/s, 241.6 MiB/s ( 2.0 Gbit/s)
2 MiB blocks: 129.5 IO/s, 259.0 MiB/s ( 2.2 Gbit/s)
4 MiB blocks: 66.5 IO/s, 266.0 MiB/s ( 2.2 Gbit/s)
8 MiB blocks: 34.1 IO/s, 272.9 MiB/s ( 2.3 Gbit/s)
16 MiB blocks: 17.5 IO/s, 280.6 MiB/s ( 2.4 Gbit/s)
The throughput of HDD is between 518.1 kbit/s and 850.9 Mbit/s. Raw data:
# python iops-2011-02-11 /dev/sda
/dev/sda, 1.00 TB, 32 threads:
512 B blocks: 126.5 IO/s, 63.2 KiB/s (518.1 kbit/s)
1 KiB blocks: 115.4 IO/s, 115.4 KiB/s (945.2 kbit/s)
2 KiB blocks: 104.6 IO/s, 209.2 KiB/s ( 1.7 Mbit/s)
4 KiB blocks: 100.0 IO/s, 399.8 KiB/s ( 3.3 Mbit/s)
8 KiB blocks: 97.2 IO/s, 777.7 KiB/s ( 6.4 Mbit/s)
16 KiB blocks: 93.3 IO/s, 1.5 MiB/s ( 12.2 Mbit/s)
32 KiB blocks: 85.9 IO/s, 2.7 MiB/s ( 22.5 Mbit/s)
64 KiB blocks: 78.0 IO/s, 4.9 MiB/s ( 40.9 Mbit/s)
128 KiB blocks: 64.0 IO/s, 8.0 MiB/s ( 67.1 Mbit/s)
256 KiB blocks: 46.1 IO/s, 11.5 MiB/s ( 96.6 Mbit/s)
512 KiB blocks: 35.7 IO/s, 17.9 MiB/s (149.8 Mbit/s)
1 MiB blocks: 30.2 IO/s, 30.2 MiB/s (253.5 Mbit/s)
# python iops-2011-02-11 -n 8 /dev/sda
/dev/sda, 1.00 TB, 8 threads:
512 B blocks: 112.7 IO/s, 56.3 KiB/s (461.4 kbit/s)
1 KiB blocks: 108.8 IO/s, 108.8 KiB/s (891.1 kbit/s)
2 KiB blocks: 101.6 IO/s, 203.2 KiB/s ( 1.7 Mbit/s)
4 KiB blocks: 97.8 IO/s, 391.1 KiB/s ( 3.2 Mbit/s)
8 KiB blocks: 96.1 IO/s, 768.8 KiB/s ( 6.3 Mbit/s)
16 KiB blocks: 90.0 IO/s, 1.4 MiB/s ( 11.8 Mbit/s)
32 KiB blocks: 85.0 IO/s, 2.7 MiB/s ( 22.3 Mbit/s)
64 KiB blocks: 81.2 IO/s, 5.1 MiB/s ( 42.6 Mbit/s)
128 KiB blocks: 78.4 IO/s, 9.8 MiB/s ( 82.3 Mbit/s)
256 KiB blocks: 52.1 IO/s, 13.0 MiB/s (109.2 Mbit/s)
512 KiB blocks: 36.2 IO/s, 18.1 MiB/s (151.8 Mbit/s)
1 MiB blocks: 36.4 IO/s, 36.4 MiB/s (305.3 Mbit/s)
2 MiB blocks: 28.3 IO/s, 56.6 MiB/s (474.4 Mbit/s)
4 MiB blocks: 18.5 IO/s, 74.0 MiB/s (621.2 Mbit/s)
8 MiB blocks: 11.4 IO/s, 91.2 MiB/s (765.3 Mbit/s)
16 MiB blocks: 6.3 IO/s, 101.4 MiB/s (850.9 Mbit/s)
In the second run I lowered number of concurent process, which let the test run with bigger blocks. Those big blocks are close to sequential reading and you can see that HDD is quite close to SSD here. Or at least the gap is not as big as with smaller blocks where the difference is by order of magnitude.
Now I created 40GB partitions on both disc and created software raid1 using both disc (using mdadm). The throughput is between 57.8 Mbit/s and 2.5 Gbit/s. This is very very close to pure SSD performance. Mind that I do not care about absolute numbers, but about the relative change. Relative to pure SSD and HDD performance. Raw data:
# python iops-2011-02-11 /dev/md0
/dev/md0, 42.92 GB, 32 threads:
512 B blocks: 14104.5 IO/s, 6.9 MiB/s ( 57.8 Mbit/s)
1 KiB blocks: 13733.6 IO/s, 13.4 MiB/s (112.5 Mbit/s)
2 KiB blocks: 13097.5 IO/s, 25.6 MiB/s (214.6 Mbit/s)
4 KiB blocks: 11847.7 IO/s, 46.3 MiB/s (388.2 Mbit/s)
8 KiB blocks: 11534.3 IO/s, 90.1 MiB/s (755.9 Mbit/s)
16 KiB blocks: 7303.0 IO/s, 114.1 MiB/s (957.2 Mbit/s)
32 KiB blocks: 4129.3 IO/s, 129.0 MiB/s ( 1.1 Gbit/s)
64 KiB blocks: 2250.5 IO/s, 140.7 MiB/s ( 1.2 Gbit/s)
128 KiB blocks: 1198.1 IO/s, 149.8 MiB/s ( 1.3 Gbit/s)
256 KiB blocks: 759.2 IO/s, 189.8 MiB/s ( 1.6 Gbit/s)
512 KiB blocks: 457.2 IO/s, 228.6 MiB/s ( 1.9 Gbit/s)
1 MiB blocks: 255.7 IO/s, 255.7 MiB/s ( 2.1 Gbit/s)
2 MiB blocks: 136.3 IO/s, 272.6 MiB/s ( 2.3 Gbit/s)
4 MiB blocks: 71.3 IO/s, 285.3 MiB/s ( 2.4 Gbit/s)
8 MiB blocks: 36.4 IO/s, 291.5 MiB/s ( 2.4 Gbit/s)
16 MiB blocks: 18.9 IO/s, 303.0 MiB/s ( 2.5 Gbit/s)
Just for the record I bechmarked how change the performance of the MD raid1 on the two HDD.
# python iops-2011-02-11 /dev/sdb3
/dev/sdb3, 2.00 TB, 32 threads:
512 B blocks: 38.3 IO/s, 19.1 KiB/s (156.8 kbit/s)
1 KiB blocks: 37.2 IO/s, 37.2 KiB/s (304.6 kbit/s)
2 KiB blocks: 48.9 IO/s, 97.7 KiB/s (800.7 kbit/s)
4 KiB blocks: 44.9 IO/s, 179.6 KiB/s ( 1.5 Mbit/s)
8 KiB blocks: 37.4 IO/s, 299.1 KiB/s ( 2.5 Mbit/s)
16 KiB blocks: 38.2 IO/s, 611.5 KiB/s ( 5.0 Mbit/s)
32 KiB blocks: 32.0 IO/s, 1.0 MiB/s ( 8.4 Mbit/s)
64 KiB blocks: 23.4 IO/s, 1.5 MiB/s ( 12.3 Mbit/s)
# python iops-2011-02-11 /dev/sda3
/dev/sda3, 2.00 TB, 32 threads:
512 B blocks: 41.8 IO/s, 20.9 KiB/s (171.1 kbit/s)
1 KiB blocks: 41.6 IO/s, 41.6 KiB/s (340.5 kbit/s)
2 KiB blocks: 42.1 IO/s, 84.3 KiB/s (690.4 kbit/s)
4 KiB blocks: 42.3 IO/s, 169.3 KiB/s ( 1.4 Mbit/s)
8 KiB blocks: 34.5 IO/s, 276.3 KiB/s ( 2.3 Mbit/s)
16 KiB blocks: 43.1 IO/s, 689.8 KiB/s ( 5.7 Mbit/s)
32 KiB blocks: 36.0 IO/s, 1.1 MiB/s ( 9.4 Mbit/s)
64 KiB blocks: 34.6 IO/s, 2.2 MiB/s ( 18.1 Mbit/s)
# python iops-2011-02-11 /dev/md2
/dev/md2, 2.00 TB, 32 threads:
512 B blocks: 63.6 IO/s, 31.8 KiB/s (260.6 kbit/s)
1 KiB blocks: 48.1 IO/s, 48.1 KiB/s (394.0 kbit/s)
2 KiB blocks: 57.8 IO/s, 115.6 KiB/s (947.2 kbit/s)
4 KiB blocks: 37.7 IO/s, 150.6 KiB/s ( 1.2 Mbit/s)
8 KiB blocks: 54.4 IO/s, 435.1 KiB/s ( 3.6 Mbit/s)
16 KiB blocks: 48.1 IO/s, 768.8 KiB/s ( 6.3 Mbit/s)
32 KiB blocks: 40.6 IO/s, 1.3 MiB/s ( 10.6 Mbit/s)
64 KiB blocks: 44.5 IO/s, 2.8 MiB/s ( 23.3 Mbit/s)
MD RAID1 of two HDD gives you nearly twice as big perfomance of each individual disc. When you combine SSD and HDD, then the resulting raid will have the same performance as single SSD. The additional HDD will give you redundancy, and will not help you in performance, but on the other hand it will not be bottleneck neither.
For write tests I used FIO.
Here are raw data:
HDD
]# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/mnt/new/test.block --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
test: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.0.9
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/923K /s] [0 /230 iops] [eta 00m:02s]
test: (groupid=0, jobs=1): err= 0: pid=21855: Thu Aug 27 06:55:32 2015
write: io=4096.0MB, bw=306738 B/s, iops=74 , runt=14002034msec
cpu : usr=0.10%, sys=0.66%, ctx=873849, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=299KB/s, minb=299KB/s, maxb=299KB/s, mint=14002034msec, maxt=14002034msec
Disk stats (read/write):
sda: ios=1459/1425151, merge=1013/1295320, ticks=13140546/1152434770, in_queue=1166246234, util=100.00%
SSD
# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/tmp/a/test.block --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
test: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.0.9
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/71460K /s] [0 /17.9K iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=25103: Thu Aug 27 09:41:47 2015
write: io=4096.0MB, bw=68711KB/s, iops=17177 , runt= 61043msec
cpu : usr=8.94%, sys=88.57%, ctx=5348, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=68710KB/s, minb=68710KB/s, maxb=68710KB/s, mint=61043msec, maxt=61043msec
Disk stats (read/write):
sdb: ios=0/1047843, merge=0/35459, ticks=0/135364, in_queue=133722, util=97.71%
So for HDD I got iops=74 and for SSD iops=17177. Which is again huge difference. Now I run the test suite on MD RAID1, which I created in fist part of this blogpost. Raw data:
# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/tmp/b/test.block --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
test: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.0.9
Starting 1 process
Jobs: 1 (f=1): [w] [99.9% done] [0K/12859K /s] [0 /3214 iops] [eta 00m:06s]
test: (groupid=0, jobs=1): err= 0: pid=25535: Thu Aug 27 11:20:56 2015
write: io=4096.0MB, bw=858435 B/s, iops=209 , runt=5003250msec
cpu : usr=0.12%, sys=1.42%, ctx=158288, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=838KB/s, minb=838KB/s, maxb=838KB/s, mint=5003250msec, maxt=5003250msec
Disk stats (read/write):
md0: ios=14/1158475, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=7/1105716, aggrmerge=0/54041, aggrticks=10/210009015, aggrin_queue=210007845, aggrutil=99.28%
sdb: ios=3/1061184, merge=0/98573, ticks=1/225035, in_queue=222790, util=1.47%
loop0: ios=11/1150248, merge=0/9509, ticks=19/419792995, in_queue=419792900, util=99.28%
The result for RAID1 is iops=209. This is close to HDD performance (74 iops) .
Performance in writing sucks. Still 3 times better then just HDD alone, but still order of magnitude slower than SSD alone.
The result is definitelly not black and white. If I replace one HDD in RAID1 with SSD, then I will get between 3 and 100 times better performance in reading. If I replace the second HDD with SDD then I will get additional 2 times increase of performance. So here replacing one disk is huge benefit. I will keep rendunance and the boost of performance is huge. For the writes the boost is "only" three times bigger. So it really matter what your usage is. If you write a lot then it can make sense to replace both disk with SSD disks. This is not my case, I usually just read a lot and write only few data. Therefore I will replace one of my disks in array with SSD now. And the second sometime later.