kokizzu/#btrfs benchmark for daily used desktop OS

In this article I would like to Compare performance of current popular file systems: BTRFS, XFS and EXT4 based on normal daily use cases for developers: no random writing on single file, lots of small files in build dir. What is the best choice? Do the choices vary when using different hardware: SATA SSD, NVME SSD, mechanic HDD?

The BTRFS also come with important features such as compression. However, compression of file system in most cases are nonsense nowadays. It won't save any space or increase IO performance. Because normal files in PC/laptop are all compressed already: (image, pdf, media, xls/doc, hdf, ccache, read-only database ...). I think only super large build directory benifits from file system compression. The git repo itself actually has compression already. Thus if the .git is way larger than source tree, it won't make too much sense to compress the entire source dir. Fortunately, the btrfs is smart enough to determine which files are worth compressing.

The result is rather surprising: the performance varied a lot depends on what kind of hardware I am using and what kind of data I have; the worst choices can be any of those. However, the best choice seems stick.

• write test: cp -a <src> <dest> && sync
• before write test I delete the old items in and fstrim the mount point.
• tar read test: tar -c <dest>/<data> | pv -f /dev/null
• cp read test: cp -a <dest> /tmp/<test root>
• before each read test we remount the partition to drop the system cache

For simple benchmark, I used several ffmpeg and libboost build dirs as example: ffmpeg

LZO
Processed 1541 files, 20427 regular extents (20427 refs), 542 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       47%      1.2G         2.6G         2.6G       
none       100%      306M         306M         306M       
lzo         41%      970M         2.3G         2.3G       
ZLIB
Processed 1541 files, 20399 regular extents (20399 refs), 632 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       36%      964M         2.6G         2.6G       
none       100%      294M         294M         294M       
zlib        28%      670M         2.3G         2.3G       
ZSTD:1
Processed 1541 files, 20399 regular extents (20399 refs), 632 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       34%      917M         2.6G         2.6G       
none       100%      294M         294M         294M       
zstd        26%      622M         2.3G         2.3G       
ZSTD:3
Processed 1541 files, 20399 regular extents (20399 refs), 632 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       34%      917M         2.6G         2.6G       
none       100%      294M         294M         294M       
zstd        26%      622M         2.3G         2.3G  

libboost dir (149k/2GiB) contains lots of small files. The apparent size of du is 1.6GB, but the actual disk size is 2.0GB, implying that there are lots of file smaller than 4kiB:

LZO
Processed 148537 files, 60482 regular extents (60482 refs), 92581 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       37%      647M         1.6G         1.6G       
none       100%      4.3M         4.3M         4.3M       
lzo         37%      643M         1.6G         1.6G       

ZLIB
Processed 148537 files, 59862 regular extents (59862 refs), 93201 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       24%      417M         1.6G         1.6G       
none       100%      228K         228K         228K       
zlib        24%      416M         1.6G         1.6G       

ZSTD:1
Processed 148537 files, 59862 regular extents (59862 refs), 93201 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       22%      392M         1.6G         1.6G       
none       100%      228K         228K         228K       
zstd        22%      392M         1.6G         1.6G       

ZSTD:3
Processed 148537 files, 59862 regular extents (59862 refs), 93201 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       22%      392M         1.6G         1.6G       
none       100%      228K         228K         228K       
zstd        22%      392M         1.6G         1.6G

So ZSTD compress the data pretty well. LZO provide less compression but still decent ~50%. I also noticed the ZLIB impose huge pressure on CPU and very slow. So in the following tests I ignored it.

I also tested the speed or the write/read via simple cp -a <source> <dest> on a Intel Core i7-9750H machine.

write test to file system on a same usb SSD (files are already read into memory):

We can see the speed is almost the same. This indicates the penality on write performance is quite huge when zstd is on. Because it actually only write 1/3 of the data.

read test from above file system to a ram disk, system cache is dropped before each copy command.

peak speed is obtained via iostat. With zstd:1 the I/O is slower but not much comparing to the write operation. However, the big compression ratio compensate this and yield the fastest read speed. It is 130% faster than non-compressed condition, since the size is 1/3 of the original.

I did not test other combination or zlib, since the zlib is well-known slower than zstd:1

However, this test has a flaw that I do not sync after cp, thus the data may not finished the write-operation to the disk at all after cp command finished. In following more thourough tests, I use cp xxx && sync as write speed test.

/dev/sdd:480103981056B:scsi:512:512:gpt:INTEL SS DSC2BW480A4:;
1:17408B:1073741823B:1073724416B:free;
1:1073741824B:27917287423B:26843545600B:ext4:usb ssd test ext4:;
3:27917287424B:54760833023B:26843545600B:btrfs:usb ssd test btrfs:;
5:54760833024B:81604378623B:26843545600B:xfs:usb ssd test xfs:;

This disk has about 300MiB/s sequencial write speed and 500MiB/s sequencial read speed.

Writing 1.4k files in size of 2.7GiB on all file systems gave similar write speed as writing single big file. the non-compressed btrfs has a bit less write and read performance than xfs and ext4. Compressed btrfs give way better performance. zstd:3 is the best with 55% more write speed and 133% more read speed. It is followed by zstd:1 with 35%/133% increase. The LZO has less improvement but still give 43% more write speed and 79% more read speed.

Interestingly, tar read test yield completely different scheme: none compressed btrfs give the best reasult, all compressed btrfs give 30% drop in speed.

Writing 149k/2GB files on all file systems is way slower: ~30% of normal writing speed and ~10% of normal reading speed. XFS seems perform worst when many small files invovled. btrfs in general proivdes higher performance (> 18% write and 21% read improvement over ext4) in this case. Unlike the 1.4k-file test, all 3 compressed methods can yielded best write performance providing 35% more speed over ext4, from time to time. The non-compressed one is also quite similar to the best one.

The read speeds are different though. All 3 compressed ones have almost same reading speed with about 60% increase. But the non-compressed one is a bit slower with only 22% improvement over ext4. And in this case the tar read speed trend is the same as cp read speeds. Though tar has about 20% more speed when using compressed btrfs comparing to the cp command.

I then test it on an almost fresh nvme SSD:

/dev/nvme0n1:512110190592B:nvme:512:512:gpt:INTEL SSDPEKNW512G8:;
6:296470183936B:350157275135B:53687091200B:xfs:linux-nvme-data:;
7:350157275136B:377000820735B:26843545600B:btrfs:btrfs test:;
8:377000820736B:403844366335B:26843545600B:ext4:ext4test:;

For a 2.7GiB ffmpeg build dir containing 1.5k files, the speeds are all > 500MB/s which is close to the single large file copy speed (~600-700MB/s). I test it several times with random shuffled test orders. The typical results are showing in following table. We can see lzo always has the fastest write speed, though the compression ratio is only 47%. The ZSTD:1 has much higher ratio of 34% however it seems to pay a big penality. Unlike lzo/zstd, ZLIB shows very high CPU usage and very slow in the prelimiary tests. So I did not include it in the repeated tests. The ext4 and xfs speeds yield from this test are kind of unstable, some times to drop to 100-200MB/s. But in most cases they are around 500-700MB/s

For a 2.0GiB boost build dir containing 149k files, the I/O speeds are far slower around 200MB/s in all cases. The btrfs.lzo and non-compressed btrfs are only slightly faster than ext4 and xfs during writing. But the zstd:1 is way slower than the rest, only 60-70% speed of non-compressed btrfs. The read speed of all btrfs tests are 20% faster than ext4 and xfs. This may indicate btrfs handle small files better. The compression ratio in this case does not affect the I/O much because the bottle neck is the ammount of files instead of size, though LZO and ZSTD:1 provide 37% and 22% ratios respectively.

An interesting observation is that tar read speed and the cp read speed are all very different in both 1.5k/2.7GiB and 148k/2GiB tests. However, the effects are in a different direction. In 1.5k/2.7GiB case the LZO/ZSTD:1 show ~50% performance drop in tar-read comparing to cp tests of themselves and the tar tests of other file systems. This indicates the file reading methods of tar and cp have some fundamental difference and worth further study.

In 148k/2GiB case the compressed file systems do not show any performance drop in tar tests. However, tar tests of btrfs are ~25% faster than cp tests. The possible reason can be that large number of file creation operations in RAM disk is still costy.

/dev/sdc:5000981077504B:scsi:512:4096:gpt:Seagate One Touch HDD:;
1:17408B:107374182399B:107374164992B:free;
1:107374182400B:134217727999B:26843545600B:ext4:speed test ext4:;
2:134217728000B:161061273599B:26843545600B:xfs:speed test xfs:;
3:161061273600B:187904819199B:26843545600B:btrfs:speed test btrfs:;

./fs-user-benchmark.py --no-fstrim --source ~/.cget/cache/builds/ffmpeg -t '{"root_path":"/media/dracula/hddxfstest/test", "postfix":"0"}' -t '{"root_path":"/media/dracula/hddext4test/test", "postfix":"0"}' -t '{"root_path":"/media/dracula/hddxfstest/test", "postfix":"1"}' -t '{"root_path":"/media/dracula/hddext4test/test", "postfix":"1"}' -t '{"root_path":"/media/dracula/hddbtrfstest/test", "compress_type":"none"}' -t '{"root_path":"/media/dracula/hddbtrfstest/test", "compress_type":"lzo"}' -t '{"root_path":"/media/dracula/hddbtrfstest/test", "compress_type":"zstd:1"}' -t '{"root_path":"/media/dracula/hddbtrfstest/test", "compress_type":"zstd:3"}'

Here is typical results from 1.4k/2.7GB build folder. btrfs is faster than xfs and ext4 in general. ext4 is the slowest. zstd and lzo has huge advantage than none compressed btrfs. Because here the bottle neck is the disk I/O. Smaller data matter a lot. ZSTD:3 impose some pressure on CPU but does not give much advantage than ZSTD:1.

Writing 149k/2GB files to mechanic HDD is extremely slow for xfs and non-compressed btrfs. The speed is as low as 1MB/s for btrfs almost impossible to use. zstd is 100x faster than non-compressed one. ext4 is 5x faster than xfs, 45x faster than btrfs. lzo in this case only have 50% speed of zstd:1. zstd:3 is 10% slower than zstd:1 too.

Reading from it is even slower. btrfs in general is much faser than ext4, the xfs is the slowest only 7MiB/s.

For normal build dirs:

For build dirs contain large number of small files:

• SATA-SSD: I recommend btrfs zstd:3. But any FS seems ok, though XFS is the worst choice.
• NVME-SSD: we should stick to btrfs lzo. It is way faster than the rest. However, others are still usable.
• HDD: I recommend btrfs zstd:1. We should definitely avoid btrfs here, because it is almost not usable at all: 1MiB/s in many-small-files case.

Though the many-small-files case yield very bad performance in general, the SSDs still perform far better than HDD. If I need to work on HDD, I will need to be very careful about where to put small files, definitely not on standard btrfs, though it claims to inline small files^1^.

zstd:1 still has huge penalty on I/O speed on modern PC system. The compression is way slower than decompression. The tests on relatively slow SATA SSD show no obvious because the bottle net is still disk I/O. On NVME SSD the penalty become far sound.

In other hand, if the drive is much slower such as HDD, usually only have less than 100MB/s. Then the benefit of smaller data size is far more beneficial than SSDs. Indeed, the degree of benefits completely depends on the information density of the data. If zstd/lzo can yield reasonable compression ratio (<50%) it might worth to turn on compression on slow storage devices.

LZO can give mild compression around 50% for most of my build dir. It requires way less CPU than ZSTD:1. Thus on fast internal SSDs, LZO is preferred.