kokizzu/#btrfs benchmark for daily used desktop OS

Compression of file system in most cases are nonsense nowadays. It won't save any space or increase IO performance. Because normal files in PC/laptop are all compressed already: (image, pdf, media, xls/doc, hdf, ccache, read-only database ...). I think only super large build directory benifits from file system compression. The git repo itself actually has compression already. Thus if the .git is way larger than source tree, it won't make too much sense to compress the entire source dir. Fortunately, the btrfs is smart enough to determine which files are worth compressing.

For simple benchmark, I used several ffmpeg and libboost build dirs as example: ffmpeg

LZO
Processed 1541 files, 20427 regular extents (20427 refs), 542 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       47%      1.2G         2.6G         2.6G       
none       100%      306M         306M         306M       
lzo         41%      970M         2.3G         2.3G       
ZLIB
Processed 1541 files, 20399 regular extents (20399 refs), 632 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       36%      964M         2.6G         2.6G       
none       100%      294M         294M         294M       
zlib        28%      670M         2.3G         2.3G       
ZSTD:1
Processed 1541 files, 20399 regular extents (20399 refs), 632 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       34%      917M         2.6G         2.6G       
none       100%      294M         294M         294M       
zstd        26%      622M         2.3G         2.3G       
ZSTD:3
Processed 1541 files, 20399 regular extents (20399 refs), 632 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       34%      917M         2.6G         2.6G       
none       100%      294M         294M         294M       
zstd        26%      622M         2.3G         2.3G  

libboots:

LZO
Processed 148537 files, 60482 regular extents (60482 refs), 92581 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       37%      647M         1.6G         1.6G       
none       100%      4.3M         4.3M         4.3M       
lzo         37%      643M         1.6G         1.6G       

ZLIB
Processed 148537 files, 59862 regular extents (59862 refs), 93201 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       24%      417M         1.6G         1.6G       
none       100%      228K         228K         228K       
zlib        24%      416M         1.6G         1.6G       

ZSTD:1
Processed 148537 files, 59862 regular extents (59862 refs), 93201 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       22%      392M         1.6G         1.6G       
none       100%      228K         228K         228K       
zstd        22%      392M         1.6G         1.6G       

ZSTD:3
Processed 148537 files, 59862 regular extents (59862 refs), 93201 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       22%      392M         1.6G         1.6G       
none       100%      228K         228K         228K       
zstd        22%      392M         1.6G         1.6G

So ZSTD compress the data pretty well. LZO provide less compression but still decent ~50%. I also noticed the ZLIB impose huge pressure on CPU and very slow. So in the following tests I ignored it.

I also tested the speed or the write/read via simple cp -a <source> <dest> on a Intel Core i7-9750H machine.

write test to file system on a same usb SSD (files are already read into memory):

We can see the speed is almost the same. This indicates the penality on write performance is quite huge when zstd is on. Because it actually only write 1/3 of the data.

read test from above file system to a ram disk, system cache is dropped before each copy command.

peak speed is obtained via iostat. With zstd:1 the I/O is slower but not much comparing to the write operation. However, the big compression ratio compensate this and yield the fastest read speed. It is 130% faster than non-compressed condition, since the size is 1/3 of the original.

I did not test other combination or zlib, since the zlib is well-known slower than zstd:1

However, this test has a flaw that I do not sync after cp, thus the data may not finished the write-operation to the disk at all after cp command finished. In following more thourough tests, I use cp xxx && sync as write speed test.

I then test it on an almost fresh nvme SSD:

/dev/nvme0n1:512110190592B:nvme:512:512:gpt:INTEL SSDPEKNW512G8:;
6:296470183936B:350157275135B:53687091200B:xfs:linux-nvme-data:;
7:350157275136B:377000820735B:26843545600B:btrfs:btrfs test:;
8:377000820736B:403844366335B:26843545600B:ext4:ext4test:;

• write test: cp -a <src> <dest> && sync
• before write test I delete the old items in and fstrim the mount point.
• tar read test: tar -c <dest>/<data> | pv -f /dev/null
• cp read test: cp -a <dest> /tmp/<test root>
• before each read test we remount the partition to drop the system cache

For a 2.7GiB ffmpeg build dir containing 1.5k files, the speeds are all > 500MB/s which is close to the single large file copy speed (~600-700MB/s). I test it several times with random shuffled test orders. The typical results are showing in following table. We can see lzo always has the fastest write speed, though the compression ratio is only 47%. The ZSTD:1 has much higher ratio of 34% however it seems to pay a big penality. Unlike lzo/zstd, ZLIB shows very high CPU usage and very slow in the prelimiary tests. So I did not include it in the repeated tests. The ext4 and xfs speeds yield from this test are kind of unstable, some times to drop to 100-200MB/s. But in most cases they are around 500-700MB/s

For a 2.0GiB boost build dir containing 149k files, the I/O speeds are far slower around 200MB/s in all cases. The btrfs.lzo and non-compressed btrfs are only slightly faster than ext4 and xfs during writing. But the zstd:1 is way slower than the rest, only 60-70% speed of non-compressed btrfs. The read speed of all btrfs tests are 20% faster than ext4 and xfs. This may indicate btrfs handle small files better. The compression ratio in this case does not affect the I/O much because the bottle neck is the ammount of files instead of size, though LZO and ZSTD:1 provide 37% and 22% ratios respectively.

An interesting observation is that tar read speed and the cp read speed are all very different in both 1.5k/2.7GiB and 148k/2GiB tests. However, the effects are in a different direction. In 1.5k/2.7GiB case the LZO/ZSTD:1 show ~50% performance drop in tar-read comparing to cp tests of themselves and the tar tests of other file systems. This indicates the file reading methods of tar and cp have some fundamental difference and worth further study.

In 148k/2GiB case the compressed file systems do not show any performance drop in tar tests. However, tar tests of btrfs are ~25% faster than cp tests. The possible reason can be that large number of file creation operations in RAM disk is still costy.

/dev/sdc:5000981077504B:scsi:512:4096:gpt:Seagate One Touch HDD:;
1:17408B:107374182399B:107374164992B:free;
1:107374182400B:134217727999B:26843545600B:ext4:speed test ext4:;
2:134217728000B:161061273599B:26843545600B:xfs:speed test xfs:;
3:161061273600B:187904819199B:26843545600B:btrfs:speed test btrfs:;

./fs-user-benchmark.py --no-fstrim --source ~/.cget/cache/builds/ffmpeg -t '{"root_path":"/media/dracula/hddxfstest/test", "postfix":"0"}' -t '{"root_path":"/media/dracula/hddext4test/test", "postfix":"0"}' -t '{"root_path":"/media/dracula/hddxfstest/test", "postfix":"1"}' -t '{"root_path":"/media/dracula/hddext4test/test", "postfix":"1"}' -t '{"root_path":"/media/dracula/hddbtrfstest/test", "compress_type":"none"}' -t '{"root_path":"/media/dracula/hddbtrfstest/test", "compress_type":"lzo"}' -t '{"root_path":"/media/dracula/hddbtrfstest/test", "compress_type":"zstd:1"}' -t '{"root_path":"/media/dracula/hddbtrfstest/test", "compress_type":"zstd:3"}'

Here is typical results from 1.4k/2.7GB build folder. btrfs is faster than xfs and ext4 in general. ext4 is the slowest. zstd and lzo has huge advantage than none compressed btrfs. Because here the bottle neck is the disk I/O. Smaller data matter a lot. ZSTD:3 impose some pressure on CPU but does not give much advantage than ZSTD:1.

zstd:1 still has huge penalty on I/O speed on modern PC system. The compression is way slower than decompression. My tests is done on relatively slow SSD thus the penalty is not that obvious. I belive if we use nvme SSD, the impact will be much bigger.

In other hand, if the drive is much slower such as HDD, usually only have less than 100MB/s. Then the benifit of smaller data size is far more benificial than SSDs. Indeed, the degree of benifits completely depends on the information density of the data. If zstd can yield reasonable compression ratio (<50%) it might worth to turn on compression on slow storage devices.

LZO can give mild compression around 50% for most of my build dir. It requires way less CPU than ZSTD:1. Thus on fast internal SSDs, LZO is preferred.