Compressibility detection in btrfs and zstd

Based on linux kernel 7.2.4.

For btrfs + zstd, in fact, there are several measures to "avoid ineffective compression". There's indeed some redundant logic that could be removed, but for now, the results seem acceptable.

From top to bottom:

Check flags and attrs

In ./fs/btrfs/btrfs_inode.h:474:btrfs_inode_can_compress, it checks no-cow and no-sum flags of inode, which allow chattr +m xxx to disable compression for file and dir manually.

// ./fs/btrfs/inode.c:729:inode_need_compress
if (unlikely(!btrfs_inode_can_compress(inode))) {
    DEBUG_WARN("BTRFS: unexpected compression for ino %llu", btrfs_ino(inode));
    return 0;
}

Then, other checking logics：

// ... check_inline
// ... defrag_compress
if (end + 1 - start <= fs_info->sectorsize &&
    (!check_inline || (start > 0 || end + 1 < inode->disk_i_size)))
    return 0;
// ... inode->flags & BTRFS_INODE_NOCOMPRESS

Heuristic detection

In the end of inode_need_compress function, it calls ./fs/btrfs/compression.c:1560:btrfs_compress_heuristic.

The specific implementation is complex, so I won't paste code here. In summary, it samples the byte distribution.

Sampling rules ./fs/btrfs/compression.c:620-643:

For high-entropy data like h264 video, common results are: byte sets approaches 256, with large core byte set.

Fallback after compression

After heuristic detection, ./fs/btrfs/inode.c:947 calls btrfs_compress_bio() to compress then compare with origin size, if it is even bigger, mark as NOCOMPRESS.

if (total_compressed + blocksize > total_in)
    goto mark_incompressible;

Inside zstd

The zstd is trimmed and adapted, in ./lib/zstd of the kernel.

It works like this, for 128 KiB block:

So zstd avoids keeping a bad compressed result, unlike btrfs avoids starting work.

At the start of a block (./lib/zstd/compress/zstd_compress.c:3187):

if (srcSize < MIN_CBLOCK_SIZE + ZSTD_blockHeaderSize + 1 + 1) {
    ...
    return ZSTDbss_noCompress;
}

TODO: