in our case, we will do checkpoint by period, in some case, with period checkpoint, the L0 file is small, when L0 file number reach 4, it will trigger compaction and pick up L1 file together. which made huge write amplification。
in this case, maybe we can compaction L0 file to L0 instead of L1 ,when total size of L0 files is smaller than
RocksDB can perform intra-L0 compaction in certain cases. The logic can be found in LevelCompactionBuilder::PickIntraL0Compaction(). First of all, the number of L0 files need to be greater than or equal to level0_file_num_compaction_trigger + 2. More checking in FindIntraL0Compaction().
Reply 2 (author):
PickIntraL0Compaction will be perform only when
PickFileToCompact fail, this only happen when all L1 file are compacting. in common case, when L0 file reach
level0_file_num_compaction_trigger + 2, it will be compact with some L1 file, if L0 are too small , it will make great write amplification. maybe we can add some check , if total size of L0 is too small (each L0 file may be very small if we do checkpoint frequent ) , we should perfrom
Reply 3 (author):
I had made a patch in my project, which enable L0 innner compaction when total size of L0 files are small than
target_file_size_base , and it work well in online service.
when we do L0 inner compaction when total size of L0 is too small , we can reduce a lot write amplification compare to compact L0 with L1. maybe i can push my patch back to rocksdb.