Stop K/V listings at the end of their range #111

koolkhel · 2026-10-27T10:26:30+03:00

koolkhel commented

2026-10-27 10:26:30 +03:00

Description

K/V blocks emptied by deletes are never freed (the "FIXME We may want to merge blocks" in kv_db.cpp). A listing that runs past the end of its range walks every empty block up to the next existing key, re-checking each block's version on an OSD.

The NFS proxy runs into this all the time:

the emptiness check of RMDIR lists the removed directory's entries;
RENAME over a directory does the same for the target;
READDIR lists to the end of a directory.

For a new directory, these listings run past the end of the directory entries, through the emptied blocks of every deleted entry and inode, up to the next live key. So RMDIR gets slower with every file ever deleted.

We hit it on a test cluster of 5 VMs and 25 OSDs. After a few metadata-heavy benchmark runs, RMDIR took 190–410 ms instead of 4–5 ms. In the test added here, 50 RMDIRs take 262 ms on a new filesystem and 6954 ms after 10,000 K/V keys were created and deleted.

1. Add an end key to K/V listings

vitastorkv_dbw_t::list_start(start, end) stops at the first key >= end, and does not read any block beyond it.
list_start(start) is unchanged, with the same symbol.
vitastor-kv list|dump|dumpjson <start> <end> now pass the end argument they already parsed.
Test: tests/test_kv_list_bounds.sh, which needs no mount.

2. Stop NFS prefix scans at the end of the prefix

RMDIR, RENAME over a directory, READDIR and the two defrag scans pass the end of their key prefix.
This also fixes RENAME of a directory over an empty directory failing with ENOTEMPTY. The check took the first key after the prefix, which belongs to another directory or inode, for an entry.
Test: tests/test_nfs_list_bounds.sh. It times 50 RMDIRs before and after creating and deleting 10,000 keys right after the newest directories' entries, and fails above twice. It also renames a directory over an empty one.

Not included: freeing or merging empty blocks, so the image still grows with churn. A listing inside a range whose keys were mostly deleted still walks its empty blocks.

Testing:

tests/test_kv_list_bounds.sh, test_kv_stress.sh and IMMEDIATE_COMMIT=1 test_kv_stress.sh pass, on Ubuntu 24.04.
test_nfs_list_bounds.sh, test_nfs.sh, test_nfs_long_readdir.sh and test_kv_list_bounds.sh pass on a Debian-12-based VM, built in node:16-bookworm. That build had no ASAN: the ASAN build crashed at start under the tests' ASAN_OPTIONS on that VM.
test_nfs_list_bounds.sh fails on master at the RMDIR check: 262 ms before, 6954 ms after.
.gitea/workflows/test.yml is regenerated with tests-to-yaml.pl.
The runs above were on b1fd7548. The series rebases onto current master without changes, master has not touched the code since, and vitastor-kv and vitastor-nfs build.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md

### Description K/V blocks emptied by deletes are never freed (the "FIXME We may want to merge blocks" in `kv_db.cpp`). A listing that runs past the end of its range walks every empty block up to the next existing key, re-checking each block's version on an OSD. The NFS proxy runs into this all the time: - the emptiness check of RMDIR lists the removed directory's entries; - RENAME over a directory does the same for the target; - READDIR lists to the end of a directory. For a new directory, these listings run past the end of the directory entries, through the emptied blocks of every deleted entry and inode, up to the next live key. So RMDIR gets slower with every file ever deleted. We hit it on a test cluster of 5 VMs and 25 OSDs. After a few metadata-heavy benchmark runs, RMDIR took 190–410 ms instead of 4–5 ms. In the test added here, 50 RMDIRs take 262 ms on a new filesystem and 6954 ms after 10,000 K/V keys were created and deleted. **1. Add an end key to K/V listings** - `vitastorkv_dbw_t::list_start(start, end)` stops at the first key `>= end`, and does not read any block beyond it. - `list_start(start)` is unchanged, with the same symbol. - `vitastor-kv list|dump|dumpjson <start> <end>` now pass the `end` argument they already parsed. - Test: `tests/test_kv_list_bounds.sh`, which needs no mount. **2. Stop NFS prefix scans at the end of the prefix** - RMDIR, RENAME over a directory, READDIR and the two defrag scans pass the end of their key prefix. - This also fixes RENAME of a directory over an empty directory failing with ENOTEMPTY. The check took the first key after the prefix, which belongs to another directory or inode, for an entry. - Test: `tests/test_nfs_list_bounds.sh`. It times 50 RMDIRs before and after creating and deleting 10,000 keys right after the newest directories' entries, and fails above twice. It also renames a directory over an empty one. **Not included:** freeing or merging empty blocks, so the image still grows with churn. A listing inside a range whose keys were mostly deleted still walks its empty blocks. **Testing:** - `tests/test_kv_list_bounds.sh`, `test_kv_stress.sh` and `IMMEDIATE_COMMIT=1 test_kv_stress.sh` pass, on Ubuntu 24.04. - `test_nfs_list_bounds.sh`, `test_nfs.sh`, `test_nfs_long_readdir.sh` and `test_kv_list_bounds.sh` pass on a Debian-12-based VM, built in `node:16-bookworm`. That build had no ASAN: the ASAN build crashed at start under the tests' `ASAN_OPTIONS` on that VM. - `test_nfs_list_bounds.sh` fails on master at the RMDIR check: 262 ms before, 6954 ms after. - `.gitea/workflows/test.yml` is regenerated with `tests-to-yaml.pl`. - The runs above were on b1fd7548. The series rebases onto current master without changes, master has not touched the code since, and `vitastor-kv` and `vitastor-nfs` build. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md

koolkhel added 2 commits 2026-10-27 10:26:31 +03:00

Add an end key to K/V listings bf89e7e0d2

A listing goes on from block to block until it finds a key, and blocks
emptied by deletes are never freed. So a listing that runs past the end
of its range reads every empty block up to the next existing key, each
with a version check on an OSD. After many creates and deletes, this
made a single RMDIR in the NFS proxy take hundreds of milliseconds.

list_start(start, end) stops at the first key >= end and does not read
blocks beyond it. vitastor-kv list/dump/dumpjson now pass their end
argument.

Stop NFS prefix scans at the end of the prefix c3303a4122

The emptiness checks of RMDIR and of RENAME over a directory, READDIR
and the defrag scans list a key prefix. They now stop at the end of the
prefix instead of running on to the next existing key through blocks
emptied by deletes, which made RMDIR slower with every deleted file.

This also fixes RENAME of a directory over an empty directory failing
with ENOTEMPTY: its check took the first key after the prefix, which
belongs to another directory or inode, for an entry.

vitalif commented

2026-10-28 00:10:00 +03:00

Замержил без этих тестов, но в test_nfs добавил тест на переименование поверх пустой директории

vitalif closed this pull request

2026-10-28 00:10:00 +03:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vitalif/vitastor#111

Allow edits from maintainers