vitalif/vitastor
5
6
Fork 12
Code Issues 4 Pull Requests 6 Actions Releases 111 Wiki Activity

Store v1: Fix two flusher hangs with deletions #114

Open
koolkhel wants to merge 3 commits from koolkhel/vitastor:v1-lost-flush into master
pull from: koolkhel/vitastor:v1-lost-flush
merge into: vitalif:master
Conversation 1 Commits 3 Files Changed 5 +91 -2
koolkhel commented 2026-10-27 10:26:38 +03:00
Contributor
Copy Link

Description

This is v2 of the patches sent to you on 2026-10-21. Following your review, the flush check of patch 1 runs only with the new --hardened option of test_blockstore_chaos, because it scans all of dirty_db; patch 2's subject names the option. Patches 2 and 3 are unchanged otherwise.

Under sustained load, an OSD on store v1 can stop accepting writes for good: its journal fills up, every write fails with EAGAIN, and the client retries forever, so all I/O touching that OSD hangs until the OSD restarts. We hit it with an NFS server that reuses inode numbers, so the data objects of deleted files are deleted and rewritten from version 1 all the time. The gateway stalled after 4-8 minutes of S3 load. gdb on the stuck OSD showed the full journal pinned by one stable BIG_WRITE entry at used_start, which was not in the flush queue.

1. Check that stable versions get flushed in test_blockstore_chaos --hardened

  • A lost flush shows only as writes refused with EAGAIN, which the test accepts as journal pressure. With the new --hardened option, the test checks after every drain that the newest stable version of every object is being flushed, or queued under a version that is still in dirty_db and not older than it.
  • The check scans all of dirty_db, so it is off by default.
  • With it, --p-delete 30 finds lost flushes on master: --seed 285 --config 21, --seed 112 --config 21, --seed 118 --config 21, --seed 439 --config 18 and 19, all with --hardened --steps 30000 --p-delete 30.

2. Store v1: Do not lose the flush of a write over a deletion

  • The flush queue keeps one version per object, the highest one queued.
  • A write over a deletion asks for the deletion to be flushed. If the flusher is already flushing it, that queues the deletion again.
  • The write is renumbered from version 1 once the deletion is gone, and queueing it is ignored because the queued deletion version is higher. The flusher skips the deletion it can't find, so the write is never flushed and the journal can't be trimmed past it.
  • Fix: a queued version that is no longer in dirty_db is replaced by the one being queued.

3. Store v1: Wake up the flusher when journal sectors are written

  • The flusher puts back entries whose journal sector isn't written yet and stops when nothing else is flushable. Writing the sector moves dirty_start, but nothing woke the flusher, so writes waiting for journal space or for a deletion hung with a full queue of flushable entries.
  • It happens with immediate_commit=all too: --seed 5 --config 12 --steps 30000 --p-delete 30 deadlocks on master.

3cf04530 ("Fix exotic hang on an unflushable deletion") does not cure either case: all the seeds above fail on master, which includes it.

Testing (Ubuntu 24.04, RelWithDebInfo; first on a83a0aed, then rebased onto 3e803e07 without changes):

  • The seeds above fail with commit 1 alone when run with --hardened, pass without it, and pass with the whole series either way.
  • --hardened changes nothing measurable in the test itself: 48 seeds with default weights take 37.85 s without it and 37.87 s with it, because the check runs only at clean restarts and at the end.
  • On 3e803e07: test_blockstore_v1 passes, the seeds pass, and so does a default 200-seed run of test_blockstore_chaos --hardened.
  • make build_tests test, as in .gitea/workflows/test.yml: 15 of 15 tests pass, including test_blockstore_chaos (default run, 200 seeds) and test_pg_chaos.
  • The same three changes (modulo names) run in our product. With them, the S3 load that stalled ran three times on a 5-VM, 25-OSD cluster without a stall and with no EAGAIN in the OSD logs. A sweep of 1,500 seeds on each of the 12 v1 configurations (--steps 30000 --p-delete 30, 18,000 runs) found no lost flush and no deadlock.

Not included: with --p-delete 30 the chaos test also fails on master in other ways that this series doesn't change. A deleted object comes back after a clean restart (--seed 1064 --config 12). There is "BUG: Unexpected dirty_entry ... unstable state during flush" with fsync=1 (--seed 60 --config 21). "big_write journal_entry was allocated over another object" (--seed 144 --config 15). And unexpected content after a power outage (--seed 841 --config 20). A follow-up series fixes them. It builds on this one, so it will come as a separate PR once this one is merged.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description This is v2 of the patches sent to you on 2026-10-21. Following your review, the flush check of patch 1 runs only with the new `--hardened` option of test_blockstore_chaos, because it scans all of dirty_db; patch 2's subject names the option. Patches 2 and 3 are unchanged otherwise. Under sustained load, an OSD on store v1 can stop accepting writes for good: its journal fills up, every write fails with EAGAIN, and the client retries forever, so all I/O touching that OSD hangs until the OSD restarts. We hit it with an NFS server that reuses inode numbers, so the data objects of deleted files are deleted and rewritten from version 1 all the time. The gateway stalled after 4-8 minutes of S3 load. gdb on the stuck OSD showed the full journal pinned by one stable BIG_WRITE entry at used_start, which was not in the flush queue. **1. Check that stable versions get flushed in test_blockstore_chaos --hardened** - A lost flush shows only as writes refused with EAGAIN, which the test accepts as journal pressure. With the new `--hardened` option, the test checks after every drain that the newest stable version of every object is being flushed, or queued under a version that is still in dirty_db and not older than it. - The check scans all of dirty_db, so it is off by default. - With it, `--p-delete 30` finds lost flushes on master: `--seed 285 --config 21`, `--seed 112 --config 21`, `--seed 118 --config 21`, `--seed 439 --config 18` and `19`, all with `--hardened --steps 30000 --p-delete 30`. **2. Store v1: Do not lose the flush of a write over a deletion** - The flush queue keeps one version per object, the highest one queued. - A write over a deletion asks for the deletion to be flushed. If the flusher is already flushing it, that queues the deletion again. - The write is renumbered from version 1 once the deletion is gone, and queueing it is ignored because the queued deletion version is higher. The flusher skips the deletion it can't find, so the write is never flushed and the journal can't be trimmed past it. - Fix: a queued version that is no longer in dirty_db is replaced by the one being queued. **3. Store v1: Wake up the flusher when journal sectors are written** - The flusher puts back entries whose journal sector isn't written yet and stops when nothing else is flushable. Writing the sector moves dirty_start, but nothing woke the flusher, so writes waiting for journal space or for a deletion hung with a full queue of flushable entries. - It happens with immediate_commit=all too: `--seed 5 --config 12 --steps 30000 --p-delete 30` deadlocks on master. 3cf04530 ("Fix exotic hang on an unflushable deletion") does not cure either case: all the seeds above fail on master, which includes it. **Testing** (Ubuntu 24.04, RelWithDebInfo; first on a83a0aed, then rebased onto 3e803e07 without changes): - The seeds above fail with commit 1 alone when run with `--hardened`, pass without it, and pass with the whole series either way. - `--hardened` changes nothing measurable in the test itself: 48 seeds with default weights take 37.85 s without it and 37.87 s with it, because the check runs only at clean restarts and at the end. - On 3e803e07: `test_blockstore_v1` passes, the seeds pass, and so does a default 200-seed run of `test_blockstore_chaos --hardened`. - `make build_tests test`, as in `.gitea/workflows/test.yml`: 15 of 15 tests pass, including `test_blockstore_chaos` (default run, 200 seeds) and `test_pg_chaos`. - The same three changes (modulo names) run in our product. With them, the S3 load that stalled ran three times on a 5-VM, 25-OSD cluster without a stall and with no EAGAIN in the OSD logs. A sweep of 1,500 seeds on each of the 12 v1 configurations (`--steps 30000 --p-delete 30`, 18,000 runs) found no lost flush and no deadlock. **Not included:** with `--p-delete 30` the chaos test also fails on master in other ways that this series doesn't change. A deleted object comes back after a clean restart (`--seed 1064 --config 12`). There is "BUG: Unexpected dirty_entry ... unstable state during flush" with fsync=1 (`--seed 60 --config 21`). "big_write journal_entry was allocated over another object" (`--seed 144 --config 15`). And unexpected content after a power outage (`--seed 841 --config 20`). A follow-up series fixes them. It builds on this one, so it will come as a separate PR once this one is merged. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 3 commits 2026-10-27 10:26:39 +03:00
Check that stable versions get flushed in test_blockstore_chaos --hardened 80ef5486f0 
A store v1 stable version that no flush reaches pins the journal for
good, and the store only shows it as writes refused with EAGAIN, which
the test takes for journal pressure. With --hardened, check after every
drain that the newest stable version of every object is being flushed,
or queued under a version that is still in dirty_db and not older than
it. The check scans all of dirty_db, so it is off by default.
Store v1: Do not lose the flush of a write over a deletion (test_blockstore_chaos --hardened --seed 285 --config 21 --p-delete 30) 98559f4fda 
The flush queue keeps one version per object, the highest one queued.
A write over a deletion asks for the deletion to be flushed, and if the
flusher is already flushing it, that queues the deletion again. Once the
deletion is gone, the write is renumbered from version 1, and queueing it
is ignored because the queued deletion version is higher. The flusher
then skips the deletion it can't find, the write is never flushed, and
the journal can't be trimmed past it: it fills up and every write fails
with EAGAIN until a restart.

Replace a queued version that is no longer in dirty_db with the one
being queued. Objects that are deleted and rewritten over and over run
into this regularly; ours were the data of an NFS server that reuses
inode numbers.
Store v1: Wake up the flusher when journal sectors are written (test_blockstore_chaos --seed 5 --config 12 --p-delete 30) 8ac5cb9510 
The flusher puts back the entries whose journal sector isn't written yet
and stops when it finds nothing else to flush. Writing the sector moves
dirty_start and makes them flushable, but nothing woke the flusher then:
when the writes waiting for journal space or for a deletion were the only
thing left, they hung forever, with the queue full of flushable entries
and a trim requested. Happens with immediate_commit=all too.
vitalif commented 2026-10-28 00:15:20 +03:00
Owner
Copy Link

Мне тут не нравится вот этот фикс

  1. Store v1: Do not lose the flush of a write over a deletion

The write is renumbered from version 1 once the deletion is gone, and queueing it is ignored because the queued deletion version is higher. The flusher skips the deletion it can't find, so the write is never flushed and the journal can't be trimmed past it.

Сфигали в очередь flush добавляется write одновременно с уже находящимся там delete, если write произошёл ПОВЕРХ этого самого delete, и должен его дождаться? Вот это выглядит как баг и выглядит что именно это править надо, а не просто накрывать проверкой if is_gone.

Мне тут не нравится вот этот фикс >2. Store v1: Do not lose the flush of a write over a deletion >The write is renumbered from version 1 once the deletion is gone, and queueing it is ignored because the queued deletion version is higher. The flusher skips the deletion it can't find, so the write is never flushed and the journal can't be trimmed past it. Сфигали в очередь flush добавляется write одновременно с уже находящимся там delete, если write произошёл ПОВЕРХ этого самого delete, и должен его дождаться? Вот это выглядит как баг и выглядит что именно это править надо, а не просто накрывать проверкой if is_gone.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u v1-lost-flush:koolkhel-v1-lost-flush
git checkout koolkhel-v1-lost-flush
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
vitalif
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#114
No description provided.
Delete Branch "koolkhel/vitastor:v1-lost-flush"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?