Store v2: Wait for fsyncs submitted by other operations before completing a sync #107

Open
koolkhel wants to merge 1 commits from koolkhel/vitastor:v2-chaos-105267292 into master
Contributor

Description

do_sync() returned at once when nothing was left unsynced, but these counters are reset when an fsync is submitted, not when it completes. So a stabilize or rollback whose metadata block was written before another operation submitted its fsync completed before that fsync did, and a power outage could undo it: test_blockstore_chaos --seed 105267292 sees a rolled back version come back.

Count fsyncs in flight and wait for them as well. Comparing fsynced_lsn with completed_lsn instead isn't enough: an unfinished earlier write keeps completed_lsn below the entries of the operation being synced.

Tested:

  • test_blockstore: new test_rollback_waits_for_inflight_fsync, with and without an unfinished earlier write. Both fail without the fix: the rollback completes while the fsync is held.
  • test_blockstore_chaos --seed 105267292, --seeds 1-960, and --config 3/4/5 × seeds 100001-101000 pass.
  • With #124 on top, where some rollbacks become BS_OP_ROLLBACK_WRITE: --config 5 --seed 100085 and the same 3000 runs pass.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description `do_sync()` returned at once when nothing was left unsynced, but these counters are reset when an fsync is submitted, not when it completes. So a stabilize or rollback whose metadata block was written before another operation submitted its fsync completed before that fsync did, and a power outage could undo it: `test_blockstore_chaos --seed 105267292` sees a rolled back version come back. Count fsyncs in flight and wait for them as well. Comparing `fsynced_lsn` with `completed_lsn` instead isn't enough: an unfinished earlier write keeps `completed_lsn` below the entries of the operation being synced. Tested: - `test_blockstore`: new `test_rollback_waits_for_inflight_fsync`, with and without an unfinished earlier write. Both fail without the fix: the rollback completes while the fsync is held. - `test_blockstore_chaos --seed 105267292`, `--seeds 1-960`, and `--config 3/4/5` × seeds 100001-101000 pass. - With #124 on top, where some rollbacks become BS_OP_ROLLBACK_WRITE: `--config 5 --seed 100085` and the same 3000 runs pass. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-10-26 21:02:34 +03:00
do_sync() returned at once when nothing was left unsynced, but these
counters are reset when an fsync is submitted, not when it completes. So
a stabilize or rollback whose metadata block was written before another
operation submitted its fsync completed before that fsync did, and a
power outage could undo it: test_blockstore_chaos --seed 105267292 sees
a rolled back version come back.

Wait until everything completed is fsynced instead. A waiting sync still
counts as unfinished, so that the next one doesn't overtake it and have
its pending fsyncs reset.
vitalif reviewed 2026-10-27 00:30:40 +03:00
@@ -107,6 +107,16 @@ int blockstore_impl_t::do_sync(blockstore_op_t *op, int base_state)
}
if (!has_unsynced())
{
if ((!dsk.disable_meta_fsync || !dsk.disable_journal_fsync) &&
Owner

тут точно обе правки нужны? выглядит как будто то ли два разных кейса поправлено, то ли один, но двумя способами. верхней правки не хватит ли?

тут точно обе правки нужны? выглядит как будто то ли два разных кейса поправлено, то ли один, но двумя способами. верхней правки не хватит ли?
koolkhel added 1 commit 2026-11-18 12:52:47 +03:00
When nothing is left to sync, do_sync() waits until fsynced_lsn reaches
completed_lsn. But completed_lsn only advances when all lower LSNs
complete. While an earlier write is unfinished, a rollback or stabilize
whose metadata write is only covered by an fsync that another operation
has just submitted sees fsynced_lsn == completed_lsn, and is
acknowledged before that fsync completes. A power outage can then undo
it.

Count fsyncs in flight and make do_sync() wait for them as well.

Found by test_blockstore_chaos --config 5 --seed 100085 with
BS_OP_ROLLBACK_WRITE support applied.
Author
Contributor

One more gap in the same sync, found by stress-testing #124 on top of this PR: test_blockstore_chaos --config 5 --seed 100085 still failed there.

The LSN check here compares fsynced_lsn with completed_lsn, but completed_lsn only advances when all lower LSNs complete. While an earlier write is still unfinished, a rollback or stabilize whose metadata write is covered only by an fsync another operation has just submitted sees fsynced_lsn == completed_lsn. It is acknowledged before that fsync completes, and a power outage then undoes it.

312a457b counts fsyncs in flight (submit_fsyncs() adds them, their completion subtracts) and makes do_sync() wait for them when nothing is left to sync, before the LSN check. test_rollback_waits_for_inflight_fsync gets a variant with an unfinished earlier write, which fails without the fix.

Tested: test_blockstore, test_heap, test_blockstore_chaos --seeds 1-960 and configs 3-5 × seeds 100001-101000 all pass. With #124 on top, seed 100085 passes.

One more gap in the same sync, found by stress-testing #124 on top of this PR: `test_blockstore_chaos --config 5 --seed 100085` still failed there. The LSN check here compares `fsynced_lsn` with `completed_lsn`, but `completed_lsn` only advances when all lower LSNs complete. While an earlier write is still unfinished, a rollback or stabilize whose metadata write is covered only by an fsync another operation has just submitted sees `fsynced_lsn == completed_lsn`. It is acknowledged before that fsync completes, and a power outage then undoes it. 312a457b counts fsyncs in flight (`submit_fsyncs()` adds them, their completion subtracts) and makes `do_sync()` wait for them when nothing is left to sync, before the LSN check. `test_rollback_waits_for_inflight_fsync` gets a variant with an unfinished earlier write, which fails without the fix. Tested: test_blockstore, test_heap, `test_blockstore_chaos --seeds 1-960` and configs 3-5 × seeds 100001-101000 all pass. With #124 on top, seed 100085 passes.
koolkhel force-pushed v2-chaos-105267292 from 312a457bc5 to b950937bac 2026-11-18 14:20:01 +03:00 Compare
Author
Contributor

тут точно обе правки нужны? выглядит как будто то ли два разных кейса поправлено, то ли один, но двумя способами. верхней правки не хватит ли?

It was one fix in two parts, and it was incomplete. I tested each edit separately:

  • has_unfinished_sync for a waiting sync, alone, changes nothing. Before the LSN wait a sync could only wait for an SQE, and that breaks out of the loop before this line. test_blockstore_chaos --seed 105267292 still fails.
  • The LSN wait alone needs the first edit. A sync parked in WAIT_META_FSYNC lets a later sync overtake it and submit its fsyncs. When the first one wakes, has_unfinished_sync is set, the later sync's pending_ops is reset in loop() while its fsync is in flight, and the fsync callback hits assert(wait_count >= 0). Seed 105267292 aborts there.
  • Both together still miss the case from my last comment: an unfinished earlier write keeps completed_lsn low, and a rollback is acked before the fsync that covers it.

The fsync counter covers all three by itself. When nothing is unsynced and no fsync is in flight, every completed write is already covered by a finished fsync, so the LSN wait never has anything left to wait for. The counter's wait has no wait state, so the first edit isn't needed either. I've reduced the PR to that: b950937b, one commit with the counter and both test variants, in place of 933f2302 and 312a457b. test_blockstore and seed 105267292 pass, and so does --config 5 --seed 100085 with #124 on top. The same change with the first edit still in, where it is unreachable, passed --seeds 1-960 and configs 3-5 × seeds 100001-101000. Those sweeps are running again on the squashed commit, and I'll add the results to the description.

One property of the counter: a sync waits for any fsync in flight, so under traffic that keeps submitting a new fsync before the last one completes, it can wait for a long time. The wait for the flusher's syncing_buffer just above it behaves the same way. If that matters, an op could record the number of the last fsync batch submitted and wait only for batches up to it. I can do that instead.

> тут точно обе правки нужны? выглядит как будто то ли два разных кейса поправлено, то ли один, но двумя способами. верхней правки не хватит ли? It was one fix in two parts, and it was incomplete. I tested each edit separately: - `has_unfinished_sync` for a waiting sync, alone, changes nothing. Before the LSN wait a sync could only wait for an SQE, and that breaks out of the loop before this line. `test_blockstore_chaos --seed 105267292` still fails. - The LSN wait alone needs the first edit. A sync parked in WAIT_META_FSYNC lets a later sync overtake it and submit its fsyncs. When the first one wakes, `has_unfinished_sync` is set, the later sync's `pending_ops` is reset in `loop()` while its fsync is in flight, and the fsync callback hits `assert(wait_count >= 0)`. Seed 105267292 aborts there. - Both together still miss the case from my last comment: an unfinished earlier write keeps `completed_lsn` low, and a rollback is acked before the fsync that covers it. The fsync counter covers all three by itself. When nothing is unsynced and no fsync is in flight, every completed write is already covered by a finished fsync, so the LSN wait never has anything left to wait for. The counter's wait has no wait state, so the first edit isn't needed either. I've reduced the PR to that: b950937b, one commit with the counter and both test variants, in place of 933f2302 and 312a457b. test_blockstore and seed 105267292 pass, and so does `--config 5 --seed 100085` with #124 on top. The same change with the first edit still in, where it is unreachable, passed `--seeds 1-960` and configs 3-5 × seeds 100001-101000. Those sweeps are running again on the squashed commit, and I'll add the results to the description. One property of the counter: a sync waits for any fsync in flight, so under traffic that keeps submitting a new fsync before the last one completes, it can wait for a long time. The wait for the flusher's `syncing_buffer` just above it behaves the same way. If that matters, an op could record the number of the last fsync batch submitted and wait only for batches up to it. I can do that instead.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u v2-chaos-105267292:koolkhel-v2-chaos-105267292
git checkout koolkhel-v2-chaos-105267292
Sign in to join this conversation.
No Reviewers
No Label
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#107
No description provided.