Store v1: Don't let the ack of an older big write unblock later small writes #149

Merged
vitalif merged 1 commits from koolkhel/vitastor:blockstore-v1-wait-big-ack into master 2026-11-22 23:06:51 +03:00
Contributor

Description

The same problem as #129, in the immediate_commit=all path.

With immediate_commit=all a small write following an in-flight big write waits in WAIT_BIG until that big write is acked. The "Unblock small writes" loop in blockstore_impl_t::continue_write() (the write ack) unblocked every later WAIT_BIG write of the object, including the ones waiting for a newer big write which was still in flight. Their journal entries could then reach the disk before the newer big write's entry, and after a crash or restart the replay put them on top of the older big write, giving a version that was never written.

The loop now stops at the next big write: the small writes after it are unblocked by its own ack.

It needs three writes to one object in the blockstore at once: big A, big B, small C, with A acked before B. The OSD primary runs one write per object at a time, so as far as I can see from the code this is only possible on a secondary of a replicated pool, with writes left in flight by dropped connections:

  • a secondary write which is already in the blockstore is not cancelled when the connection to the primary drops;
  • BS_OP_LIST does not wait for it and lists an in-flight WRITE_STABLE as stable, so after re-peering a new write to the same object runs next to it;
  • two such drops while A is still in flight give A, B and C together. If that OSD then dies before B is journaled, its replica has C on top of A, and the other replicas have the same version on top of B.

I only checked this part by reading the code, I didn't reproduce it with OSDs. EC pools don't seem to be affected, because there the leftover write is listed as unstable and the next write waits for its rollback or stabilize.

Tested:

  • test_small_write_waits_for_its_own_big_write_imm in test_blockstore_v1: v1 (big), v2 (big) and v3 (small) are submitted at once, the data write of v2 is held. Without the fix v3 completes while v2 is in flight, and after a restart the object comes back as v3 written over v1. With the fix v3 waits and the object comes back as v1. All other test_blockstore_v1 tests pass.
  • test_blockstore_chaos --seeds 1-300 on all v1 configs (12-23): all pass. The chaos test runs one operation per object at a time, so it only shows that nothing regressed.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description The same problem as #129, in the immediate_commit=all path. With immediate_commit=all a small write following an in-flight big write waits in `WAIT_BIG` until that big write is acked. The "Unblock small writes" loop in `blockstore_impl_t::continue_write()` (the write ack) unblocked every later `WAIT_BIG` write of the object, including the ones waiting for a newer big write which was still in flight. Their journal entries could then reach the disk before the newer big write's entry, and after a crash or restart the replay put them on top of the older big write, giving a version that was never written. The loop now stops at the next big write: the small writes after it are unblocked by its own ack. It needs three writes to one object in the blockstore at once: big A, big B, small C, with A acked before B. The OSD primary runs one write per object at a time, so as far as I can see from the code this is only possible on a secondary of a replicated pool, with writes left in flight by dropped connections: - a secondary write which is already in the blockstore is not cancelled when the connection to the primary drops; - `BS_OP_LIST` does not wait for it and lists an in-flight `WRITE_STABLE` as stable, so after re-peering a new write to the same object runs next to it; - two such drops while A is still in flight give A, B and C together. If that OSD then dies before B is journaled, its replica has C on top of A, and the other replicas have the same version on top of B. I only checked this part by reading the code, I didn't reproduce it with OSDs. EC pools don't seem to be affected, because there the leftover write is listed as unstable and the next write waits for its rollback or stabilize. Tested: - `test_small_write_waits_for_its_own_big_write_imm` in `test_blockstore_v1`: v1 (big), v2 (big) and v3 (small) are submitted at once, the data write of v2 is held. Without the fix v3 completes while v2 is in flight, and after a restart the object comes back as v3 written over v1. With the fix v3 waits and the object comes back as v1. All other `test_blockstore_v1` tests pass. - `test_blockstore_chaos --seeds 1-300` on all v1 configs (12-23): all pass. The chaos test runs one operation per object at a time, so it only shows that nothing regressed. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-11-22 19:17:57 +03:00
With immediate_commit=all a small write following an in-flight big write waits
in WAIT_BIG until that big write is acked, because its journal entry depends on
the entry of the big write. The ack of a big write unblocked every later
WAIT_BIG write of the object, including the ones waiting for a newer big write
which was still in flight. Their journal entries could then reach the disk
before the newer big write's entry, and after a crash or restart the replay put
them on top of the older big write, giving a version that was never written.

Stop at the next big write: the small writes after it are unblocked by its own
ack. The same was fixed for syncs in ack_sync().
vitalif merged commit e9570efe99 into master 2026-11-22 23:06:51 +03:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#149
No description provided.