Store v1: Don't let the ack of an older big write unblock later small writes #149
Reference in New Issue
Block a user
No description provided.
Delete Branch "koolkhel/vitastor:blockstore-v1-wait-big-ack"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
The same problem as #129, in the immediate_commit=all path.
With immediate_commit=all a small write following an in-flight big write waits in
WAIT_BIGuntil that big write is acked. The "Unblock small writes" loop inblockstore_impl_t::continue_write()(the write ack) unblocked every laterWAIT_BIGwrite of the object, including the ones waiting for a newer big write which was still in flight. Their journal entries could then reach the disk before the newer big write's entry, and after a crash or restart the replay put them on top of the older big write, giving a version that was never written.The loop now stops at the next big write: the small writes after it are unblocked by its own ack.
It needs three writes to one object in the blockstore at once: big A, big B, small C, with A acked before B. The OSD primary runs one write per object at a time, so as far as I can see from the code this is only possible on a secondary of a replicated pool, with writes left in flight by dropped connections:
BS_OP_LISTdoes not wait for it and lists an in-flightWRITE_STABLEas stable, so after re-peering a new write to the same object runs next to it;I only checked this part by reading the code, I didn't reproduce it with OSDs. EC pools don't seem to be affected, because there the leftover write is listed as unstable and the next write waits for its rollback or stabilize.
Tested:
test_small_write_waits_for_its_own_big_write_immintest_blockstore_v1: v1 (big), v2 (big) and v3 (small) are submitted at once, the data write of v2 is held. Without the fix v3 completes while v2 is in flight, and after a restart the object comes back as v3 written over v1. With the fix v3 waits and the object comes back as v1. All othertest_blockstore_v1tests pass.test_blockstore_chaos --seeds 1-300on all v1 configs (12-23): all pass. The chaos test runs one operation per object at a time, so it only shows that nothing regressed.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA