Store v1: Don't let a sync of an older big write unblock later small writes #129
Reference in New Issue
Block a user
No description provided.
Delete Branch "koolkhel/vitastor:chaos-seed-133"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
A small write following an unsynced big write waits in
WAIT_BIGuntil that big write is synced, because a big write only gets its journal entry from the sync.ack_sync()unblocked everyWAIT_BIGwrite of the object after any synced big write, including the ones waiting for a newer big write which that sync didn't cover. Their journal entries could then reach the disk before the newer big write's entry, and after a crash or restart the replay put them on top of the older big write, giving a version that was never written.Sequence on one object (immediate_commit != all): big write v1 acked; sync S starts (takes only v1); big write v2 acked; small write v3 queued in
WAIT_BIGbehind v2; S completes andmark_sync(v1)releases v3, whose entry is journaled at once; a STABLE/ROLLBACK rewrites that journal sector; crash before v2's entry is written. Replay sees BIG_WRITE v1 + SMALL_WRITE v3 and reports v3 = v3's bytes over v1.Stop unblocking at the next big write of the object: its own sync unblocks the small writes after it. A big write which is itself in
WAIT_BIG(after a delete) is still released.Tested:
test_blockstore_chaos --seed 133 --config 17fails before the fix ("came back from a power outage holding unexpected content") and passes after it.test_small_write_waits_for_its_own_big_writeintest_blockstore_v1(no checksums, plain restart instead of a power cut): fails without the fix, passes with it; the whole suite passes.test_blockstore_chaosconfigs 15, 16, 17, 21, 22, 23: seeds 1-1000 with 10k steps and seeds 1-200 with 50k steps; before the fix only seed 133 of config 17 fails, after it none, and no hangs.Not changed:
write.cpphas a similar release-everything loop for big writes acked under immediate_commit=all; it would only misfire with three writes to one object in flight at once.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA