Store v1: Don't let a sync of an older big write unblock later small writes #129
Open
koolkhel
wants to merge 1 commits from
koolkhel/vitastor:chaos-seed-133 into master
pull from: koolkhel/vitastor:chaos-seed-133
merge into: vitalif:master
vitalif:master
vitalif:tombstones
vitalif:primary-stickiness
vitalif:libvfn
vitalif:canceling
vitalif:immediate-repeering
vitalif:rdma-close-tcp
vitalif:use-degraded-writes
vitalif:fix-write-delete-flow
vitalif:openssl-3.0.12
vitalif:openssl-capture-secrets
vitalif:ignore-small-rdma-end
vitalif:encrypt-batch
vitalif:faster-peering-calc
vitalif:http-hello
vitalif:xxh3-bench
vitalif:create-user-etcd-based
vitalif:aesxts_streaming_nosupport
vitalif:refactor-heap-continue-write
vitalif:ston3lu-patch-0127
vitalif:rdma-credits
vitalif:heap-meta
vitalif:sendmsg-zc-no-more-error
vitalif:fix-clear-refs
vitalif:veeam-trick
vitalif:test-etcd-schizo
vitalif:test
vitalif:test1
vitalif:test2
vitalif:cow-meta
vitalif:kv-readahead
vitalif:qemu-fix-crash-idea
vitalif:eventloop-auto-init-trigger
vitalif:hugo-docs
vitalif:node-mutex-and-postpone
vitalif:check-writeback
vitalif:msgr-iothreads-v2
vitalif:antietcd
vitalif:rel-1.4
vitalif:test-fix-ec-unknown-state-51
vitalif:kv-update
vitalif:kv-debug
vitalif:hotfix-1.2.0
vitalif:rdma-simple-nodp
vitalif:zerocopy-tcp-send
vitalif:csi-staging
vitalif:hotfix-1.1.0
vitalif:hotfix-1.0.0
vitalif:fsync-feedback
vitalif:cached-reads
vitalif:qemu-send-loop
vitalif:test-double-alloc
vitalif:hier-failure-domains
vitalif:mon-self-restart
vitalif:epoch-deletions
vitalif:rdma-flow-control
vitalif:csi-use-vitastor-cli
vitalif:rdma-v2
vitalif:etcd-hide-base64
vitalif:lrc-matrix
vitalif:rm-left-on-dead
vitalif:nfs-proxy-old
vitalif:non-odp-rdma
vitalif:test-zctr
vitalif:separate-data-connections
vitalif:sec_osd_msgr
vitalif:test-assert
vitalif:nbd-vmsplice
vitalif:allow-etcd-address-option
vitalif:rdma-zerocopy
vitalif:rel-0.5
vitalif:openonload
vitalif:sync-io-test
vitalif:trace-sync
vitalif:trace-sqes
vitalif:test-submit-and-wait
vitalif:test-sq-poll
vitalif:blocking-uring-test
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
No description provided.
Delete Branch "koolkhel/vitastor:chaos-seed-133"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
A small write following an unsynced big write waits in
WAIT_BIGuntil that big write is synced, because a big write only gets its journal entry from the sync.ack_sync()unblocked everyWAIT_BIGwrite of the object after any synced big write, including the ones waiting for a newer big write which that sync didn't cover. Their journal entries could then reach the disk before the newer big write's entry, and after a crash or restart the replay put them on top of the older big write, giving a version that was never written.Sequence on one object (immediate_commit != all): big write v1 acked; sync S starts (takes only v1); big write v2 acked; small write v3 queued in
WAIT_BIGbehind v2; S completes andmark_sync(v1)releases v3, whose entry is journaled at once; a STABLE/ROLLBACK rewrites that journal sector; crash before v2's entry is written. Replay sees BIG_WRITE v1 + SMALL_WRITE v3 and reports v3 = v3's bytes over v1.Stop unblocking at the next big write of the object: its own sync unblocks the small writes after it. A big write which is itself in
WAIT_BIG(after a delete) is still released.Tested:
test_blockstore_chaos --seed 133 --config 17fails before the fix ("came back from a power outage holding unexpected content") and passes after it.test_small_write_waits_for_its_own_big_writeintest_blockstore_v1(no checksums, plain restart instead of a power cut): fails without the fix, passes with it; the whole suite passes.test_blockstore_chaosconfigs 15, 16, 17, 21, 22, 23: seeds 1-1000 with 10k steps and seeds 1-200 with 50k steps; before the fix only seed 133 of config 17 fails, after it none, and no hangs.Not changed:
write.cpphas a similar release-everything loop for big writes acked under immediate_commit=all; it would only misfire with three writes to one object in flight at once.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.