Store v2: Wait for fsyncs submitted by other operations before completing a sync #107
Open
koolkhel
wants to merge 1 commits from
koolkhel/vitastor:v2-chaos-105267292 into master
pull from: koolkhel/vitastor:v2-chaos-105267292
merge into: vitalif:master
vitalif:master
vitalif:tombstones
vitalif:primary-stickiness
vitalif:libvfn
vitalif:canceling
vitalif:immediate-repeering
vitalif:rdma-close-tcp
vitalif:use-degraded-writes
vitalif:fix-write-delete-flow
vitalif:openssl-3.0.12
vitalif:openssl-capture-secrets
vitalif:ignore-small-rdma-end
vitalif:encrypt-batch
vitalif:faster-peering-calc
vitalif:http-hello
vitalif:xxh3-bench
vitalif:create-user-etcd-based
vitalif:aesxts_streaming_nosupport
vitalif:refactor-heap-continue-write
vitalif:ston3lu-patch-0127
vitalif:rdma-credits
vitalif:heap-meta
vitalif:sendmsg-zc-no-more-error
vitalif:fix-clear-refs
vitalif:veeam-trick
vitalif:test-etcd-schizo
vitalif:test
vitalif:test1
vitalif:test2
vitalif:cow-meta
vitalif:kv-readahead
vitalif:qemu-fix-crash-idea
vitalif:eventloop-auto-init-trigger
vitalif:hugo-docs
vitalif:node-mutex-and-postpone
vitalif:check-writeback
vitalif:msgr-iothreads-v2
vitalif:antietcd
vitalif:rel-1.4
vitalif:test-fix-ec-unknown-state-51
vitalif:kv-update
vitalif:kv-debug
vitalif:hotfix-1.2.0
vitalif:rdma-simple-nodp
vitalif:zerocopy-tcp-send
vitalif:csi-staging
vitalif:hotfix-1.1.0
vitalif:hotfix-1.0.0
vitalif:fsync-feedback
vitalif:cached-reads
vitalif:qemu-send-loop
vitalif:test-double-alloc
vitalif:hier-failure-domains
vitalif:mon-self-restart
vitalif:epoch-deletions
vitalif:rdma-flow-control
vitalif:csi-use-vitastor-cli
vitalif:rdma-v2
vitalif:etcd-hide-base64
vitalif:lrc-matrix
vitalif:rm-left-on-dead
vitalif:nfs-proxy-old
vitalif:non-odp-rdma
vitalif:test-zctr
vitalif:separate-data-connections
vitalif:sec_osd_msgr
vitalif:test-assert
vitalif:nbd-vmsplice
vitalif:allow-etcd-address-option
vitalif:rdma-zerocopy
vitalif:rel-0.5
vitalif:openonload
vitalif:sync-io-test
vitalif:trace-sync
vitalif:trace-sqes
vitalif:test-submit-and-wait
vitalif:test-sq-poll
vitalif:blocking-uring-test
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
No description provided.
Delete Branch "koolkhel/vitastor:v2-chaos-105267292"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
do_sync()returned at once when nothing was left unsynced, but these counters are reset when an fsync is submitted, not when it completes. So a stabilize or rollback whose metadata block was written before another operation submitted its fsync completed before that fsync did, and a power outage could undo it:test_blockstore_chaos --seed 105267292sees a rolled back version come back.Count fsyncs in flight and wait for them as well. Comparing
fsynced_lsnwithcompleted_lsninstead isn't enough: an unfinished earlier write keepscompleted_lsnbelow the entries of the operation being synced.Tested:
test_blockstore: newtest_rollback_waits_for_inflight_fsync, with and without an unfinished earlier write. Both fail without the fix: the rollback completes while the fsync is held.test_blockstore_chaos --seed 105267292,--seeds 1-960, and--config 3/4/5× seeds 100001-101000 pass.--config 5 --seed 100085and the same 3000 runs pass.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA
@@ -107,6 +107,16 @@ int blockstore_impl_t::do_sync(blockstore_op_t *op, int base_state)}if (!has_unsynced()){if ((!dsk.disable_meta_fsync || !dsk.disable_journal_fsync) &&тут точно обе правки нужны? выглядит как будто то ли два разных кейса поправлено, то ли один, но двумя способами. верхней правки не хватит ли?
One more gap in the same sync, found by stress-testing #124 on top of this PR:
test_blockstore_chaos --config 5 --seed 100085still failed there.The LSN check here compares
fsynced_lsnwithcompleted_lsn, butcompleted_lsnonly advances when all lower LSNs complete. While an earlier write is still unfinished, a rollback or stabilize whose metadata write is covered only by an fsync another operation has just submitted seesfsynced_lsn == completed_lsn. It is acknowledged before that fsync completes, and a power outage then undoes it.312a457bcounts fsyncs in flight (submit_fsyncs()adds them, their completion subtracts) and makesdo_sync()wait for them when nothing is left to sync, before the LSN check.test_rollback_waits_for_inflight_fsyncgets a variant with an unfinished earlier write, which fails without the fix.Tested: test_blockstore, test_heap,
test_blockstore_chaos --seeds 1-960and configs 3-5 × seeds 100001-101000 all pass. With #124 on top, seed 100085 passes.312a457bc5tob950937bacIt was one fix in two parts, and it was incomplete. I tested each edit separately:
has_unfinished_syncfor a waiting sync, alone, changes nothing. Before the LSN wait a sync could only wait for an SQE, and that breaks out of the loop before this line.test_blockstore_chaos --seed 105267292still fails.has_unfinished_syncis set, the later sync'spending_opsis reset inloop()while its fsync is in flight, and the fsync callback hitsassert(wait_count >= 0). Seed 105267292 aborts there.completed_lsnlow, and a rollback is acked before the fsync that covers it.The fsync counter covers all three by itself. When nothing is unsynced and no fsync is in flight, every completed write is already covered by a finished fsync, so the LSN wait never has anything left to wait for. The counter's wait has no wait state, so the first edit isn't needed either. I've reduced the PR to that:
b950937b, one commit with the counter and both test variants, in place of933f2302and312a457b. test_blockstore and seed 105267292 pass, and so does--config 5 --seed 100085with #124 on top. The same change with the first edit still in, where it is unreachable, passed--seeds 1-960and configs 3-5 × seeds 100001-101000. Those sweeps are running again on the squashed commit, and I'll add the results to the description.One property of the counter: a sync waits for any fsync in flight, so under traffic that keeps submitting a new fsync before the last one completes, it can wait for a long time. The wait for the flusher's
syncing_bufferjust above it behaves the same way. If that matters, an op could record the number of the last fsync batch submitted and wait only for batches up to it. I can do that instead.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.