osd: Keep the holders of unstable EC versions in the PG history and accept them from pg_data_size roles #131
Open
koolkhel
wants to merge 5 commits from
koolkhel/vitastor:peering-flush-minsize into master
pull from: koolkhel/vitastor:peering-flush-minsize
merge into: vitalif:master
vitalif:master
vitalif:tombstones
vitalif:primary-stickiness
vitalif:libvfn
vitalif:canceling
vitalif:immediate-repeering
vitalif:rdma-close-tcp
vitalif:use-degraded-writes
vitalif:fix-write-delete-flow
vitalif:openssl-3.0.12
vitalif:openssl-capture-secrets
vitalif:ignore-small-rdma-end
vitalif:encrypt-batch
vitalif:faster-peering-calc
vitalif:http-hello
vitalif:xxh3-bench
vitalif:create-user-etcd-based
vitalif:aesxts_streaming_nosupport
vitalif:refactor-heap-continue-write
vitalif:ston3lu-patch-0127
vitalif:rdma-credits
vitalif:heap-meta
vitalif:sendmsg-zc-no-more-error
vitalif:fix-clear-refs
vitalif:veeam-trick
vitalif:test-etcd-schizo
vitalif:test
vitalif:test1
vitalif:test2
vitalif:cow-meta
vitalif:kv-readahead
vitalif:qemu-fix-crash-idea
vitalif:eventloop-auto-init-trigger
vitalif:hugo-docs
vitalif:node-mutex-and-postpone
vitalif:check-writeback
vitalif:msgr-iothreads-v2
vitalif:antietcd
vitalif:rel-1.4
vitalif:test-fix-ec-unknown-state-51
vitalif:kv-update
vitalif:kv-debug
vitalif:hotfix-1.2.0
vitalif:rdma-simple-nodp
vitalif:zerocopy-tcp-send
vitalif:csi-staging
vitalif:hotfix-1.1.0
vitalif:hotfix-1.0.0
vitalif:fsync-feedback
vitalif:cached-reads
vitalif:qemu-send-loop
vitalif:test-double-alloc
vitalif:hier-failure-domains
vitalif:mon-self-restart
vitalif:epoch-deletions
vitalif:rdma-flow-control
vitalif:csi-use-vitastor-cli
vitalif:rdma-v2
vitalif:etcd-hide-base64
vitalif:lrc-matrix
vitalif:rm-left-on-dead
vitalif:nfs-proxy-old
vitalif:non-odp-rdma
vitalif:test-zctr
vitalif:separate-data-connections
vitalif:sec_osd_msgr
vitalif:test-assert
vitalif:nbd-vmsplice
vitalif:allow-etcd-address-option
vitalif:rdma-zerocopy
vitalif:rel-0.5
vitalif:openonload
vitalif:sync-io-test
vitalif:trace-sync
vitalif:trace-sqes
vitalif:test-submit-and-wait
vitalif:test-sq-poll
vitalif:blocking-uring-test
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
No description provided.
Delete Branch "koolkhel/vitastor:peering-flush-minsize"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
Your rule from the chat: the commit threshold for an unstable EC version must equal the history check. So it goes back to pg_data_size (it is pg_size since
368daa4b), and the OSDs holding an unstable target version go into the PG history, reported before the flush. A peering without some of those holders then doesn't activate the PG, instead of rolling back a version that is already stabilized on them. Reporting the current OSD set in place of the holders isn't enough, because the remaining OSDs are still enough of that set to activate the PG.The cases are EC 2+1 with pg_minsize 2 and EC 3+2 with pg_minsize 4. v2 reaches the listed number of OSDs, then the primary stabilizes its own part and dies. "Degraded write" is a client write to a PG degraded to pg_minsize OSDs. Each cell shows whether the object is complete when the primary returns, then whether the PG is inactive while the primary is down:
The PG only stays inactive when the version reached exactly pg_data_size OSDs, until the lost holder returns or is removed.
Commits:
osd: Test a lost primary of EC objects after its own part is stabilized: the cases above, plustest_peering_flush_then_holder_lost, where one holder is lost after the flush and before recovery.osd: Report the PG epoch and OSD set before peering flushes. Flush batches wait for the epoch to be reported, like writes, and a waiting batch holds a repeer or a stop. The write path's report moves toreport_pg_epoch().osd: Put the holders of unstable EC target versions into the PG history.osd: Accept unstable EC versions on peering from pg_data_size roles again. Intest_peering_partial_unstable_ec_rolls_back, subcases A and D now commit the chain on pg_data_size roles and put its holders in the history. B and C are unchanged.osd: Test lost primaries of degraded and misplaced EC objects.There is no new store operation and no new on-disk format. If you take this, the rollback-write of #124 isn't needed for this bug.
Questions
Tested:
osd_test: all pass. On master the new tests fail for 2+1 v2 on 3, 3+2 v2 on 5 and the degraded write to 3+2.test_peering_flush_degraded_primary_lost(EC 2+2 and 3+2 with a role without a piece, a new object and over a stable v1) andtest_peering_flush_misplaced_primary_lost(EC 2+1, a version on an OSD outside cur_set): the object is complete with this PR. On master the misplaced object with the version on all three OSDs ends incomplete.test_pg_chaosseeds 1-40, 1996 and 192 more EC seeds: no failures, the same as master. Chaos runs EC 2+2 with pg_minsize 2, so it doesn't exercise this. Seed 1996 from368daa4bpasses even without the holders.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA
osd: Accept unstable EC versions from pg_minsize roles and keep their holders in the PG historyto osd: Keep the holders of unstable EC versions in the PG history and accept them from pg_data_size rolese93600dcf4to960b39ef30View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.