WIP: Roll back a peering flush under a new PG epoch #124
Draft
koolkhel
wants to merge 3 commits from
koolkhel/vitastor:peering-flush-lone-stable into master
pull from: koolkhel/vitastor:peering-flush-lone-stable
merge into: vitalif:master
vitalif:master
vitalif:tombstones
vitalif:primary-stickiness
vitalif:libvfn
vitalif:canceling
vitalif:immediate-repeering
vitalif:rdma-close-tcp
vitalif:use-degraded-writes
vitalif:fix-write-delete-flow
vitalif:openssl-3.0.12
vitalif:openssl-capture-secrets
vitalif:ignore-small-rdma-end
vitalif:encrypt-batch
vitalif:faster-peering-calc
vitalif:http-hello
vitalif:xxh3-bench
vitalif:create-user-etcd-based
vitalif:aesxts_streaming_nosupport
vitalif:refactor-heap-continue-write
vitalif:ston3lu-patch-0127
vitalif:rdma-credits
vitalif:heap-meta
vitalif:sendmsg-zc-no-more-error
vitalif:fix-clear-refs
vitalif:veeam-trick
vitalif:test-etcd-schizo
vitalif:test
vitalif:test1
vitalif:test2
vitalif:cow-meta
vitalif:kv-readahead
vitalif:qemu-fix-crash-idea
vitalif:eventloop-auto-init-trigger
vitalif:hugo-docs
vitalif:node-mutex-and-postpone
vitalif:check-writeback
vitalif:msgr-iothreads-v2
vitalif:antietcd
vitalif:rel-1.4
vitalif:test-fix-ec-unknown-state-51
vitalif:kv-update
vitalif:kv-debug
vitalif:hotfix-1.2.0
vitalif:rdma-simple-nodp
vitalif:zerocopy-tcp-send
vitalif:csi-staging
vitalif:hotfix-1.1.0
vitalif:hotfix-1.0.0
vitalif:fsync-feedback
vitalif:cached-reads
vitalif:qemu-send-loop
vitalif:test-double-alloc
vitalif:hier-failure-domains
vitalif:mon-self-restart
vitalif:epoch-deletions
vitalif:rdma-flow-control
vitalif:csi-use-vitastor-cli
vitalif:rdma-v2
vitalif:etcd-hide-base64
vitalif:lrc-matrix
vitalif:rm-left-on-dead
vitalif:nfs-proxy-old
vitalif:non-odp-rdma
vitalif:test-zctr
vitalif:separate-data-connections
vitalif:sec_osd_msgr
vitalif:test-assert
vitalif:nbd-vmsplice
vitalif:allow-etcd-address-option
vitalif:rdma-zerocopy
vitalif:rel-0.5
vitalif:openonload
vitalif:sync-io-test
vitalif:trace-sync
vitalif:trace-sqes
vitalif:test-submit-and-wait
vitalif:test-sq-poll
vitalif:blocking-uring-test
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
No description provided.
Delete Branch "koolkhel/vitastor:peering-flush-lone-stable"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
A peering flush can leave an EC object incomplete for good. If the primary stabilizes its own part and loses its peers before theirs arrive, the next peering, without it, can roll back the remaining unstable parts of a version that is already stable on the lost OSD. When that OSD returns, its lone stable part is the newest version and the object stays incomplete. The condition from
368daa4bmoves this window rather than closing it.Work in progress, following your design: the rollback leaves a new version under a new PG epoch, so the lone stable part becomes outdated. This PR so far has the store side for Store v1:
osd: Add a test for a lone stable EC part left by a peering flush on a lost primary:test_peering_flush_primary_lost, EC 2+1,immediate_commit=all. On master the partial-write case passes and the full-write case fails atassert(!incomplete). With the pre-368daa4b condition it's the other way round.Store v1: Add BS_OP_ROLLBACK_WRITE, a rollback that assigns a new version.op->version. It takes the same{oid, version}list as BS_OP_ROLLBACK, plusop->versionas the new version for every listed object.JE_ROLLBACK_WRITEentry with both versions, the bitmap and, for an empty version (below), its data block. It needs no journal space for data.vitastor-disk: Support JE_ROLLBACK_WRITE journal entries: dump-journal and write-journal, plus raw-resize and trim, which treat the empty version's block like a big write's.Still to do: the OSD side (allocate and report the new epoch with the current OSD set before the flush's rollbacks, send the new op, then commit the new versions), the network op and its capability flag, and Store v2. Older versions treat the new journal entry as the end of the journal, so the behaviour will need an opt-in flag.
Tested:
test_blockstore_v1: all pass, with new teststest_rollback_write(with and without checksums, with and without simulated power loss after each step),test_rollback_write_no_base(an unstable big write, a delete and a never-written object as the base),test_rollback_write_no_space,test_rollback_write_errorsandtest_rollback_write_over_unsynced.test_blockstore_chaos --seeds 1-960: all pass. The chaos test now turns half of the Store v1 rollbacks into BS_OP_ROLLBACK_WRITE, so Store v1 two-phase seeds replay differently from here on. Its model doesn't reach the no-base case; the unit tests cover it.vitastor-disk: a journal with both kinds of entry survives a dump-journal/write-journal round trip, and raw-resize remaps the empty version's block.osd_test: still fails only attest_peering_flush_primary_lost[full write], as expected until the OSD side is done.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA
WIP: osd: Add a test for a lone stable EC part left by a peering flush on a lost primaryto WIP: Roll back a peering flush under a new PG epochView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.