Roll back a peering flush under a new PG epoch #124
Closed
koolkhel
wants to merge 11 commits from
koolkhel/vitastor:peering-flush-lone-stable into master
pull from: koolkhel/vitastor:peering-flush-lone-stable
merge into: vitalif:master
vitalif:master
vitalif:tombstones
vitalif:primary-stickiness
vitalif:libvfn
vitalif:canceling
vitalif:immediate-repeering
vitalif:rdma-close-tcp
vitalif:use-degraded-writes
vitalif:fix-write-delete-flow
vitalif:openssl-3.0.12
vitalif:openssl-capture-secrets
vitalif:ignore-small-rdma-end
vitalif:encrypt-batch
vitalif:faster-peering-calc
vitalif:http-hello
vitalif:xxh3-bench
vitalif:create-user-etcd-based
vitalif:aesxts_streaming_nosupport
vitalif:refactor-heap-continue-write
vitalif:ston3lu-patch-0127
vitalif:rdma-credits
vitalif:heap-meta
vitalif:sendmsg-zc-no-more-error
vitalif:fix-clear-refs
vitalif:veeam-trick
vitalif:test-etcd-schizo
vitalif:test
vitalif:test1
vitalif:test2
vitalif:cow-meta
vitalif:kv-readahead
vitalif:qemu-fix-crash-idea
vitalif:eventloop-auto-init-trigger
vitalif:hugo-docs
vitalif:node-mutex-and-postpone
vitalif:check-writeback
vitalif:msgr-iothreads-v2
vitalif:antietcd
vitalif:rel-1.4
vitalif:test-fix-ec-unknown-state-51
vitalif:kv-update
vitalif:kv-debug
vitalif:hotfix-1.2.0
vitalif:rdma-simple-nodp
vitalif:zerocopy-tcp-send
vitalif:csi-staging
vitalif:hotfix-1.1.0
vitalif:hotfix-1.0.0
vitalif:fsync-feedback
vitalif:cached-reads
vitalif:qemu-send-loop
vitalif:test-double-alloc
vitalif:hier-failure-domains
vitalif:mon-self-restart
vitalif:epoch-deletions
vitalif:rdma-flow-control
vitalif:csi-use-vitastor-cli
vitalif:rdma-v2
vitalif:etcd-hide-base64
vitalif:lrc-matrix
vitalif:rm-left-on-dead
vitalif:nfs-proxy-old
vitalif:non-odp-rdma
vitalif:test-zctr
vitalif:separate-data-connections
vitalif:sec_osd_msgr
vitalif:test-assert
vitalif:nbd-vmsplice
vitalif:allow-etcd-address-option
vitalif:rdma-zerocopy
vitalif:rel-0.5
vitalif:openonload
vitalif:sync-io-test
vitalif:trace-sync
vitalif:trace-sqes
vitalif:test-submit-and-wait
vitalif:test-sq-poll
vitalif:blocking-uring-test
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
No description provided.
Delete Branch "koolkhel/vitastor:peering-flush-lone-stable"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
An EC object can be left incomplete for good by a partially executed stabilize. The primary commits an unstable version by sending STABILIZE to every OSD separately. If it completes its own part and dies before the others arrive, the next peering, without it, sees the version unstable on fewer roles, rolls it back on the OSDs present, and nothing newer is ever written to the object. When the lost OSD returns, its stable part is the newest version, and with fewer than pg_data_size parts of it the object is incomplete.
This is reachable in two ways, with a single failure of the primary:
immediate_commit=all:submit_primary_stab_subops()sends the local BS_OP_STABLE and the SEC_STABILIZEs in parallel. The primary dies after its local STABLE, and so does the client that was writing: nobody retries the write.submit_pg_flush_ops()(test_peering_flush_primary_lostin this PR).The write was never acknowledged in either case (under
immediate_commit=allthe ack follows the stabilization), so rolling it back is legal. What is missing is something that tells the next peerings that the stable part on the lost OSD is outdated.Why no target threshold fixes it
The next peering decides between commit and rollback by how many roles have the unstable version. A write can reach exactly as many OSDs as the threshold, and the peering without the lost primary then sees one fewer. So every rule fails some write, as the osd_test scenarios show (EC 2+1, pg_minsize 2; EC 3+2, pg_minsize 4; v2 written to the listed number of OSDs, the primary stabilizes its own part and dies; FAIL = the object is incomplete when the primary returns):
368daa4b)368daa4b)The PR keeps the current rule. The tests in the PR are the two 2+1 cases and the 3+2 case with v2 on all five OSDs, which the current rule gets wrong without rollback-write.
Design
Following your design: where peering would roll back an EC object, it leaves a new version under a new PG epoch, which is reported to etcd with the current OSD set first. Any part of the rolled back version left on an absent OSD is then outdated.
BS_OP_ROLLBACK_WRITE. It takes the same{oid, version}list as BS_OP_ROLLBACK plusop->version, the new version of every listed object. Each object is rolled back with the preconditions of BS_OP_ROLLBACK, and the new version is added on top as a synced, but unstable, zero-length write that refers to the data and the external bitmap of the version rolled back to, without copying them. BS_OP_STABLE commits it later like any write. It needs no journal space for data. The new version must be above every existing one (-EINVAL otherwise), and repeating the operation is a no-op.JE_ROLLBACK_WRITEwith both versions, the bitmap and the block of an empty version. Replay runs the rollback first, which frees the taken over block, then claims it for the new version. Journal space for the later commit is only reserved for objects which aren't unstable already.BS_HEAP_ROLLBACK_WRITE, which stores the version rolled back to at its end and rolls back every older entry above it. Both steps are persisted at once. The taken over block is shared by the new entry and the rolled back big write until both become garbage together:mark_garbage_up_to()already frees a block shared by neighbouring entries once, andmark_used_blocks()now counts it once at start. A block is only taken over from a big write which isn't rolled back yet: it's unstable then, so compaction can't free it in the meantime.enable_rollback_write, for both stores. Turning it on upgrades the Store v1 journal to version 3, or the Store v2 metadata superblock to format 4, durably at start (not in readonly mode), and versions without this PR refuse to start with either. Without the guard, older versions would take JE_ROLLBACK_WRITE for the end of the journal, and refuse or (withskip_corrupted_meta_entries) drop the v2 entries. An upgraded journal or superblock stays upgraded when the option is turned off again.vitastor-disk prepareputs the option into the superblock of new OSDs, on by default (--enable_rollback_write 0turns it off); existing OSDs keep running without it until it's added withupdate-sb.rollback_writein the SHOW_CONFIG features, likepg_locks. On the wire it's SEC_ROLLBACK with a new flagOSD_OP_ROLLBACK_WRITEand the new version in previously unused bytes of the op header, so its payload, PG lock and statistics handling stay as they are. The primary only uses it for an object when its own store and every OSD it would send it to support it; otherwise the object is rolled back as before.submit_pg_flush_ops()takes the pieces of its target version (from the object state, or cur_set for a clean object or one without a target version) and sends them BS_OP_ROLLBACK_WRITE to(epoch << (64-PG_EPOCH_BITS)) | 1, above every listed version, then stabilizes the new version once every rollback-write has succeeded. A primary that dies in between leaves an unstable new version, which the next peering commits or rolls back by the usual rule.Commits
osd: Add a test for a lone stable EC part left by a peering flush on a lost primary:test_peering_flush_primary_lost, EC 2+1,immediate_commit=all, v2 written to 2 or to 3 OSDs. On master the partial case passes and the full case fails; with the condition before368daa4bit's the other way round.Store v1: Add BS_OP_ROLLBACK_WRITE, a rollback that assigns a new version: the operation, JE_ROLLBACK_WRITE, the block takeover and the journal reservation. Store v2 rejects it at this point. Half of the Store v1 rollbacks oftest_blockstore_chaosbecome rollback-writes, so Store v1 two-phase seeds (seed % 24 in 12..17) replay differently from here on.vitastor-disk: Support JE_ROLLBACK_WRITE journal entries: dump-journal, write-journal, and the empty version's block in raw-resize and trim.Store v1: Add enable_rollback_write and journal version 3: the option, the journal upgrade, andblockstore_i::is_rollback_write_enabled()for the OSD. The trim takes the journal version from memory now.vitastor-disk: Support journal version 3, enable rollback-write for new OSDs.Store v2: Support BS_OP_ROLLBACK_WRITE: the heap entry, the block takeover and its accounting at start, metadata format 4, and the option docs for both stores. Store v2 two-phase chaos seeds (seed % 24 in 0..5) replay differently from here on.vitastor-disk: Support Store v2 rollback-write entries: dump-meta, write-meta and resize.osd: Report rollback-write support and execute it on secondaries: the SHOW_CONFIG feature and the flagged SEC_ROLLBACK.osd: Roll back peering flushes of EC objects with a new version: the OSD side above.test_pg_chaos: Add --rollback-write mode: the primary on Store v1 with the option, fake peers that report the feature and apply flagged SEC_ROLLBACKs.osd: Test a lost peering flush on EC 3+2 with pg_minsize 4.Tested
On top of master
8582f823:osd_test: all pass. New:test_sec_rollback_write,test_peering_flush_primary_lost(also with an OSD without the feature, as a peer and as the primary),test_peering_flush_rollback_write_lost,test_peering_flush_incomplete,test_peering_flush_waits_for_epoch,test_peering_flush_repeer_waits_for_epoch,test_peering_flush_primary_lost_minsize. On master alonetest_peering_flush_primary_lost[full write]fails, as expected.test_blockstore_v1: all pass. New:test_rollback_write(with and without checksums and power losses),test_rollback_write_errors,test_rollback_write_no_base(an unstable big write, a deletion and a never written object as the base),test_rollback_write_no_space(a full device, with the block takeover and a power loss),test_rollback_write_journal_space(100 unstable objects rolled back at once on a 1 MB journal, which failed with -EAGAIN while they were counted twice),test_rollback_write_over_unsynced,test_rollback_write_option(readonly open writes nothing, a trim in the upgrading session keeps version 3),test_rollback_write_v1_size_start.test_blockstore: all pass. New:test_rollback_write,_errors,_disabled,_meta_format,_no_base(the takeover across power losses and compaction),_no_space(a full device, the takeover across a power loss and compaction, and the block freed once on deletion),_over_unsynced,_lost_before_sync.test_heap,test_allocator: all pass.test_blockstore_chaos --seeds 1-960: all pass. With ASan and UBSan, seeds 1-240: all pass, no sanitizer reports.test_blockstore_chaosconfigurations 3, 4 and 5 (Store v2, fsync, two-phase) × seeds 100001-101000, plus--config 4 --seed 100449: 3 of 3000 fail (config 3 and config 5 seed 100796, config 4 seed 100549), all with "came back from a power outage holding unexpected content". That is the fsync race of #107: they pass with #107 applied. Master alone fails 100449 the same way and nothing else; master with #107 passes all of them. With #107, one seed of the same class is left on the branch, config 5 seed 100085. It passes with the earlier version of that fix, which also makes the flusher'sfsync_buffer()wait for fsyncs in flight, and with that version the branch passes all 3001 seeds. On master with #107, configurations 3, 4 and 5 × seeds 100001-103000 all pass.test_pg_chaos, default seeds 1-80: all pass.test_pg_chaos --ec --rollback-write(the primary on Store v1), seeds 1-40: 12 fail; master with its primary switched to Store v1,--ec, seeds 1-40: 17 fail. Every failure of the branch is of a class that master has as well: "OSD did not finish peering", "the PG did not settle", theack_sync()andforget_unstable_write()assertions of Store v1, and "Tried to write ... after delete, but already have". Over seeds 1-200 it's 60 failures against 79, with the same classes.Questions
An object rolled back to nothing with a cur_set role without a piece falls back to a plain rollback (the unflushed deletion case above). With two or more parity chunks, a stabilized empty version on an OSD that is absent later can then become a lone stable part:
A fix could be to give the empty version to the roles that have pieces instead of falling back. Should it?
The new version can land on OSDs outside cur_set. For a misplaced object, the pieces of the target version are taken from its object state and may be on OSDs that aren't in cur_set, but the epoch is reported with cur_set only. Is that enough, or should the history record the OSDs written to?
A flag or a new opcode on the wire. SEC_ROLLBACK with
OSD_OP_ROLLBACK_WRITEand the version in previously unused header bytes, guarded by therollback_writefeature. A new secondary op would need its own PG lock and statistics handling. Is the flag fine?Store v2 metadata format 4 and its name,
BLOCKSTORE_META_FORMAT_HEAP_RBW.A flag bit or a new entry type in Store v2.
BS_HEAP_ROLLBACK_WRITE(0x20) is a modifier of SMALL_WRITE and BIG_WRITE, so all code that handles writes handles the new entries, and only the rollback walks check the flag. A new type would have to be added to every switch over entry types.Store v1 to heap conversion. vitastor-disk's conversion of Store v1 metadata with its journal to the heap format (
index_journal_by_object()) refuses rollback_write entries as unknown. As far as I can see,write_json_meta()doesn't reach that path yet. Should it convert them?The block takeover in Store v2 relies on the taken over big write not being rolled back already; when it is, or when the newest big entry is a BIG_INTENT, a new block is allocated, and -ENOSPC is still possible. That only happens after an earlier plain rollback of a newer big write of the same object. Is the shared block, counted once at start, acceptable to you, or would you prefer another way?
Repeers and stops wait for the epoch report of a waiting flush batch, so a PG whose etcd doesn't answer can't repeer or stop until it does, the same as with a queued write. OK?
vitastor-disk prepareenables the option by default for new OSDs, while it's off by default in the OSD. Do you want it on everywhere, or off in prepare too?test_pg_chaos --rollback-writeruns the primary on Store v1, where test_pg_chaos has failures of its own. Should the mode use Store v2 now that it supports the operation?Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA
WIP: osd: Add a test for a lone stable EC part left by a peering flush on a lost primaryto WIP: Roll back a peering flush under a new PG epoche6212f7c2fto7dc4442f6fWIP: Roll back a peering flush under a new PG epochto Roll back a peering flush under a new PG epochClosing for now: the bug is fixed by #131 (
ac3f6f85). It keeps the OSDs holding a target version the flush stabilizes in the PG history, so a peering without them doesn't roll back a version that is already stable on one of them. Rollback-write isn't needed for it. The branch stays in case it comes back.Pull request closed