Roll back a peering flush under a new PG epoch #124

Closed
koolkhel wants to merge 11 commits from koolkhel/vitastor:peering-flush-lone-stable into master
Contributor

Description

An EC object can be left incomplete for good by a partially executed stabilize. The primary commits an unstable version by sending STABILIZE to every OSD separately. If it completes its own part and dies before the others arrive, the next peering, without it, sees the version unstable on fewer roles, rolls it back on the OSDs present, and nothing newer is ever written to the object. When the lost OSD returns, its stable part is the newest version, and with fewer than pg_data_size parts of it the object is incomplete.

This is reachable in two ways, with a single failure of the primary:

  • The ordinary EC write path with immediate_commit=all: submit_primary_stab_subops() sends the local BS_OP_STABLE and the SEC_STABILIZEs in parallel. The primary dies after its local STABLE, and so does the client that was writing: nobody retries the write.
  • A peering flush: the same STABILIZEs sent by submit_pg_flush_ops() (test_peering_flush_primary_lost in this PR).

The write was never acknowledged in either case (under immediate_commit=all the ack follows the stabilization), so rolling it back is legal. What is missing is something that tells the next peerings that the stable part on the lost OSD is outdated.

Why no target threshold fixes it

The next peering decides between commit and rollback by how many roles have the unstable version. A write can reach exactly as many OSDs as the threshold, and the peering without the lost primary then sees one fewer. So every rule fails some write, as the osd_test scenarios show (EC 2+1, pg_minsize 2; EC 3+2, pg_minsize 4; v2 written to the listed number of OSDs, the primary stabilizes its own part and dies; FAIL = the object is incomplete when the primary returns):

Rule for an unstable target version Rollback-write 2+1, v2 on 2 2+1, v2 on 3 3+2, v2 on 3 3+2, v2 on 4 3+2, v2 on 5
n_roles >= pg_size (current, 368daa4b) no pass FAIL pass pass FAIL
n_roles >= pg_data_size (before 368daa4b) no FAIL pass FAIL pass pass
n_roles >= pg_minsize no FAIL pass pass FAIL pass
n_roles >= pg_size yes pass pass pass pass pass
n_roles >= pg_data_size yes pass pass pass pass pass
n_roles >= pg_minsize yes pass pass pass pass pass

The PR keeps the current rule. The tests in the PR are the two 2+1 cases and the 3+2 case with v2 on all five OSDs, which the current rule gets wrong without rollback-write.

Design

Following your design: where peering would roll back an EC object, it leaves a new version under a new PG epoch, which is reported to etcd with the current OSD set first. Any part of the rolled back version left on an absent OSD is then outdated.

  • Store operation BS_OP_ROLLBACK_WRITE. It takes the same {oid, version} list as BS_OP_ROLLBACK plus op->version, the new version of every listed object. Each object is rolled back with the preconditions of BS_OP_ROLLBACK, and the new version is added on top as a synced, but unstable, zero-length write that refers to the data and the external bitmap of the version rolled back to, without copying them. BS_OP_STABLE commits it later like any write. It needs no journal space for data. The new version must be above every existing one (-EINVAL otherwise), and repeating the operation is a no-op.
    • Nothing to roll back to (no version at or below the target, or a deletion): the new version is an empty big write, like a zero-length write to an absent object. It takes over the data block of the big write it rolls back (the first write after a deletion or to an absent object is always big), so a full device doesn't make it fail where a plain rollback would succeed. A free block is only needed when there is no such write.
    • Store v1: a new journal entry JE_ROLLBACK_WRITE with both versions, the bitmap and the block of an empty version. Replay runs the rollback first, which frees the taken over block, then claims it for the new version. Journal space for the later commit is only reserved for objects which aren't unstable already.
    • Store v2: a single heap entry per object, a zero-length SMALL_WRITE or an empty BIG_WRITE with a new flag BS_HEAP_ROLLBACK_WRITE, which stores the version rolled back to at its end and rolls back every older entry above it. Both steps are persisted at once. The taken over block is shared by the new entry and the rolled back big write until both become garbage together: mark_garbage_up_to() already frees a block shared by neighbouring entries once, and mark_used_blocks() now counts it once at start. A block is only taken over from a big write which isn't rolled back yet: it's unstable then, so compaction can't free it in the meantime.
  • Downgrade guards and opt-in. The operation is only accepted with the new OSD option enable_rollback_write, for both stores. Turning it on upgrades the Store v1 journal to version 3, or the Store v2 metadata superblock to format 4, durably at start (not in readonly mode), and versions without this PR refuse to start with either. Without the guard, older versions would take JE_ROLLBACK_WRITE for the end of the journal, and refuse or (with skip_corrupted_meta_entries) drop the v2 entries. An upgraded journal or superblock stays upgraded when the option is turned off again. vitastor-disk prepare puts the option into the superblock of new OSDs, on by default (--enable_rollback_write 0 turns it off); existing OSDs keep running without it until it's added with update-sb.
  • Capability negotiation. An OSD whose store has the option on reports rollback_write in the SHOW_CONFIG features, like pg_locks. On the wire it's SEC_ROLLBACK with a new flag OSD_OP_ROLLBACK_WRITE and the new version in previously unused bytes of the op header, so its payload, PG lock and statistics handling stay as they are. The primary only uses it for an object when its own store and every OSD it would send it to support it; otherwise the object is rolled back as before.
  • OSD. For an EC object with something to roll back, submit_pg_flush_ops() takes the pieces of its target version (from the object state, or cur_set for a clean object or one without a target version) and sends them BS_OP_ROLLBACK_WRITE to (epoch << (64-PG_EPOCH_BITS)) | 1, above every listed version, then stabilizes the new version once every rollback-write has succeeded. A primary that dies in between leaves an unstable new version, which the next peering commits or rolls back by the usual rule.
    • Every flush batch now waits until its epoch is reported with the current OSD set, like writes do, also when it has no new versions; and a batch waiting for the report holds a repeer or a stop of the PG, like a queued write does. Without the second part a repeer takes the same epoch again and writes the new version to its smaller OSD set without reporting it.
    • An incomplete object (fewer than pg_data_size pieces of the target version) is rolled back as before: a new version on its pieces would outdate the pieces of that version left on absent OSDs.
    • An object without a target version gets an empty new version only if every cur_set role has a piece of it, because an OSD without one may have an unflushed deletion there.
    • EC only: replicated pools keep plain rollbacks, any copy there is a whole object.

Commits

  1. osd: Add a test for a lone stable EC part left by a peering flush on a lost primary: test_peering_flush_primary_lost, EC 2+1, immediate_commit=all, v2 written to 2 or to 3 OSDs. On master the partial case passes and the full case fails; with the condition before 368daa4b it's the other way round.
  2. Store v1: Add BS_OP_ROLLBACK_WRITE, a rollback that assigns a new version: the operation, JE_ROLLBACK_WRITE, the block takeover and the journal reservation. Store v2 rejects it at this point. Half of the Store v1 rollbacks of test_blockstore_chaos become rollback-writes, so Store v1 two-phase seeds (seed % 24 in 12..17) replay differently from here on.
  3. vitastor-disk: Support JE_ROLLBACK_WRITE journal entries: dump-journal, write-journal, and the empty version's block in raw-resize and trim.
  4. Store v1: Add enable_rollback_write and journal version 3: the option, the journal upgrade, and blockstore_i::is_rollback_write_enabled() for the OSD. The trim takes the journal version from memory now.
  5. vitastor-disk: Support journal version 3, enable rollback-write for new OSDs.
  6. Store v2: Support BS_OP_ROLLBACK_WRITE: the heap entry, the block takeover and its accounting at start, metadata format 4, and the option docs for both stores. Store v2 two-phase chaos seeds (seed % 24 in 0..5) replay differently from here on.
  7. vitastor-disk: Support Store v2 rollback-write entries: dump-meta, write-meta and resize.
  8. osd: Report rollback-write support and execute it on secondaries: the SHOW_CONFIG feature and the flagged SEC_ROLLBACK.
  9. osd: Roll back peering flushes of EC objects with a new version: the OSD side above.
  10. test_pg_chaos: Add --rollback-write mode: the primary on Store v1 with the option, fake peers that report the feature and apply flagged SEC_ROLLBACKs.
  11. osd: Test a lost peering flush on EC 3+2 with pg_minsize 4.

Tested

On top of master 8582f823:

  • Every commit builds osd_test, test_blockstore_v1, test_blockstore, test_heap, test_allocator, test_blockstore_chaos, test_pg_chaos, vitastor-disk and vitastor-osd.
  • osd_test: all pass. New: test_sec_rollback_write, test_peering_flush_primary_lost (also with an OSD without the feature, as a peer and as the primary), test_peering_flush_rollback_write_lost, test_peering_flush_incomplete, test_peering_flush_waits_for_epoch, test_peering_flush_repeer_waits_for_epoch, test_peering_flush_primary_lost_minsize. On master alone test_peering_flush_primary_lost[full write] fails, as expected.
  • test_blockstore_v1: all pass. New: test_rollback_write (with and without checksums and power losses), test_rollback_write_errors, test_rollback_write_no_base (an unstable big write, a deletion and a never written object as the base), test_rollback_write_no_space (a full device, with the block takeover and a power loss), test_rollback_write_journal_space (100 unstable objects rolled back at once on a 1 MB journal, which failed with -EAGAIN while they were counted twice), test_rollback_write_over_unsynced, test_rollback_write_option (readonly open writes nothing, a trim in the upgrading session keeps version 3), test_rollback_write_v1_size_start.
  • test_blockstore: all pass. New: test_rollback_write, _errors, _disabled, _meta_format, _no_base (the takeover across power losses and compaction), _no_space (a full device, the takeover across a power loss and compaction, and the block freed once on deletion), _over_unsynced, _lost_before_sync.
  • test_heap, test_allocator: all pass.
  • test_blockstore_chaos --seeds 1-960: all pass. With ASan and UBSan, seeds 1-240: all pass, no sanitizer reports.
  • test_blockstore_chaos configurations 3, 4 and 5 (Store v2, fsync, two-phase) × seeds 100001-101000, plus --config 4 --seed 100449: 3 of 3000 fail (config 3 and config 5 seed 100796, config 4 seed 100549), all with "came back from a power outage holding unexpected content". That is the fsync race of #107: they pass with #107 applied. Master alone fails 100449 the same way and nothing else; master with #107 passes all of them. With #107, one seed of the same class is left on the branch, config 5 seed 100085. It passes with the earlier version of that fix, which also makes the flusher's fsync_buffer() wait for fsyncs in flight, and with that version the branch passes all 3001 seeds. On master with #107, configurations 3, 4 and 5 × seeds 100001-103000 all pass.
  • test_pg_chaos, default seeds 1-80: all pass.
  • test_pg_chaos --ec --rollback-write (the primary on Store v1), seeds 1-40: 12 fail; master with its primary switched to Store v1, --ec, seeds 1-40: 17 fail. Every failure of the branch is of a class that master has as well: "OSD did not finish peering", "the PG did not settle", the ack_sync() and forget_unstable_write() assertions of Store v1, and "Tried to write ... after delete, but already have". Over seeds 1-200 it's 60 failures against 79, with the same classes.

Questions

  1. An object rolled back to nothing with a cur_set role without a piece falls back to a plain rollback (the unflushed deletion case above). With two or more parity chunks, a stabilized empty version on an OSD that is absent later can then become a lone stable part:

    1. EC 2+2, OSD 4 is down. An object is written to OSDs 1, 2 and 3, unstable, nothing older. The peering rolls it back to nothing with an empty new version V on OSDs 1, 2 and 3. OSD 1 stabilizes its V and dies.
    2. OSDs 2, 3 and 4 activate the PG: the history set {1, 2, 3} has 2 >= pg_data_size OSDs present. V is unstable on 2 of 4 roles, so it's rolled back to nothing again, and OSD 4 has no piece, so it's a plain rollback.
    3. OSD 1 returns with a stable V and no other part of it: the object is incomplete.

    A fix could be to give the empty version to the roles that have pieces instead of falling back. Should it?

  2. The new version can land on OSDs outside cur_set. For a misplaced object, the pieces of the target version are taken from its object state and may be on OSDs that aren't in cur_set, but the epoch is reported with cur_set only. Is that enough, or should the history record the OSDs written to?

  3. A flag or a new opcode on the wire. SEC_ROLLBACK with OSD_OP_ROLLBACK_WRITE and the version in previously unused header bytes, guarded by the rollback_write feature. A new secondary op would need its own PG lock and statistics handling. Is the flag fine?

  4. Store v2 metadata format 4 and its name, BLOCKSTORE_META_FORMAT_HEAP_RBW.

  5. A flag bit or a new entry type in Store v2. BS_HEAP_ROLLBACK_WRITE (0x20) is a modifier of SMALL_WRITE and BIG_WRITE, so all code that handles writes handles the new entries, and only the rollback walks check the flag. A new type would have to be added to every switch over entry types.

  6. Store v1 to heap conversion. vitastor-disk's conversion of Store v1 metadata with its journal to the heap format (index_journal_by_object()) refuses rollback_write entries as unknown. As far as I can see, write_json_meta() doesn't reach that path yet. Should it convert them?

  7. The block takeover in Store v2 relies on the taken over big write not being rolled back already; when it is, or when the newest big entry is a BIG_INTENT, a new block is allocated, and -ENOSPC is still possible. That only happens after an earlier plain rollback of a newer big write of the same object. Is the shared block, counted once at start, acceptable to you, or would you prefer another way?

  8. Repeers and stops wait for the epoch report of a waiting flush batch, so a PG whose etcd doesn't answer can't repeer or stop until it does, the same as with a queued write. OK?

  9. vitastor-disk prepare enables the option by default for new OSDs, while it's off by default in the OSD. Do you want it on everywhere, or off in prepare too?

  10. test_pg_chaos --rollback-write runs the primary on Store v1, where test_pg_chaos has failures of its own. Should the mode use Store v2 now that it supports the operation?

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description An EC object can be left incomplete for good by a partially executed stabilize. The primary commits an unstable version by sending STABILIZE to every OSD separately. If it completes its own part and dies before the others arrive, the next peering, without it, sees the version unstable on fewer roles, rolls it back on the OSDs present, and nothing newer is ever written to the object. When the lost OSD returns, its stable part is the newest version, and with fewer than pg_data_size parts of it the object is incomplete. This is reachable in two ways, with a single failure of the primary: - The ordinary EC write path with `immediate_commit=all`: `submit_primary_stab_subops()` sends the local BS_OP_STABLE and the SEC_STABILIZEs in parallel. The primary dies after its local STABLE, and so does the client that was writing: nobody retries the write. - A peering flush: the same STABILIZEs sent by `submit_pg_flush_ops()` (`test_peering_flush_primary_lost` in this PR). The write was never acknowledged in either case (under `immediate_commit=all` the ack follows the stabilization), so rolling it back is legal. What is missing is something that tells the next peerings that the stable part on the lost OSD is outdated. #### Why no target threshold fixes it The next peering decides between commit and rollback by how many roles have the unstable version. A write can reach exactly as many OSDs as the threshold, and the peering without the lost primary then sees one fewer. So every rule fails some write, as the osd_test scenarios show (EC 2+1, pg_minsize 2; EC 3+2, pg_minsize 4; v2 written to the listed number of OSDs, the primary stabilizes its own part and dies; FAIL = the object is incomplete when the primary returns): | Rule for an unstable target version | Rollback-write | 2+1, v2 on 2 | 2+1, v2 on 3 | 3+2, v2 on 3 | 3+2, v2 on 4 | 3+2, v2 on 5 | |---|---|---|---|---|---|---| | n_roles >= pg_size (current, 368daa4b) | no | pass | FAIL | pass | pass | FAIL | | n_roles >= pg_data_size (before 368daa4b) | no | FAIL | pass | FAIL | pass | pass | | n_roles >= pg_minsize | no | FAIL | pass | pass | FAIL | pass | | n_roles >= pg_size | yes | pass | pass | pass | pass | pass | | n_roles >= pg_data_size | yes | pass | pass | pass | pass | pass | | n_roles >= pg_minsize | yes | pass | pass | pass | pass | pass | The PR keeps the current rule. The tests in the PR are the two 2+1 cases and the 3+2 case with v2 on all five OSDs, which the current rule gets wrong without rollback-write. #### Design Following your design: where peering would roll back an EC object, it leaves a new version under a new PG epoch, which is reported to etcd with the current OSD set first. Any part of the rolled back version left on an absent OSD is then outdated. - **Store operation `BS_OP_ROLLBACK_WRITE`.** It takes the same `{oid, version}` list as BS_OP_ROLLBACK plus `op->version`, the new version of every listed object. Each object is rolled back with the preconditions of BS_OP_ROLLBACK, and the new version is added on top as a synced, but unstable, zero-length write that refers to the data and the external bitmap of the version rolled back to, without copying them. BS_OP_STABLE commits it later like any write. It needs no journal space for data. The new version must be above every existing one (-EINVAL otherwise), and repeating the operation is a no-op. - Nothing to roll back to (no version at or below the target, or a deletion): the new version is an empty big write, like a zero-length write to an absent object. It takes over the data block of the big write it rolls back (the first write after a deletion or to an absent object is always big), so a full device doesn't make it fail where a plain rollback would succeed. A free block is only needed when there is no such write. - **Store v1**: a new journal entry `JE_ROLLBACK_WRITE` with both versions, the bitmap and the block of an empty version. Replay runs the rollback first, which frees the taken over block, then claims it for the new version. Journal space for the later commit is only reserved for objects which aren't unstable already. - **Store v2**: a single heap entry per object, a zero-length SMALL_WRITE or an empty BIG_WRITE with a new flag `BS_HEAP_ROLLBACK_WRITE`, which stores the version rolled back to at its end and rolls back every older entry above it. Both steps are persisted at once. The taken over block is shared by the new entry and the rolled back big write until both become garbage together: `mark_garbage_up_to()` already frees a block shared by neighbouring entries once, and `mark_used_blocks()` now counts it once at start. A block is only taken over from a big write which isn't rolled back yet: it's unstable then, so compaction can't free it in the meantime. - **Downgrade guards and opt-in.** The operation is only accepted with the new OSD option `enable_rollback_write`, for both stores. Turning it on upgrades the Store v1 journal to version 3, or the Store v2 metadata superblock to format 4, durably at start (not in readonly mode), and versions without this PR refuse to start with either. Without the guard, older versions would take JE_ROLLBACK_WRITE for the end of the journal, and refuse or (with `skip_corrupted_meta_entries`) drop the v2 entries. An upgraded journal or superblock stays upgraded when the option is turned off again. `vitastor-disk prepare` puts the option into the superblock of new OSDs, on by default (`--enable_rollback_write 0` turns it off); existing OSDs keep running without it until it's added with `update-sb`. - **Capability negotiation.** An OSD whose store has the option on reports `rollback_write` in the SHOW_CONFIG features, like `pg_locks`. On the wire it's SEC_ROLLBACK with a new flag `OSD_OP_ROLLBACK_WRITE` and the new version in previously unused bytes of the op header, so its payload, PG lock and statistics handling stay as they are. The primary only uses it for an object when its own store and every OSD it would send it to support it; otherwise the object is rolled back as before. - **OSD.** For an EC object with something to roll back, `submit_pg_flush_ops()` takes the pieces of its target version (from the object state, or cur_set for a clean object or one without a target version) and sends them BS_OP_ROLLBACK_WRITE to `(epoch << (64-PG_EPOCH_BITS)) | 1`, above every listed version, then stabilizes the new version once every rollback-write has succeeded. A primary that dies in between leaves an unstable new version, which the next peering commits or rolls back by the usual rule. - Every flush batch now waits until its epoch is reported with the current OSD set, like writes do, also when it has no new versions; and a batch waiting for the report holds a repeer or a stop of the PG, like a queued write does. Without the second part a repeer takes the same epoch again and writes the new version to its smaller OSD set without reporting it. - An incomplete object (fewer than pg_data_size pieces of the target version) is rolled back as before: a new version on its pieces would outdate the pieces of that version left on absent OSDs. - An object without a target version gets an empty new version only if every cur_set role has a piece of it, because an OSD without one may have an unflushed deletion there. - EC only: replicated pools keep plain rollbacks, any copy there is a whole object. #### Commits 1. `osd: Add a test for a lone stable EC part left by a peering flush on a lost primary`: `test_peering_flush_primary_lost`, EC 2+1, `immediate_commit=all`, v2 written to 2 or to 3 OSDs. On master the partial case passes and the full case fails; with the condition before 368daa4b it's the other way round. 2. `Store v1: Add BS_OP_ROLLBACK_WRITE, a rollback that assigns a new version`: the operation, JE_ROLLBACK_WRITE, the block takeover and the journal reservation. Store v2 rejects it at this point. Half of the Store v1 rollbacks of `test_blockstore_chaos` become rollback-writes, so Store v1 two-phase seeds (seed % 24 in 12..17) replay differently from here on. 3. `vitastor-disk: Support JE_ROLLBACK_WRITE journal entries`: dump-journal, write-journal, and the empty version's block in raw-resize and trim. 4. `Store v1: Add enable_rollback_write and journal version 3`: the option, the journal upgrade, and `blockstore_i::is_rollback_write_enabled()` for the OSD. The trim takes the journal version from memory now. 5. `vitastor-disk: Support journal version 3, enable rollback-write for new OSDs`. 6. `Store v2: Support BS_OP_ROLLBACK_WRITE`: the heap entry, the block takeover and its accounting at start, metadata format 4, and the option docs for both stores. Store v2 two-phase chaos seeds (seed % 24 in 0..5) replay differently from here on. 7. `vitastor-disk: Support Store v2 rollback-write entries`: dump-meta, write-meta and resize. 8. `osd: Report rollback-write support and execute it on secondaries`: the SHOW_CONFIG feature and the flagged SEC_ROLLBACK. 9. `osd: Roll back peering flushes of EC objects with a new version`: the OSD side above. 10. `test_pg_chaos: Add --rollback-write mode`: the primary on Store v1 with the option, fake peers that report the feature and apply flagged SEC_ROLLBACKs. 11. `osd: Test a lost peering flush on EC 3+2 with pg_minsize 4`. #### Tested On top of master 8582f823: - Every commit builds osd_test, test_blockstore_v1, test_blockstore, test_heap, test_allocator, test_blockstore_chaos, test_pg_chaos, vitastor-disk and vitastor-osd. - `osd_test`: all pass. New: `test_sec_rollback_write`, `test_peering_flush_primary_lost` (also with an OSD without the feature, as a peer and as the primary), `test_peering_flush_rollback_write_lost`, `test_peering_flush_incomplete`, `test_peering_flush_waits_for_epoch`, `test_peering_flush_repeer_waits_for_epoch`, `test_peering_flush_primary_lost_minsize`. On master alone `test_peering_flush_primary_lost[full write]` fails, as expected. - `test_blockstore_v1`: all pass. New: `test_rollback_write` (with and without checksums and power losses), `test_rollback_write_errors`, `test_rollback_write_no_base` (an unstable big write, a deletion and a never written object as the base), `test_rollback_write_no_space` (a full device, with the block takeover and a power loss), `test_rollback_write_journal_space` (100 unstable objects rolled back at once on a 1 MB journal, which failed with -EAGAIN while they were counted twice), `test_rollback_write_over_unsynced`, `test_rollback_write_option` (readonly open writes nothing, a trim in the upgrading session keeps version 3), `test_rollback_write_v1_size_start`. - `test_blockstore`: all pass. New: `test_rollback_write`, `_errors`, `_disabled`, `_meta_format`, `_no_base` (the takeover across power losses and compaction), `_no_space` (a full device, the takeover across a power loss and compaction, and the block freed once on deletion), `_over_unsynced`, `_lost_before_sync`. - `test_heap`, `test_allocator`: all pass. - `test_blockstore_chaos --seeds 1-960`: all pass. With ASan and UBSan, seeds 1-240: all pass, no sanitizer reports. - `test_blockstore_chaos` configurations 3, 4 and 5 (Store v2, fsync, two-phase) × seeds 100001-101000, plus `--config 4 --seed 100449`: 3 of 3000 fail (config 3 and config 5 seed 100796, config 4 seed 100549), all with "came back from a power outage holding unexpected content". That is the fsync race of #107: they pass with #107 applied. Master alone fails 100449 the same way and nothing else; master with #107 passes all of them. With #107, one seed of the same class is left on the branch, config 5 seed 100085. It passes with the earlier version of that fix, which also makes the flusher's `fsync_buffer()` wait for fsyncs in flight, and with that version the branch passes all 3001 seeds. On master with #107, configurations 3, 4 and 5 × seeds 100001-103000 all pass. - `test_pg_chaos`, default seeds 1-80: all pass. - `test_pg_chaos --ec --rollback-write` (the primary on Store v1), seeds 1-40: 12 fail; master with its primary switched to Store v1, `--ec`, seeds 1-40: 17 fail. Every failure of the branch is of a class that master has as well: "OSD did not finish peering", "the PG did not settle", the `ack_sync()` and `forget_unstable_write()` assertions of Store v1, and "Tried to write ... after delete, but already have". Over seeds 1-200 it's 60 failures against 79, with the same classes. #### Questions 1. **An object rolled back to nothing with a cur_set role without a piece** falls back to a plain rollback (the unflushed deletion case above). With two or more parity chunks, a stabilized empty version on an OSD that is absent later can then become a lone stable part: 1. EC 2+2, OSD 4 is down. An object is written to OSDs 1, 2 and 3, unstable, nothing older. The peering rolls it back to nothing with an empty new version V on OSDs 1, 2 and 3. OSD 1 stabilizes its V and dies. 2. OSDs 2, 3 and 4 activate the PG: the history set {1, 2, 3} has 2 >= pg_data_size OSDs present. V is unstable on 2 of 4 roles, so it's rolled back to nothing again, and OSD 4 has no piece, so it's a plain rollback. 3. OSD 1 returns with a stable V and no other part of it: the object is incomplete. A fix could be to give the empty version to the roles that have pieces instead of falling back. Should it? 2. **The new version can land on OSDs outside cur_set.** For a misplaced object, the pieces of the target version are taken from its object state and may be on OSDs that aren't in cur_set, but the epoch is reported with cur_set only. Is that enough, or should the history record the OSDs written to? 3. **A flag or a new opcode on the wire.** SEC_ROLLBACK with `OSD_OP_ROLLBACK_WRITE` and the version in previously unused header bytes, guarded by the `rollback_write` feature. A new secondary op would need its own PG lock and statistics handling. Is the flag fine? 4. **Store v2 metadata format 4** and its name, `BLOCKSTORE_META_FORMAT_HEAP_RBW`. 5. **A flag bit or a new entry type in Store v2.** `BS_HEAP_ROLLBACK_WRITE` (0x20) is a modifier of SMALL_WRITE and BIG_WRITE, so all code that handles writes handles the new entries, and only the rollback walks check the flag. A new type would have to be added to every switch over entry types. 6. **Store v1 to heap conversion.** vitastor-disk's conversion of Store v1 metadata with its journal to the heap format (`index_journal_by_object()`) refuses rollback_write entries as unknown. As far as I can see, `write_json_meta()` doesn't reach that path yet. Should it convert them? 7. **The block takeover in Store v2** relies on the taken over big write not being rolled back already; when it is, or when the newest big entry is a BIG_INTENT, a new block is allocated, and -ENOSPC is still possible. That only happens after an earlier plain rollback of a newer big write of the same object. Is the shared block, counted once at start, acceptable to you, or would you prefer another way? 8. **Repeers and stops wait for the epoch report** of a waiting flush batch, so a PG whose etcd doesn't answer can't repeer or stop until it does, the same as with a queued write. OK? 9. **`vitastor-disk prepare` enables the option by default** for new OSDs, while it's off by default in the OSD. Do you want it on everywhere, or off in prepare too? 10. **`test_pg_chaos --rollback-write`** runs the primary on Store v1, where test_pg_chaos has failures of its own. Should the mode use Store v2 now that it supports the operation? ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-11-17 18:52:30 +03:00
Peering commits or rolls back an unstable EC version by sending STABILIZE
or ROLLBACK to every OSD separately. If the primary completes its own part
and loses its peers before theirs arrive, the next peering, without it,
decides on the remaining unstable parts alone and may roll back a version
that is already stable on the lost OSD. When it returns, that lone stable
part is the target and the object stays incomplete.

The test runs this for a version written to 2 of 3 OSDs and to all 3.
The current condition (n_roles >= pg_size) handles the first case and
fails the second; the one before 368daa4b (n_roles >= pg_data_size)
does the reverse.
koolkhel added 2 commits 2026-11-17 20:47:02 +03:00
When peering rolls an EC object back on some OSDs while another one has
already committed its part, that lone stable part leaves the object
incomplete forever. A rollback which leaves a new version under a newer
PG epoch makes such leftovers outdated instead.

BS_OP_ROLLBACK_WRITE rolls the listed objects back like BS_OP_ROLLBACK
and puts op->version on top of each of them as a synced, but unstable,
zero-length write. The new version refers to the data and the external
bitmap of the version rolled back to without copying them, so it needs
no journal space for data, and BS_OP_STABLE commits it like any write.
If there is nothing to roll back to, the new version is an empty big
write to a newly allocated block, like a zero-length write to an absent
object. It is journaled as a new JE_ROLLBACK_WRITE entry with both
versions, the block and the bitmap. Store v2 rejects the operation with
-EINVAL for now.

Store v1 two-phase chaos seeds (seed % 24 in 12..17) replay differently
from here on.
Dump them and accept them in write-journal. Keep the data block of an
empty new version allocated in resize and trim, and remap it in resize
like the block of a big write.
koolkhel changed title from WIP: osd: Add a test for a lone stable EC part left by a peering flush on a lost primary to WIP: Roll back a peering flush under a new PG epoch 2026-11-17 20:50:35 +03:00
koolkhel force-pushed peering-flush-lone-stable from e6212f7c2f to 7dc4442f6f 2026-11-18 10:15:48 +03:00 Compare
koolkhel changed title from WIP: Roll back a peering flush under a new PG epoch to Roll back a peering flush under a new PG epoch 2026-11-18 14:54:51 +03:00
Author
Contributor

Closing for now: the bug is fixed by #131 (ac3f6f85). It keeps the OSDs holding a target version the flush stabilizes in the PG history, so a peering without them doesn't roll back a version that is already stable on one of them. Rollback-write isn't needed for it. The branch stays in case it comes back.

Closing for now: the bug is fixed by https://git.yourcmc.ru/vitalif/vitastor/pulls/131 (ac3f6f85). It keeps the OSDs holding a target version the flush stabilizes in the PG history, so a peering without them doesn't roll back a version that is already stable on one of them. Rollback-write isn't needed for it. The branch stays in case it comes back.
koolkhel closed this pull request 2026-11-19 14:16:50 +03:00

Pull request closed

Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#124
No description provided.