WIP: Roll back a peering flush under a new PG epoch #124

Draft
koolkhel wants to merge 3 commits from koolkhel/vitastor:peering-flush-lone-stable into master
Contributor

Description

A peering flush can leave an EC object incomplete for good. If the primary stabilizes its own part and loses its peers before theirs arrive, the next peering, without it, can roll back the remaining unstable parts of a version that is already stable on the lost OSD. When that OSD returns, its lone stable part is the newest version and the object stays incomplete. The condition from 368daa4b moves this window rather than closing it.

Work in progress, following your design: the rollback leaves a new version under a new PG epoch, so the lone stable part becomes outdated. This PR so far has the store side for Store v1:

  1. osd: Add a test for a lone stable EC part left by a peering flush on a lost primary: test_peering_flush_primary_lost, EC 2+1, immediate_commit=all. On master the partial-write case passes and the full-write case fails at assert(!incomplete). With the pre-368daa4b condition it's the other way round.
  2. Store v1: Add BS_OP_ROLLBACK_WRITE, a rollback that assigns a new version.
    • A new opcode, because existing BS_OP_ROLLBACK callers don't set op->version. It takes the same {oid, version} list as BS_OP_ROLLBACK, plus op->version as the new version for every listed object.
    • Each object is rolled back with the same preconditions as BS_OP_ROLLBACK. The new version is then added on top as a synced, unstable, zero-length small write that keeps the data and bitmap of the version rolled back to. BS_OP_STABLE commits it later.
    • It is journaled as a new JE_ROLLBACK_WRITE entry with both versions, the bitmap and, for an empty version (below), its data block. It needs no journal space for data.
    • The version rolled back to is synced first: the new entry must not reach the journal before its base. The new version must be newer than every existing version, otherwise -EINVAL; repeating the op is a no-op.
    • With nothing to roll back to (no version at or below the target, or a delete), the new version is an empty zero-length big write to a newly allocated block, the same thing Store v1 does for a zero-length write to an absent object. This is the one case that needs a free data block: on a full device the op waits for the flusher like a big write, or fails with -ENOSPC where a plain rollback would succeed.
    • Store v2 rejects the op for now.
  3. vitastor-disk: Support JE_ROLLBACK_WRITE journal entries: dump-journal and write-journal, plus raw-resize and trim, which treat the empty version's block like a big write's.

Still to do: the OSD side (allocate and report the new epoch with the current OSD set before the flush's rollbacks, send the new op, then commit the new versions), the network op and its capability flag, and Store v2. Older versions treat the new journal entry as the end of the journal, so the behaviour will need an opt-in flag.

Tested:

  • test_blockstore_v1: all pass, with new tests test_rollback_write (with and without checksums, with and without simulated power loss after each step), test_rollback_write_no_base (an unstable big write, a delete and a never-written object as the base), test_rollback_write_no_space, test_rollback_write_errors and test_rollback_write_over_unsynced.
  • test_blockstore_chaos --seeds 1-960: all pass. The chaos test now turns half of the Store v1 rollbacks into BS_OP_ROLLBACK_WRITE, so Store v1 two-phase seeds replay differently from here on. Its model doesn't reach the no-base case; the unit tests cover it.
  • vitastor-disk: a journal with both kinds of entry survives a dump-journal/write-journal round trip, and raw-resize remaps the empty version's block.
  • osd_test: still fails only at test_peering_flush_primary_lost[full write], as expected until the OSD side is done.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description A peering flush can leave an EC object incomplete for good. If the primary stabilizes its own part and loses its peers before theirs arrive, the next peering, without it, can roll back the remaining unstable parts of a version that is already stable on the lost OSD. When that OSD returns, its lone stable part is the newest version and the object stays incomplete. The condition from 368daa4b moves this window rather than closing it. Work in progress, following your design: the rollback leaves a new version under a new PG epoch, so the lone stable part becomes outdated. This PR so far has the store side for Store v1: 1. `osd: Add a test for a lone stable EC part left by a peering flush on a lost primary`: `test_peering_flush_primary_lost`, EC 2+1, `immediate_commit=all`. On master the partial-write case passes and the full-write case fails at `assert(!incomplete)`. With the pre-368daa4b condition it's the other way round. 2. `Store v1: Add BS_OP_ROLLBACK_WRITE, a rollback that assigns a new version`. - A new opcode, because existing BS_OP_ROLLBACK callers don't set `op->version`. It takes the same `{oid, version}` list as BS_OP_ROLLBACK, plus `op->version` as the new version for every listed object. - Each object is rolled back with the same preconditions as BS_OP_ROLLBACK. The new version is then added on top as a synced, unstable, zero-length small write that keeps the data and bitmap of the version rolled back to. BS_OP_STABLE commits it later. - It is journaled as a new `JE_ROLLBACK_WRITE` entry with both versions, the bitmap and, for an empty version (below), its data block. It needs no journal space for data. - The version rolled back to is synced first: the new entry must not reach the journal before its base. The new version must be newer than every existing version, otherwise -EINVAL; repeating the op is a no-op. - With nothing to roll back to (no version at or below the target, or a delete), the new version is an empty zero-length big write to a newly allocated block, the same thing Store v1 does for a zero-length write to an absent object. This is the one case that needs a free data block: on a full device the op waits for the flusher like a big write, or fails with -ENOSPC where a plain rollback would succeed. - Store v2 rejects the op for now. 3. `vitastor-disk: Support JE_ROLLBACK_WRITE journal entries`: dump-journal and write-journal, plus raw-resize and trim, which treat the empty version's block like a big write's. Still to do: the OSD side (allocate and report the new epoch with the current OSD set before the flush's rollbacks, send the new op, then commit the new versions), the network op and its capability flag, and Store v2. Older versions treat the new journal entry as the end of the journal, so the behaviour will need an opt-in flag. Tested: - `test_blockstore_v1`: all pass, with new tests `test_rollback_write` (with and without checksums, with and without simulated power loss after each step), `test_rollback_write_no_base` (an unstable big write, a delete and a never-written object as the base), `test_rollback_write_no_space`, `test_rollback_write_errors` and `test_rollback_write_over_unsynced`. - `test_blockstore_chaos --seeds 1-960`: all pass. The chaos test now turns half of the Store v1 rollbacks into BS_OP_ROLLBACK_WRITE, so Store v1 two-phase seeds replay differently from here on. Its model doesn't reach the no-base case; the unit tests cover it. - `vitastor-disk`: a journal with both kinds of entry survives a dump-journal/write-journal round trip, and raw-resize remaps the empty version's block. - `osd_test`: still fails only at `test_peering_flush_primary_lost[full write]`, as expected until the OSD side is done. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-11-17 18:52:30 +03:00
Peering commits or rolls back an unstable EC version by sending STABILIZE
or ROLLBACK to every OSD separately. If the primary completes its own part
and loses its peers before theirs arrive, the next peering, without it,
decides on the remaining unstable parts alone and may roll back a version
that is already stable on the lost OSD. When it returns, that lone stable
part is the target and the object stays incomplete.

The test runs this for a version written to 2 of 3 OSDs and to all 3.
The current condition (n_roles >= pg_size) handles the first case and
fails the second; the one before 368daa4b (n_roles >= pg_data_size)
does the reverse.
koolkhel added 2 commits 2026-11-17 20:47:02 +03:00
When peering rolls an EC object back on some OSDs while another one has
already committed its part, that lone stable part leaves the object
incomplete forever. A rollback which leaves a new version under a newer
PG epoch makes such leftovers outdated instead.

BS_OP_ROLLBACK_WRITE rolls the listed objects back like BS_OP_ROLLBACK
and puts op->version on top of each of them as a synced, but unstable,
zero-length write. The new version refers to the data and the external
bitmap of the version rolled back to without copying them, so it needs
no journal space for data, and BS_OP_STABLE commits it like any write.
If there is nothing to roll back to, the new version is an empty big
write to a newly allocated block, like a zero-length write to an absent
object. It is journaled as a new JE_ROLLBACK_WRITE entry with both
versions, the block and the bitmap. Store v2 rejects the operation with
-EINVAL for now.

Store v1 two-phase chaos seeds (seed % 24 in 12..17) replay differently
from here on.
Dump them and accept them in write-journal. Keep the data block of an
empty new version allocated in resize and trim, and remap it in resize
like the block of a big write.
koolkhel changed title from WIP: osd: Add a test for a lone stable EC part left by a peering flush on a lost primary to WIP: Roll back a peering flush under a new PG epoch 2026-11-17 20:50:35 +03:00
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u peering-flush-lone-stable:koolkhel-peering-flush-lone-stable
git checkout koolkhel-peering-flush-lone-stable
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#124
No description provided.