osd: Keep the holders of unstable EC versions in the PG history and accept them from pg_data_size roles #131

Open
koolkhel wants to merge 5 commits from koolkhel/vitastor:peering-flush-minsize into master
Contributor

Description

Your rule from the chat: the commit threshold for an unstable EC version must equal the history check. So it goes back to pg_data_size (it is pg_size since 368daa4b), and the OSDs holding an unstable target version go into the PG history, reported before the flush. A peering without some of those holders then doesn't activate the PG, instead of rolling back a version that is already stabilized on them. Reporting the current OSD set in place of the holders isn't enough, because the remaining OSDs are still enough of that set to activate the PG.

The cases are EC 2+1 with pg_minsize 2 and EC 3+2 with pg_minsize 4. v2 reaches the listed number of OSDs, then the primary stabilizes its own part and dies. "Degraded write" is a client write to a PG degraded to pg_minsize OSDs. Each cell shows whether the object is complete when the primary returns, then whether the PG is inactive while the primary is down:

2+1, v2 on 2 2+1, v2 on 3 3+2, v2 on 3 3+2, v2 on 4 3+2, v2 on 5 degraded write, 2+1 degraded write, 3+2
master yes / no NO / no yes / no yes / no NO / no yes / yes NO / no
this PR yes / yes yes / no yes / yes yes / no yes / no yes / yes yes / no
current set instead of holders NO / no yes / no NO / no yes / no yes / no yes / yes yes / no

The PG only stays inactive when the version reached exactly pg_data_size OSDs, until the lost holder returns or is removed.

Commits:

  1. osd: Test a lost primary of EC objects after its own part is stabilized: the cases above, plus test_peering_flush_then_holder_lost, where one holder is lost after the flush and before recovery.
  2. osd: Report the PG epoch and OSD set before peering flushes. Flush batches wait for the epoch to be reported, like writes, and a waiting batch holds a repeer or a stop. The write path's report moves to report_pg_epoch().
  3. osd: Put the holders of unstable EC target versions into the PG history.
  4. osd: Accept unstable EC versions on peering from pg_data_size roles again. In test_peering_partial_unstable_ec_rolls_back, subcases A and D now commit the chain on pg_data_size roles and put its holders in the history. B and C are unchanged.
  5. osd: Test lost primaries of degraded and misplaced EC objects.

There is no new store operation and no new on-disk format. If you take this, the rollback-write of #124 isn't needed for this bug.

Questions

  1. With 3+2 and v2 on exactly 3 OSDs, the PG now waits for the primary. Master rolls v2 back in that case. Is that acceptable?
  2. A chain on pg_data_size roles is committed again with no spare piece. If a holder is lost before recovery, the PG stays inactive rather than leaving the object incomplete. Does that match what you wanted from 368daa4b?
  3. One history set is added per distinct set of holders, all in one etcd key per PG. They stay until the PG is clean.

Tested:

  • osd_test: all pass. On master the new tests fail for 2+1 v2 on 3, 3+2 v2 on 5 and the degraded write to 3+2.
  • test_peering_flush_degraded_primary_lost (EC 2+2 and 3+2 with a role without a piece, a new object and over a stable v1) and test_peering_flush_misplaced_primary_lost (EC 2+1, a version on an OSD outside cur_set): the object is complete with this PR. On master the misplaced object with the version on all three OSDs ends incomplete.
  • test_pg_chaos seeds 1-40, 1996 and 192 more EC seeds: no failures, the same as master. Chaos runs EC 2+2 with pg_minsize 2, so it doesn't exercise this. Seed 1996 from 368daa4b passes even without the holders.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description Your rule from the chat: the commit threshold for an unstable EC version must equal the history check. So it goes back to pg_data_size (it is pg_size since 368daa4b), and the OSDs holding an unstable target version go into the PG history, reported before the flush. A peering without some of those holders then doesn't activate the PG, instead of rolling back a version that is already stabilized on them. Reporting the current OSD set in place of the holders isn't enough, because the remaining OSDs are still enough of that set to activate the PG. The cases are EC 2+1 with pg_minsize 2 and EC 3+2 with pg_minsize 4. v2 reaches the listed number of OSDs, then the primary stabilizes its own part and dies. "Degraded write" is a client write to a PG degraded to pg_minsize OSDs. Each cell shows whether the object is complete when the primary returns, then whether the PG is inactive while the primary is down: | | 2+1, v2 on 2 | 2+1, v2 on 3 | 3+2, v2 on 3 | 3+2, v2 on 4 | 3+2, v2 on 5 | degraded write, 2+1 | degraded write, 3+2 | |---|---|---|---|---|---|---|---| | master | yes / no | NO / no | yes / no | yes / no | NO / no | yes / yes | NO / no | | this PR | yes / yes | yes / no | yes / yes | yes / no | yes / no | yes / yes | yes / no | | current set instead of holders | NO / no | yes / no | NO / no | yes / no | yes / no | yes / yes | yes / no | The PG only stays inactive when the version reached exactly pg_data_size OSDs, until the lost holder returns or is removed. Commits: 1. `osd: Test a lost primary of EC objects after its own part is stabilized`: the cases above, plus `test_peering_flush_then_holder_lost`, where one holder is lost after the flush and before recovery. 2. `osd: Report the PG epoch and OSD set before peering flushes`. Flush batches wait for the epoch to be reported, like writes, and a waiting batch holds a repeer or a stop. The write path's report moves to `report_pg_epoch()`. 3. `osd: Put the holders of unstable EC target versions into the PG history`. 4. `osd: Accept unstable EC versions on peering from pg_data_size roles again`. In `test_peering_partial_unstable_ec_rolls_back`, subcases A and D now commit the chain on pg_data_size roles and put its holders in the history. B and C are unchanged. 5. `osd: Test lost primaries of degraded and misplaced EC objects`. There is no new store operation and no new on-disk format. If you take this, the rollback-write of https://git.yourcmc.ru/vitalif/vitastor/pulls/124 isn't needed for this bug. ### Questions 1. With 3+2 and v2 on exactly 3 OSDs, the PG now waits for the primary. Master rolls v2 back in that case. Is that acceptable? 2. A chain on pg_data_size roles is committed again with no spare piece. If a holder is lost before recovery, the PG stays inactive rather than leaving the object incomplete. Does that match what you wanted from 368daa4b? 3. One history set is added per distinct set of holders, all in one etcd key per PG. They stay until the PG is clean. Tested: - `osd_test`: all pass. On master the new tests fail for 2+1 v2 on 3, 3+2 v2 on 5 and the degraded write to 3+2. - `test_peering_flush_degraded_primary_lost` (EC 2+2 and 3+2 with a role without a piece, a new object and over a stable v1) and `test_peering_flush_misplaced_primary_lost` (EC 2+1, a version on an OSD outside cur_set): the object is complete with this PR. On master the misplaced object with the version on all three OSDs ends incomplete. - `test_pg_chaos` seeds 1-40, 1996 and 192 more EC seeds: no failures, the same as master. Chaos runs EC 2+2 with pg_minsize 2, so it doesn't exercise this. Seed 1996 from 368daa4b passes even without the holders. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 5 commits 2026-11-19 09:58:47 +03:00
A primary stabilizes an EC object version by sending STABILIZE to each
OSD separately, both in a peering flush and after a write with
immediate_commit=all. If it completes its own part and loses its peers
before theirs arrive, the next peering, without it, decides on the
remaining unstable parts alone. If it rolls the version back, the stable
part brought back by the primary is a lone stable part, and the object
stays incomplete.

Run this for a peering flush of a version written to some or all OSDs
of EC 2+1 with pg_minsize 2 and EC 3+2 with pg_minsize 4, and for a
write to a degraded PG of pg_minsize OSDs. The object must be complete
when the primary returns, and the PG must stay active while the primary
is away whenever the others hold enough parts to decide.

The current rule (n_roles >= pg_size) fails EC 2+1 with the version on
all 3 OSDs, EC 3+2 with the version on all 5, and the degraded write to
4 OSDs of EC 3+2.
Since 368daa4b, peering targets an unstable EC version only if it is on
all pg_size roles. A primary that stabilizes its own part of a version
written to all OSDs and dies leaves the others with pg_size-1 unstable
parts, they roll the version back, and the stable part brought back by
the primary makes the object incomplete.

Target an unstable version from pg_minsize roles, the number of OSDs a
write must reach to be acknowledged. A partial chain of a failed write
on fewer than pg_minsize roles is still rolled back. Move the 368daa4b
test to pg_minsize 3: with pg_minsize equal to pg_data_size its partial
chains reach pg_minsize and are committed.

This alone still fails when the version is on exactly pg_minsize roles:
without the primary the others see one role less and roll it back.
A peering flush commits or rolls back unstable versions without
reporting anything to etcd, so the history doesn't show that the
current OSD set has changed the objects.

Like writes, make every flush batch wait until the PG epoch is reported
with the current OSD set in the history. A batch waiting for it holds a
repeer or a stop of the PG like a write does, and is dropped if the PG
is no longer active when the epoch is reported.

The current OSD set alone doesn't stop the next peering after a lost
primary: the others are still enough of it to activate the PG.
A flush stabilizes the pieces of an unstable target version one OSD at
a time. If the primary stabilizes its own piece and dies, the others see
the version on fewer roles and may roll it back, leaving a lone stable
piece on the primary.

When peering targets an EC version that is unstable somewhere, add the
set of OSDs holding its pieces to the PG history. It is reported with
the epoch before the first write or flush, and a later peering refuses
to activate the PG while fewer than pg_data_size of them are up.

With pg_minsize above pg_data_size this is not enough: a version on
pg_minsize OSDs leaves pg_minsize-1 of them, still pg_data_size or more,
and they roll it back.
Peering commits an unstable EC version from pg_minsize roles, but the
history check let a PG activate with pg_data_size OSDs of a history
set. With pg_minsize above pg_data_size, a version written to exactly
pg_minsize OSDs, stabilized on one of them and then lost with it, is
seen on fewer than pg_minsize roles by the others and rolled back,
leaving a lone stable part. This is the case both for a peering flush
and for a write to a degraded PG of pg_minsize OSDs.

For EC pools, refuse to activate the PG while fewer than pg_minsize
OSDs of a history set (or all of a smaller one) are up, the same
threshold peering commits with. Replicated pools keep pg_data_size,
because any copy there is a whole object.

The PG now stays inactive in these cases until the lost OSD returns,
and a history set of pg_minsize OSDs, kept until the PG is clean, makes
any of them a single point of failure for the PG until then.
koolkhel changed title from osd: Accept unstable EC versions from pg_minsize roles and keep their holders in the PG history to osd: Keep the holders of unstable EC versions in the PG history and accept them from pg_data_size roles 2026-11-19 10:15:50 +03:00
koolkhel force-pushed peering-flush-minsize from e93600dcf4 to 960b39ef30 2026-11-19 10:15:51 +03:00 Compare
koolkhel added 1 commit 2026-11-19 10:24:57 +03:00
Run the lost primary scenario for a version written to all OSDs of a
degraded PG, EC 2+2 with one OSD down and EC 3+2 with pg_minsize 4 and
one OSD down, both over an older stable version and as a new object
with no pieces on the missing OSD. The object must end complete or
absent, never incomplete with a lone stable part.

Also run it for a misplaced object: the PG has moved from OSDs 1, 2, 3
to 1, 2, 4 and the unstable version has a piece on OSD 3, outside the
current set. Its holders, OSD 3 included, must go to the history, and
the object must be complete when the primary returns.

Let the EC test helpers take an OSD set and a piece role per OSD.
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u peering-flush-minsize:koolkhel-peering-flush-minsize
git checkout koolkhel-peering-flush-minsize
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#131
No description provided.