osd: Keep the holders of unstable EC versions in the PG history and accept them from pg_data_size roles #131
Reference in New Issue
Block a user
No description provided.
Delete Branch "koolkhel/vitastor:peering-flush-minsize"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
Your rule from the chat: the commit threshold for an unstable EC version must equal the history check. So it goes back to pg_data_size (it is pg_size since
368daa4b), and the OSDs holding an unstable target version go into the PG history. The flush waits until that history is reported, like a write does. A peering without some of those holders then doesn't activate the PG, instead of rolling back a version that is already stabilized on them.The threshold alone isn't enough: with any threshold, a version on exactly that many OSDs is rolled back by a peering without a primary that has already stabilized its own part. On master that is v2 on all OSDs, with pg_data_size it is v2 on pg_data_size OSDs.
The cases are EC 2+1 with pg_minsize 2 and EC 3+2 with pg_minsize 4. v2 reaches the listed number of OSDs, then the primary stabilizes its own part and dies. "Degraded write" is a client write to a PG degraded to pg_minsize OSDs. Each cell shows whether the object is complete when the primary returns, then whether the PG is inactive while the primary is down:
The PG only stays inactive when the version reached exactly pg_data_size OSDs, until the lost holder returns or is removed.
Commits:
osd: Test a lost primary of EC objects after its own part is stabilized: the cases above, plustest_peering_flush_then_holder_lost, where one holder is lost after the flush and before recovery (including your 3+2 case with v2 on OSDs 1, 2, 3 and OSD 2 lost).osd: Put the holders of unstable target versions into the PG history. The holders are added where the flush action getsstable_toequal to the target version, for replicated pools too. Flush batches wait until the history is reported with the epoch that peering already bumps, and a waiting batch holds a repeer or a stop like a write. The write path is unchanged.osd: Accept unstable EC versions on peering from pg_data_size roles again. Intest_peering_partial_unstable_ec_rolls_back, subcases A and D now commit the chain on pg_data_size roles and put its holders in the history. B and C are unchanged.osd: Test lost primaries of degraded and misplaced EC objects.There is no new store operation and no new on-disk format. If you take this, the rollback-write of #124 isn't needed for this bug.
Tested:
osd_test: all pass. On master the new tests fail for 2+1 v2 on 3, 3+2 v2 on 5 and the degraded write to 3+2.test_peering_flush_degraded_primary_lost(EC 2+2 and 3+2 with a role without a piece, a new object and over a stable v1) andtest_peering_flush_misplaced_primary_lost(EC 2+1, a version on an OSD outside cur_set): the object is complete with this PR. On master the misplaced object with the version on all three OSDs ends incomplete.test_peering_flush_primary_lostfor 2+1 v2 on 2 and 3+2 v2 on 3, andtest_peering_flush_then_holder_lostfor 2+1 v2 on 2 and 3+2 v2 on 3 (the PG activates without the lost holder with the object incomplete). Holders in the history without the flush waiting for the report fail the same three: the next primary reads the history from etcd.test_pg_chaosseeds 1-40, 1996 and 192 more EC seeds: no failures, the same as master. Chaos runs EC 2+2 with pg_minsize 2, so it doesn't exercise this. Seed 1996 from368daa4bpasses even without the holders.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA
osd: Accept unstable EC versions from pg_minsize roles and keep their holders in the PG historyto osd: Keep the holders of unstable EC versions in the PG history and accept them from pg_data_size rolese93600dcf4to960b39ef300cc775ba33to0e02869bca0e02869bcatoc038bf6804