osd: Stabilize unstable writes of a removed pool before stopping its PGs #115

koolkhel · 2026-11-02T17:13:12+03:00

koolkhel commented

2026-11-02 17:13:12 +03:00

Description

This replaces #106 (#106): instead of each OSD rolling back its own store, the primary stabilizes the writes before it lets the PG go, so only the primary modifies data, as before.

stop_pg() forgets the PG's unstable writes (drop_dirty_pg_connections()) and leaves them to the next primary, which stabilizes or rolls them back during peering. But nobody peers a removed pool again, so its unstable writes stay on the OSDs forever. The old store can't trim its journal past them: the journal fills up and writes to all other pools on the same OSDs fail with -EAGAIN, restarts don't help. Only EC/XOR pools leave such writes, replicated ones write stable.

The monitor already pauses PGs of a removed pool and waits for them to go offline before it forgets them (stop_all_pgs()). Now, when the primary sees its PG paused and the pool gone from /config/pools, it:

keeps the PG active and locked, but refuses new writes and deletes with -EPIPE (reads still work), and doesn't pick recovery or scrub work from it;
waits for the operations in flight;
runs an ordinary self sync (like autosync), which stabilizes the writes on all peers;
stops the PG as usual, whatever the sync result. A failure is logged, and the writes are then forgotten as before.

The sync runs before the PG leaves the active state on purpose: secondaries accept STABILIZE only from the PG lock holder, the locks are released when the PG goes to stopping, and the next primary can't take the PG before this one reports it offline, so there is no race with a new primary's LIST.

Pauses of existing pools (PG count changes) stop at once, exactly as before: their next primaries peer them anyway.

A stuck peer can't block the stop for good: a dead or silent peer is disconnected by the keepalive (osd_idle_timeout + osd_ping_timeout, 10 s by default), which fails the sync with EPIPE. An operation stuck in a live peer's store holds the stop the same way any in-flight write already holds stop_pg() (can_stop() needs inflight == 0).

Not covered: a primary that is dead when the pool is removed, an OSD that was down during the removal and comes back with unstable writes, a pool whose PGs never become active (incomplete), and a journal that is already full, where the stabilize itself gets -EAGAIN.

Tested:

osd_test: new test_pause_stabilizes_before_stop, test_unpause_cancels_pause_sync, test_pause_sync_waits_for_inflight_write, test_pause_of_existing_pool_stops_at_once
new tests/test_rm_pool_unstable.sh (old store, EC 2+1, writes with autosync off and no client sync, then rm-pool, then 4x the journal size to another pool): fails on master with "Pool removed: writes to another pool got stuck", passes with the patch, no unstable writes of the removed pool left in the journals
test_change_pg_count.sh and test_change_pg_count_online.sh (EC, also with OLD=1), test_interrupted_rebalance.sh (EC, also with OLD=1), OLD=1 test_move_reappear.sh: pass, no pause syncs for PG count changes. An earlier version that synced on every pause also passed the PG count change tests with up to 105 syncs per run.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md

### Description This replaces #106 (https://git.yourcmc.ru/vitalif/vitastor/pulls/106): instead of each OSD rolling back its own store, the primary stabilizes the writes before it lets the PG go, so only the primary modifies data, as before. `stop_pg()` forgets the PG's unstable writes (`drop_dirty_pg_connections()`) and leaves them to the next primary, which stabilizes or rolls them back during peering. But nobody peers a removed pool again, so its unstable writes stay on the OSDs forever. The old store can't trim its journal past them: the journal fills up and writes to all other pools on the same OSDs fail with -EAGAIN, restarts don't help. Only EC/XOR pools leave such writes, replicated ones write stable. The monitor already pauses PGs of a removed pool and waits for them to go offline before it forgets them (`stop_all_pgs()`). Now, when the primary sees its PG paused and the pool gone from `/config/pools`, it: 1. keeps the PG active and locked, but refuses new writes and deletes with -EPIPE (reads still work), and doesn't pick recovery or scrub work from it; 2. waits for the operations in flight; 3. runs an ordinary self sync (like autosync), which stabilizes the writes on all peers; 4. stops the PG as usual, whatever the sync result. A failure is logged, and the writes are then forgotten as before. The sync runs before the PG leaves the active state on purpose: secondaries accept STABILIZE only from the PG lock holder, the locks are released when the PG goes to stopping, and the next primary can't take the PG before this one reports it offline, so there is no race with a new primary's LIST. Pauses of existing pools (PG count changes) stop at once, exactly as before: their next primaries peer them anyway. A stuck peer can't block the stop for good: a dead or silent peer is disconnected by the keepalive (`osd_idle_timeout` + `osd_ping_timeout`, 10 s by default), which fails the sync with EPIPE. An operation stuck in a live peer's store holds the stop the same way any in-flight write already holds `stop_pg()` (`can_stop()` needs `inflight == 0`). Not covered: a primary that is dead when the pool is removed, an OSD that was down during the removal and comes back with unstable writes, a pool whose PGs never become active (incomplete), and a journal that is already full, where the stabilize itself gets -EAGAIN. Tested: - `osd_test`: new `test_pause_stabilizes_before_stop`, `test_unpause_cancels_pause_sync`, `test_pause_sync_waits_for_inflight_write`, `test_pause_of_existing_pool_stops_at_once` - new `tests/test_rm_pool_unstable.sh` (old store, EC 2+1, writes with autosync off and no client sync, then `rm-pool`, then 4x the journal size to another pool): fails on master with "Pool removed: writes to another pool got stuck", passes with the patch, no unstable writes of the removed pool left in the journals - `test_change_pg_count.sh` and `test_change_pg_count_online.sh` (EC, also with `OLD=1`), `test_interrupted_rebalance.sh` (EC, also with `OLD=1`), `OLD=1 test_move_reappear.sh`: pass, no pause syncs for PG count changes. An earlier version that synced on every pause also passed the PG count change tests with up to 105 syncs per run. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md

koolkhel added 1 commit 2026-11-02 17:13:13 +03:00

osd: Stabilize unstable writes of a removed pool before stopping its PGs 66c8d61323

stop_pg() forgets the PG's unstable writes and leaves them to the next
primary, which stabilizes or rolls them back during peering. But nobody
peers a removed pool again, so its unstable writes stay on the OSDs
forever. The old store can't trim its journal past them: the journal
fills up and writes to all other pools on the same OSDs fail with
-EAGAIN, restarts don't help.

The monitor pauses PGs of a removed pool and waits for them to go
offline before it forgets them. Make the primary of such a PG stabilize
its unstable writes first, while the PG is still active and locked: it
refuses new writes, waits for the ones in flight, runs a sync, and then
stops the PG whatever the result. Pauses of existing pools (PG count
changes) stop at once as before.

koolkhel referenced this pull request

2026-11-02 17:13:58 +03:00

osd: Roll back unstable writes of removed pools in the old store #106

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u pause-sync-before-stop:koolkhel-pause-sync-before-stop

git checkout koolkhel-pause-sync-before-stop

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vitalif/vitastor#115

Allow edits from maintainers