osd: Pause recovery of a PG after an error instead of retrying it in a loop #143

Merged
vitalif merged 2 commits from koolkhel/vitastor:osd-recovery-error-backoff into master 2026-11-21 12:53:45 +03:00
Contributor

Description

A recovery operation that fails with an error that doesn't drop the connection (ENOSPC from a full target OSD, EIO, EDOM) was retried at once, forever: finish_recovery_op() restarts recovery immediately and the object stays degraded or misplaced. When the monitor doesn't mark the pool backfillfull, for example because the OSD is filled with data it doesn't account for (copies of PGs it no longer holds), the primary spins at thousands of failures per second, each logging two lines (write_stable subop ... failed on osd N: retval = -28 and Recovery operation failed ...: error -28). On a 5-node test cluster with one such OSD that was ~10k log lines per second per node until the space was freed, enough to fill the system disks.

Now a failed recovery operation (except EPIPE) pauses recovery of its PG for 1 second, doubling up to 60 seconds while it keeps failing, and a successful one resets the pause. pick_next_recovery() skips paused PGs, and a timer resumes recovery when the nearest pause ends (like the scrub timer). Re-peering clears the pause. The failure line now says when the PG is retried.

Tested: new tests/test_recovery_enospc.sh (fills OSD 3 through a second pool, moves pool 1 onto it with osd_backfillfull_ratio 10, counts recovery failures, then frees the space and waits for the pool to become clean). On master it fails with 128802 recovery failures in ~45 s; with the patch there are 25 (pauses 1, 2, 4, 8, 16 s) and recovery resumes by the timer after rm-data.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description A recovery operation that fails with an error that doesn't drop the connection (ENOSPC from a full target OSD, EIO, EDOM) was retried at once, forever: `finish_recovery_op()` restarts recovery immediately and the object stays degraded or misplaced. When the monitor doesn't mark the pool backfillfull, for example because the OSD is filled with data it doesn't account for (copies of PGs it no longer holds), the primary spins at thousands of failures per second, each logging two lines (`write_stable subop ... failed on osd N: retval = -28` and `Recovery operation failed ...: error -28`). On a 5-node test cluster with one such OSD that was ~10k log lines per second per node until the space was freed, enough to fill the system disks. Now a failed recovery operation (except EPIPE) pauses recovery of its PG for 1 second, doubling up to 60 seconds while it keeps failing, and a successful one resets the pause. `pick_next_recovery()` skips paused PGs, and a timer resumes recovery when the nearest pause ends (like the scrub timer). Re-peering clears the pause. The failure line now says when the PG is retried. Tested: new `tests/test_recovery_enospc.sh` (fills OSD 3 through a second pool, moves pool 1 onto it with `osd_backfillfull_ratio` 10, counts recovery failures, then frees the space and waits for the pool to become clean). On master it fails with 128802 recovery failures in ~45 s; with the patch there are 25 (pauses 1, 2, 4, 8, 16 s) and recovery resumes by the timer after `rm-data`. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-11-21 11:50:25 +03:00
A recovery operation that fails with an error that doesn't drop the
connection (ENOSPC from a full target OSD, EIO, EDOM) was retried at once,
forever: finish_recovery_op() restarts recovery immediately and the object
stays degraded or misplaced. When the monitor doesn't mark the pool
backfillfull, for example because the OSD is filled with data it doesn't
account for, the primary spins at thousands of failures per second, each
logging two lines.

Now a failed recovery operation (except EPIPE) pauses recovery of its PG
for 1 second, doubling up to 60 seconds while it keeps failing, and a
successful one resets the pause. pick_next_recovery() skips paused PGs and
a timer resumes recovery when the nearest pause ends. Re-peering clears it.
koolkhel added 1 commit 2026-11-21 12:30:29 +03:00
With recovery_queue_depth > 1, an operation submitted before a failure
could succeed during the pause and reset it, so the pause never grew past
1 second. Reset it only on a success after the pause ends. Count only
ENOSPC failures in test_recovery_enospc and check that the pause grows.
vitalif merged commit 5ec30ca166 into master 2026-11-21 12:53:45 +03:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#143
No description provided.