osd: Pause recovery of a PG after an error instead of retrying it in a loop #143
Reference in New Issue
Block a user
No description provided.
Delete Branch "koolkhel/vitastor:osd-recovery-error-backoff"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
A recovery operation that fails with an error that doesn't drop the connection (ENOSPC from a full target OSD, EIO, EDOM) was retried at once, forever:
finish_recovery_op()restarts recovery immediately and the object stays degraded or misplaced. When the monitor doesn't mark the pool backfillfull, for example because the OSD is filled with data it doesn't account for (copies of PGs it no longer holds), the primary spins at thousands of failures per second, each logging two lines (write_stable subop ... failed on osd N: retval = -28andRecovery operation failed ...: error -28). On a 5-node test cluster with one such OSD that was ~10k log lines per second per node until the space was freed, enough to fill the system disks.Now a failed recovery operation (except EPIPE) pauses recovery of its PG for 1 second, doubling up to 60 seconds while it keeps failing, and a successful one resets the pause.
pick_next_recovery()skips paused PGs, and a timer resumes recovery when the nearest pause ends (like the scrub timer). Re-peering clears the pause. The failure line now says when the PG is retried.Tested: new
tests/test_recovery_enospc.sh(fills OSD 3 through a second pool, moves pool 1 onto it withosd_backfillfull_ratio10, counts recovery failures, then frees the space and waits for the pool to become clean). On master it fails with 128802 recovery failures in ~45 s; with the patch there are 25 (pauses 1, 2, 4, 8, 16 s) and recovery resumes by the timer afterrm-data.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA