osd: Recalculate object state online when scrub finds its target version deleted #118

Merged
vitalif merged 2 commits from koolkhel/vitastor:scrub-deleted-repeer into master 2026-11-17 10:20:48 +03:00
Contributor

Description

An incomplete EC object with its newest version on one OSD only, whose piece was removed with vitastor-cli fix --check no, was forgotten by scrub: reads returned zeros in place of the removed piece, and a write made them permanent, although the previous version was still readable from the other OSDs. fix should leave the object state correct online, without re-peering the PG.

  1. osd: Keep pg_config history in sync when a write adds the current OSD set. The first write to a PG which isn't clean adds the current OSD set to pg.target_history before the new history reaches etcd. An etcd event handled in between made apply_pg_config() see the old history in pg_config and stop the PG, which was then re-peered for nothing. Here it happened on the first recovery write right after a PG state report. The write now updates the pg_config history at the same time, like peering does when it clears the history.
  2. osd: Recalculate object state online when scrub finds its target version deleted. Peering now lists every OSD holding any version of the object in its state, once per role: older pieces on their target OSDs were left out before. When scrub finds no piece of the target version but outdated pieces are left, it asks their OSDs for versions with SEC_READ_BMP and recalculates the object state through mark_object(): the newest version becomes the target, older ones stay outdated, pieces which are gone are dropped. The object is forgotten only if nothing is left, and without the unbalanced deref_object_state() it did before.

Tested: the new tests/test_scrub_deleted_incomplete.sh (EC 3+2, immediate_commit=all; the other OSDs' data files are rolled back so that the newest version is left on one OSD only). It checks that the object read right after fix and after recovery matches the previous version, and that the primary doesn't re-peer the PG. Unmodified master fails it with zeros in part 0, and a re-peer instead of the recalculation fails it too. It passes with both commits, also with OLD=1.

Locally I also ran the 68 run_tests.sh cases that need no QEMU, NBD, NFS or KV, with test_scrub.sh and test_corrupt_all.sh run without check_qemu and test_scrub.sh reading the image back with vitastor-cli dd: 66 pass. The two ANTIETCD=1 test_etcd_fail.sh cases hang in wait $FIO_PID after the etcds are restarted, and master hangs there the same way on this host (3 of 5 runs on master, 4 of 5 on this branch).

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description An incomplete EC object with its newest version on one OSD only, whose piece was removed with `vitastor-cli fix --check no`, was forgotten by scrub: reads returned zeros in place of the removed piece, and a write made them permanent, although the previous version was still readable from the other OSDs. `fix` should leave the object state correct online, without re-peering the PG. 1. **osd: Keep pg_config history in sync when a write adds the current OSD set.** The first write to a PG which isn't clean adds the current OSD set to `pg.target_history` before the new history reaches etcd. An etcd event handled in between made `apply_pg_config()` see the old history in `pg_config` and stop the PG, which was then re-peered for nothing. Here it happened on the first recovery write right after a PG state report. The write now updates the `pg_config` history at the same time, like peering does when it clears the history. 2. **osd: Recalculate object state online when scrub finds its target version deleted.** Peering now lists every OSD holding any version of the object in its state, once per role: older pieces on their target OSDs were left out before. When scrub finds no piece of the target version but outdated pieces are left, it asks their OSDs for versions with `SEC_READ_BMP` and recalculates the object state through `mark_object()`: the newest version becomes the target, older ones stay outdated, pieces which are gone are dropped. The object is forgotten only if nothing is left, and without the unbalanced `deref_object_state()` it did before. Tested: the new `tests/test_scrub_deleted_incomplete.sh` (EC 3+2, `immediate_commit=all`; the other OSDs' data files are rolled back so that the newest version is left on one OSD only). It checks that the object read right after `fix` and after recovery matches the previous version, and that the primary doesn't re-peer the PG. Unmodified master fails it with zeros in part 0, and a re-peer instead of the recalculation fails it too. It passes with both commits, also with `OLD=1`. Locally I also ran the 68 `run_tests.sh` cases that need no QEMU, NBD, NFS or KV, with `test_scrub.sh` and `test_corrupt_all.sh` run without `check_qemu` and `test_scrub.sh` reading the image back with `vitastor-cli dd`: 66 pass. The two `ANTIETCD=1 test_etcd_fail.sh` cases hang in `wait $FIO_PID` after the etcds are restarted, and master hangs there the same way on this host (3 of 5 runs on master, 4 of 5 on this branch). ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-11-16 19:21:51 +03:00
When scrub found no chunk of an unclean object's target version, it
removed the object from the PG state. But osd_set doesn't list older
versions of chunks kept on their target OSDs, so these may still exist.
An incomplete EC object with its newest version on one OSD only, whose
chunk was removed with fix --check no, then became clean: reads
returned zeros in place of that chunk, and a write made them permanent,
although the previous version was still readable from the other chunks.

Re-peer the PG instead, so that peering rebuilds the object state from
what the OSDs actually hold. This also drops a deref_object_state()
without a matching reference: scrub doesn't take one.
koolkhel force-pushed scrub-deleted-repeer from 77d1e7135f to 0795c7609a 2026-11-16 19:31:15 +03:00 Compare
koolkhel changed title from osd: Re-peer instead of forgetting an object whose target version scrub finds deleted to WIP: osd: Recalculate object state online when scrub finds its target version deleted 2026-11-16 20:26:46 +03:00
koolkhel force-pushed scrub-deleted-repeer from 0795c7609a to e1f945f443 2026-11-16 20:26:48 +03:00 Compare
koolkhel changed title from WIP: osd: Recalculate object state online when scrub finds its target version deleted to osd: Recalculate object state online when scrub finds its target version deleted 2026-11-16 21:21:00 +03:00
vitalif approved these changes 2026-11-17 10:20:38 +03:00
vitalif merged commit e1f945f443 into master 2026-11-17 10:20:48 +03:00
Sign in to join this conversation.
No Reviewers
No Label
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#118
No description provided.