osd: Recalculate object state online when scrub finds its target version deleted #118
Reference in New Issue
Block a user
No description provided.
Delete Branch "koolkhel/vitastor:scrub-deleted-repeer"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
An incomplete EC object with its newest version on one OSD only, whose piece was removed with
vitastor-cli fix --check no, was forgotten by scrub: reads returned zeros in place of the removed piece, and a write made them permanent, although the previous version was still readable from the other OSDs.fixshould leave the object state correct online, without re-peering the PG.pg.target_historybefore the new history reaches etcd. An etcd event handled in between madeapply_pg_config()see the old history inpg_configand stop the PG, which was then re-peered for nothing. Here it happened on the first recovery write right after a PG state report. The write now updates thepg_confighistory at the same time, like peering does when it clears the history.SEC_READ_BMPand recalculates the object state throughmark_object(): the newest version becomes the target, older ones stay outdated, pieces which are gone are dropped. The object is forgotten only if nothing is left, and without the unbalancedderef_object_state()it did before.Tested: the new
tests/test_scrub_deleted_incomplete.sh(EC 3+2,immediate_commit=all; the other OSDs' data files are rolled back so that the newest version is left on one OSD only). It checks that the object read right afterfixand after recovery matches the previous version, and that the primary doesn't re-peer the PG. Unmodified master fails it with zeros in part 0, and a re-peer instead of the recalculation fails it too. It passes with both commits, also withOLD=1.Locally I also ran the 68
run_tests.shcases that need no QEMU, NBD, NFS or KV, withtest_scrub.shandtest_corrupt_all.shrun withoutcheck_qemuandtest_scrub.shreading the image back withvitastor-cli dd: 66 pass. The twoANTIETCD=1 test_etcd_fail.shcases hang inwait $FIO_PIDafter the etcds are restarted, and master hangs there the same way on this host (3 of 5 runs on master, 4 of 5 on this branch).Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA
77d1e7135fto0795c7609aosd: Re-peer instead of forgetting an object whose target version scrub finds deletedto WIP: osd: Recalculate object state online when scrub finds its target version deleted0795c7609atoe1f945f443WIP: osd: Recalculate object state online when scrub finds its target version deletedto osd: Recalculate object state online when scrub finds its target version deleted