osd: use lower_bound to find head operation in pg_cancel_write_queue

std::multimap::find() is not guaranteed by the C++ standard to return the
first element of a range with equivalent keys, which may fail to retrieve
the head operation in pg.write_queue.

Switch to lower_bound() to reliably locate the head of the queue in O(log N)
time, and add an explicit check for it->first == oid to prevent falsely
matching a subsequent larger key when the target oid is missing.

docs/tests/pg-chaos.en.md

@@ -0,0 +1,213 @@
+# The PG chaos-test model (`src/test/test_pg_chaos.cpp`)
+
+A seed-driven stress test of the PG machinery: ONE real `osd_t` (the primary) with a
+real blockstore over a mock disk (entirely in RAM, `ringloop_mock`) owns a single PG
+(number 1, pool 1, four 128 K objects, 4 K granularity). The other PG members are
+fake secondary OSDs inside the test. The pool is either replicated x3 (members 2,3,
+spare 4) or EC 2+2 (members 2..4, spare 5, one object = one parity stripe). The seed
+uniquely determines the configuration and the whole event sequence; a failing run is
+reproduced with the same `--seed`.
+
+What it mainly verifies is the PG machinery (the subop fan-out,
+peering/LIST/rollback/stabilize, repeer daemons, stop_pg, INCOMPLETE, recovery) and
+partly the primary OSD's storage; checking the storage is not the test's main goal -
+test_blockstore_chaos covers it.
+
+## Configurations (10)
+
+- replicated/EC ×
+- immediate_commit none/all ×
+- csum_block none/4096/16384
+
+`--recovery` enables recovery/rebalance after a set change (off by default so that
+degraded states get exercised).
+
+## The world model
+
+Each object has an ordered version history; a version enters the model at SUBMIT
+time (the delta = the request's exact bytes/length/granule bitmap), with a state:
+- `V_SUBMITTED` - the op was sent and has no answer yet (in flight). It behaves
+  like V_OPTIONAL in every respect but one: concurrent reads are entitled not to
+  show this write's result, and that must not mark the version DEAD.
+- `V_OPTIONAL` - the write was sent but not acknowledged: it either applied (then
+  as a whole) or did not apply at all. Versions reach this state from V_SUBMITTED
+  and, in the explicit-fsync modes, from V_ACKED when the client session dies
+  before a SYNC request (that counts as a marker that the write is unacknowledged).
+  Observing an OPTIONAL write's effect in a later read restores it to ACKED;
+  observing its absence (not visible over its own range) makes it DEAD.
+- `V_ACKED` - the write was acknowledged; the model records `ack_live` (the live set
+  at ack time - used to check that the failure model wasn't exceeded). With
+  immediate_commit=all it is immediately considered on-disk durable. An ACKED write
+  must not roll back while the client's session is alive.
+- `V_SYNCED` - acknowledged and SYNCed: durable on disk, and it may never roll back
+  again at all.
+- `V_PARTIAL` - deleted, but possibly not from every OSD (the PG stayed LEFT_ON_DEAD).
+- `V_DEAD` - the version was either definitely refused (-EAGAIN, possibly -ENOSPC in
+  the future), or it was OPTIONAL and it later became certain that it was refused
+  because a subsequent read did NOT observe it. A DEAD version showing again fails
+  the test.
+
+Object contents are self-identifying: the pattern encodes the object, the version and
+the offset, so a read result decodes back into "which version wrote these bytes".
+
+Verification reads ONLY through the primary OSD and checks that the answer is
+explained by a merge of ONE consistent subset of the written versions, where every
+durable version is definitely present, every refused one is definitely absent, and
+the contents never lie about their authorship. With an incomplete deletion, EC may
+legitimately answer INCOMPLETE, and replicated pools may resurrect the object.
+
+Everything derives from version states and events, with no extra flags: an ACKED
+version cannot vanish except through an event that first breaks the session's
+guarantee - a crash (taking the session down with the process) or a
+repeer/stop_pg (which stop dirty sessions BEFORE peering starts).
+
+The test's checks look ONLY at what the write path exposes to the client; the model
+deliberately ignores things the client cannot know about, such as paired OSD
+deaths per se.
+
+## The simulated reality
+
+- **Disk simulator**: requests carry delays; `power_loss()` = power outage -
+  uncommitted writes are lost, committed ones survive. An unknown FD/opcode aborts
+  the test (a stub guard).
+- **etcd simulator**: keys/values + leases; revoke/TTL are emulated on a crash (the
+  OSD's keys are removed, its lease revoked).
+- **Peer OSD simulator**: answers per the secondary-OSD/blockstore contract -
+  read/write/delete, SYNC, LIST. When a peer OSD dies, non-SYNCed writes die with it
+  with probability `peer_cache_loss`% (default 50 - modelling a volatile disk cache);
+  SYNCed ones always survive. A peer crash by itself does NOT change the world model:
+  the model only knows what the client sees. The crash repeers the PG, the repeer
+  kills dirty client sessions (volatile ACKED writes are exactly what makes a client
+  dirty), the session dies - and THAT is what the client observes
+  (PEER_STOPPED/EPIPE), and only session_died() demotes. If a crash rolled something
+  back under a live session, that would be a rollback invisible to the client -
+  check 5 catches it.
+- **Network simulator**: service_peers runs every round. The request-phase delay:
+  a random `--p-wire`% (default 70) of each peer's queued subops gets applied; the
+  response-phase delay: a random `p_reply`% of the already-applied replies gets
+  delivered. A reply belongs to its connection: the connection died - undelivered
+  replies die with `-EPIPE` (the apply stays). When an OSD instance dies, however,
+  mutating ops still in `sent_ops` are copied into a separate "wire" queue; they are
+  applied later at the `--p-wire` rate WITHOUT an answer (the connection fell).
+  Non-mutating ops (READ/LIST) are dropped (nobody left to answer), and the harvested
+  queue is capped at 64 entries.
+
+## Operations (one per step)
+
+Each step picks an action by weight (`--p-*`; 0 disables it):
+
+- **wait (500)** - send nothing, just run the simulation (event windows).
+- **write (60)** - write a random range of a random object; bump the version; the
+  delta is remembered as OPTIONAL; at most max_inflight (8) in flight.
+- **delete (6)** - delete a random object; the version is remembered as OPTIONAL with
+  zero length (a full delete); LEFT_ON_DEAD/ack are tracked in account().
+- **read (40)** - read an object: whole with probability `full_read`%, otherwise an
+  arbitrary range; a whole read verifies the object.
+- **sync (30)** - flush the cache (fsync); on confirmation the versions are marked
+  SYNCED (durable).
+- **restart (1)** - a CLEAN OSD restart: drain the queue (all inflight finish), wait
+  for the disk to go idle, destroy, recreate, wait for the OSD to come up, then
+  re-verify: every object byte-identical afterwards ("a clean restart that lost
+  nothing").
+- **takeover (1)** - a PG handover: first stop the PG by pointing the primary at a
+  fake member in the etcd stub -> stop_pg stops the PG; right after the
+  `/pg/state/1/1` key disappears (meaning the OSD released the PG) -> recreate the OSD
+  and point the primary back at it -> re-verify the objects as in restart.
+- **crash (2)** - the primary OSD crashes: a disk power loss (un-synced writes lost),
+  lease cancel + removal of the OSD's keys without cancelling the ops in flight to
+  peer OSDs, a new instance starts, objects re-verified.
+- **peer_down (3)** - a peer OSD crashes: a victim is chosen that breaks no object's
+  quorum (all latest OPTIONAL versions down to the first SYNCED are checked), the peer
+  OSD stops and its connection drops.
+- **peer_up (3)** - a peer OSD comes back (connects - if the PG or its history still
+  involves it); then settle(): wait for the PG to come up (ACTIVE|INCOMPLETE +
+  quiescence), then run recover_pending - deferred checks of objects whose chains may
+  have re-settled.
+- **set_change (1)** - the PG moves to another OSD set through etcd; with
+  `--recovery` - recovery reads; without it, degradation is legal and the
+  "surviving copies" world is protected by the checks.
+
+At the end - wait for completion: clean up until "the PG is healthy and every op has
+finished", at most MAX_STEPS=1e6 steps -> exceeding that aborts with a "did not
+settle" error.
+
+## Op outcome accounting (account)
+
+- Completed with the expected retval -> V_ACKED. A delete without LEFT_ON_DEAD closes
+  the partial-deletion window. A delete with LEFT_ON_DEAD must be repeated after the
+  peers come back, and in EC it allows a future INCOMPLETE unless the object is
+  rewritten in the meantime.
+- Completed with EAGAIN -> V_DEAD: the version was definitely not applied; currently
+  unreachable in the test, kept as a placeholder for the future - it would also apply
+  to e.g. ENOSPC.
+- Completed with EPIPE -> the version stays OPTIONAL: a refusal is not an ack, but the
+  op may have applied (or not). An optional delete must also be repeated after the
+  peers come back and likewise allows EC INCOMPLETE until the next rewrite.
+- Completed with any other error -> the test aborts ("failed with retval").
+
+## Checks (every abort point)
+
+**Data consistency (world_check):**
+1. world_check - the answer must be explained by a subset of the versions: every
+   required (durable) one is present; bytes are answered by exactly their own version
+   ("record obj/seq/off mismatch", a bitmap granule "carries data from wrong
+   version").
+2. "shows %s but the highest..." - an old version of the object was read while later
+   writes are lost (monotonicity broken).
+3. "read shows seq over acked v's granule" - a granule's contents (obj/seq/off in the
+   record) contradict the version that owns the granule in the model: stitched or
+   torn contents instead of the acknowledged bytes.
+4. "DEAD version showed again" - refused versions must not come back.
+5. "committed version read fully absent over floor" / "established version read
+   absent over its granule" - acknowledged ops must not disappear.
+6. A read -EIO "OSD deems unrecoverable" is legal in exactly one window: an
+   unfinished EC deletion. Outside it, -EIO aborts the test - including a "clean" ack
+   delete + INCOMPLETE after a revival (the OSD lied about completeness) and a chain
+   rollback under a live session with no deletion window; any other retval != len
+   aborts the test.
+7. bitmap_len != the expected read answer length -> abort the test.
+8. "came back INCOMPLETE" after set_change/crash - same criterion: an unfinished
+   deletion window makes INCOMPLETE legal and the deletion is repeated; anything
+   else aborts the test.
+9. "came back ... which no world of the recorded versions explains" - the final
+   check of object contents after the test settles.
+10. "changed across a clean restart that lost nothing" - object immutability across
+    a clean restart.
+
+**Liveness/timing:**
+
+11. "did not settle in MAX_STEPS" - the ops never completed within a huge step count
+    (a possible self-deadlock);
+12. "operations did not complete" - stuck ops, printed in the dump; "a read of
+    object N did not complete in MAX_STEPS ... deadlock?"; "OSD did not load PGs
+    from etcd in MAX_STEPS" / "OSD did not finish peering in MAX_STEPS" - start-up
+    checks.
+13. "did not release the PG" - the OSD didn't give the PG up when commanded;
+14. "disk did not go idle" - a use-after-free guard: every disk op must finish
+    before the OSD is destroyed;
+15. "PG did not become healthy after reviving every peer" - the PG stayed down after
+    all its OSDs came back;
+16. "pending recovery/repeated deletion could not complete on a healthy PG" - a
+    deletion/recovery that never finished on a live PG.
+
+**Test-stub guards:**
+
+17. An unexpected SQE (fd/opcode) on the disk stub; "unexpected secondary opcode" on
+    a peer; arguments like `--p-reply >= 1`, `--max-inflight >= 1`, etc.
+
+**Coverage checks:**
+
+18. `peer_cache_loss<100` + a volatile config + >=8 peer deaths -> verify that the
+    variant where a peer OSD kept its journal ran at least once.
+19. >=10 dirty OSD deaths/PG handovers -> wire_harvested > 0: at least one op of the
+    primary OSD that was in flight must have been harvested into the "wire" on the
+    way down - the "delivery from a dead OSD" path was exercised.
+
+## Reporting and reproducibility
+
+The OK line: writes(refused)/deletes/syncs/crashes/restarts/takeovers/peer
+deaths(kept journal)/recoveries/set changes/verified reads/stragglers
+(applied/captured/lost)/disk ops.
+
+Determinism: one seed -> the same configuration and the same schedule; `-v` traces
+disk and peer-OSD operations, op harvesting on peer deaths, and etcd changes.

docs/tests/pg-chaos.ru.md

@@ -0,0 +1,191 @@
+# Модель хаос-теста PG (`src/test/test_pg_chaos.cpp`)
+
+Seed-driven стресс-тест PG-машинерии: ОДИН настоящий `osd_t` (primary) с настоящим
+хранилищем на заглушке диска (полностью в RAM, `ringloop_mock`) владеет одним PG
+(номер 1, пул 1, объекты — 4 шт по 128 К, гранула 4 К). Остальные члены PG —
+фейковые вторичные OSD внутри теста. PG: реплицированная x3 (members 2,3, spare 4) или
+EC 2+2 (members 2..4, spare 5, объект = один parity-stripe). Сид однозначно задаёт
+конфигурацию и всю последовательность событий; упавший прогон воспроизводится тем же `--seed`.
+
+Проверяется в первую очередь PG-машинерия (subop-веер, peering/LIST/rollback/stabilize, repeer-демоны,
+stop_pg, INCOMPLETE, recovery) и частично хранилище первичного OSD; однако, проверка хранилища —
+не главная цель теста, его покрывает test_blockstore_chaos.
+
+## Конфигурации (10)
+
+- replicated/EC ×
+- immediate_commit none/all ×
+- csum_block нет/4096/16384
+
+`--recovery` включает recovery/rebalance после set-change (по умолчанию выкл, чтобы
+гонять degraded-состояния).
+
+## Модель мира
+
+Для каждого объекта — упорядоченная история версий; версия пишется в модель в момент
+SUBMIT (дельта = точные байты/длина/битмап гранул запроса), состояние:
+- `V_SUBMITTED` — операция отправлена, ответа ещё нет (выполняется). Ведёт себя как
+  V_OPTIONAL во всём, кроме одного: параллельные чтения вправе не показывать результат
+  этой записи, и это не будет маркировать её DEAD.
+- `V_OPTIONAL` — запись отправлена, но не подтверждена, и может быть либо применена
+  (но тогда целиком), либо не применена вообще. Версии попадают в это состояние из
+  V_SUBMITTED и, в режимах с явными fsync, V_ACKED при гибели клиентской сессии до
+  запроса SYNC (это считается маркером, что запись не подтверждена).
+  Наблюдение эффекта OPTIONAL записи в последующем чтении восстанавливает её в ACKED;
+  наблюдение её отсутствия (когда она не видна в своём диапазоне) ⇒ DEAD.
+- `V_ACKED` — запись подтверждена; запоминается `ack_live` (живой набор на момент ack —
+  используется для проверки превышения модели отказов). При immediate_commit=all сразу
+  считается зафиксированной на диске. ACKED запись не должна откатываться, пока сессия
+  клиента жива.
+- `V_SYNCED` — подтверждена и SYNC-нута ⇒ зафиксирована на диске и более не должна
+  откатываться вообще.
+- `V_PARTIAL` — удалена, но, возможно, не со всех OSD (PG осталась в LEFT_ON_DEAD).
+- `V_DEAD` — версия либо точно отказана (-EAGAIN, возможно -ENOSPC в будущем), либо
+  была OPTIONAL, но после этого стало точно известно, что она откачена, так как последующее
+  чтение её уже НЕ наблюдало. Повторное появление DEAD-версии — провал теста.
+
+Содержимое объектов самоидентифицируется версией: паттерн кодирует объект, версию и смещение —
+прочитанное можно декодировать обратно в «какая версия записала эти байты».
+
+Верификация читает ТОЛЬКО через первичный OSD и проверяет, что ответ объясняется слиянием
+ОДНОГО согласованного подмножества записанных версий, где все зафиксированные точно присутствуют,
+все откаченные точно отсутствуют, а содержимое не врёт об авторстве. При неполном удалении
+разрешается INCOMPLETE в EC, а в репликах допускается воскрешение объекта.
+
+Всё выводится из состояний версий и событий, без дополнительных флагов: ACKED не может
+исчезнуть, кроме как через событие, которое сначала ломает гарантию сессии — краш (уносит
+сессию вместе с процессом) или repeer/stop_pg (останавливают dirty-сессии ДО начала пиринга).
+
+Проверки в тесте смотрят ТОЛЬКО на то, как процесс записи видит клиент, и не учитывают
+в модели, например, падения парных OSD сами по себе, так как клиент не может о них знать.
+
+## Симулируемая реальность
+
+- **Симулятор диска**: запросы с задержками; `power_loss()` = потеря питания — незафиксированные
+  записи теряются, зафиксированные остаются. Неизвестный FD/opcode → падение теста (защита заглушки).
+- **Симулятор etcd**: ключ/значение + lease; revoke/TTL эмулируются при крахе (ключи OSD'а
+  удаляются, lease отзывается).
+- **Симулятор парных OSD**: отвечает в соответствии с контрактом вторичного OSD / blockstore.
+  Чтение/запись/удаление, SYNC, LIST. При падении парного OSD не-SYNC-нутые записи гибнут
+  с вероятностью `peer_cache_loss`% (по умолчанию 50 — моделирует волатильный кэш диска),
+  SYNC-нутые выживают всегда. Модель мира на краш пира сама НЕ реагирует: она знает
+  только то, что видит клиент. Краш реперит PG, репер убивает dirty-сессии клиентов
+  (volatile ACKED — это и есть dirty-клиент), сессия гибнет — вот это клиент наблюдает
+  (PEER_STOPPED/EPIPE), и только session_died() делает demote. Если краш откатит
+  что-то при живой сессии — это ненаблюдаемый клиентом откат, проверка 5 его ловит.
+- **Симулятор сети**: service_peers, каждый раунд. Задержка на этапе запроса — применяются случайные
+  `--p-wire`% (по умолчанию 70) очередей subop'ов пира; задержка на этапе ответа — доставляются
+  случайные `p_reply`% применённых ответов. Ответ принадлежит соединению: соединение
+  умерло — недоставленные ответы гибнут с `-EPIPE` (применение остаётся). Однако, при
+  гибели экземпляра OSD мутирующие операции в `sent_ops` копируются в отдельную очередь
+  «сети»; применяются позже (`--p-wire`%) БЕЗ ответа (соединение упало). Немутирующие
+  операции (READ/LIST) отбрасываются (отвечать некому), очередь собранных операций
+  ограничивается 64.
+
+## Операции (step)
+
+На каждом шаге (веса `--p-*`; 0 выключает действие):
+
+- **wait (500)** — ничего не слать, прокрутить симуляцию (событийные окна).
+- **write (60)** — запись случайного диапазона случайного объекта; увеличение версии;
+  дельта запоминается как OPTIONAL; ≤ max_inflight (8) одновременно.
+- **delete (6)** — удаление случайного объекта; версия запоминается как OPTIONAL с
+  нулевой длиной (полное удаление); LEFT_ON_DEAD/ack отслеживаются в account().
+- **read (40)** — чтение объекта, полное с вероятностью `full_read`%, иначе произвольный
+  диапазон; при полном чтении объект проверяется.
+- **sync (30)** — синхронизация кэша (fsync); при подтверждении версии помечаются SYNCED
+  (зафиксированными).
+- **restart (1)** — ЧИСТАЯ перезагрузка OSD: очистка очереди (все inflight закончены), ожидание
+  простоя диска, уничтожение, пересоздание, ожидание подъёма OSD и перепроверка: побайтовая
+  идентичность всех объектов после («чистый перезапуск, который ничего не потерял»).
+- **takeover (1)** — передача PG: сначала остановка PG через смену первичного OSD на
+  фейковый в заглушке etcd → stop_pg останавливает PG; далее сразу после удаления ключа
+  `/pg/state/1/1` (означающего, что OSD остановил PG) → пересоздание OSD и его переустановка
+  в качестве первичного → перепроверка объектов, как в restart.
+- **crash (2)** — падение первичного OSD: потеря питания диска (потеря не-синкнутых записей),
+  отмена лизы + удаление ключей OSD без отмены операций в полёте к парным OSD, запуск нового
+  экземпляра, перепроверка объектов.
+- **peer_down (3)** — крах парного OSD: выбирается жертва, не ломающая кворум ни
+  одного объекта (проверяются все последние OPTIONAL версии до первой SYNCED), парный OSD
+  останавливается, соединение с ним разрывается.
+- **peer_up (3)** — подъём парного OSD (соединение — если PG или её история его ещё
+  касается); после — settle(): ждать подъёма PG (ACTIVE|INCOMPLETE + покой), затем
+  прогнать recover_pending — отложенные проверки объектов, чьи цепи могли перегласоваться.
+- **set_change (1)** — переезд PG на другой OSD-набор через etcd; с `--recovery` —
+  восстановительные чтения; без — деградация легальна, мир «выживших копий» защищён
+  проверками.
+
+В конце — ожидание завершения: очистка до «PG в порядке и все операции завершены», максимум
+до MAX_STEPS=1e6 шагов → если превышается, то падение с ошибкой «did not settle».
+
+## Учёт результатов операций (account)
+
+- Завершена с ожидаемым кодом возврата → V_ACKED. Удаление без LEFT_ON_DEAD закрывает
+  окно частичного удаления. Удаление с LEFT_ON_DEAD требует повтора после подъёма
+  пиров, а с EC допускает будущий INCOMPLETE, если объект не перезаписан вновь.
+- Завершена с EAGAIN → V_DEAD: версия точно не применена; в тесте сейчас недостижимо,
+  оставлено как заготовка на будущее, также применимо, например, к ENOSPC.
+- Завершена с EPIPE → версия остаётся OPTIONAL: отказ — не ack, но операция могла
+  примениться (а могла и нет). Опциональное удаление также требует повтора после
+  подъёма пиров и тоже разрешает EC INCOMPLETE до следующей перезаписи.
+- Завершена с другой ошибкой → падение теста («failed with retval»).
+
+## Проверки (все точки abort)
+
+**Согласованность данных (world_check):**
+1. world_check — ответ обязан объясняться подмножеством версий: все обязательные
+   (зафиксированные) присутствуют; байты отвечают ровно своей версией
+   («record obj/seq/off mismatch», битмап-гранула «carries data from wrong version»).
+2. «shows %s but the highest…» — прочитана старая версия объекта, последние записи
+   потеряны (монотонность нарушена).
+3. «read shows seq over acked v's granule» — содержимое гранулы (obj/seq/off в record)
+   противоречит версии, которой гранула принадлежит по модели: сборное/порванное
+   содержимое вместо подтверждённых байт.
+4. «DEAD version showed again» — откаченные версии не должны возвращаться.
+5. «committed version read fully absent over floor» / «established version read absent
+   over its granule» — подтверждённые операции не должны исчезать.
+6. read -EIO «OSD deems unrecoverable» — легален ровно в одном окне: незавершённое
+   EC-удаление. Вне его -EIO — падение теста, в т.ч. «чистый» ack удаления + INCOMPLETE
+   после revival (OSD соврал о полноте) и откат цепей при живой сессии без окна удаления;
+   прочие retval ≠ len → падение теста.
+7. bitmap_len ≠ ожидаемому в ответах чтения → падение теста.
+8. «came back INCOMPLETE» после set_change/crash — тот же критерий: окно незавершённого
+   удаления ⇒ INCOMPLETE легален и удаление повторяется; в ином случае → падение теста.
+9. «came back … which no world of the recorded versions explains» — финальная проверка
+   корректности содержимого объектов после завершения теста.
+10. «changed across a clean restart that lost nothing» — неизменность объектов после перезапуска.
+
+**Живость/тайминг:**
+
+11. «did not settle in MAX_STEPS» — операции не завершились за большое число шагов (возможна
+    самоблокировка);
+12. «operations did not complete» — залипшие операции, печатаются в дампе;
+    «a read of object N did not complete in MAX_STEPS ... deadlock?»; «OSD did not load
+    PGs from etcd in MAX_STEPS» / «OSD did not finish peering in MAX_STEPS» — проверка на старте OSD.
+13. «did not release the PG» — OSD не отпустил PG по команде;
+14. «disk did not go idle» — проверка против use-after-free, что все операции диска завершены
+    перед уничтожением OSD;
+15. «PG did not become healthy after reviving every peer» — PG не поднялась после подъёма всех OSD;
+16. «pending recovery/repeated deletion could not complete on a healthy PG» — удаление/восстановление
+    не завершено на живой PG.
+
+**Защиты тестовых заглушек:**
+
+17. Неожиданный SQE (fd/opcode) на заглушке диска; «unexpected secondary opcode» у пира;
+    аргументы `--p-reply ≥ 1`, `--max-inflight ≥ 1` и т.п.
+
+**Проверки покрытия:**
+
+18. `peer_cache_loss<100` + volatile-конфиг + ≥8 смертей пира ⇒ проверка, что хотя бы
+    раз выполнен вариант, в котором парный OSD не потерял журнал.
+19. ≥10 грязных смертей OSD/передач PG ⇒ wire_harvested > 0: хотя бы одна операция
+    первичного OSD, бывшая в полёте, обязана была собраться в «сеть» при остановке —
+    путь «доставки от мёртвого OSD» выполнен.
+
+## Отчёт и воспроизводимость
+
+OK-строка: writes(refused)/deletes/syncs/crashes/restarts/takeovers/peer deaths
+(kept journal)/recoveries/set changes/verified reads/stragglers (applied/captured/lost)/disk ops.
+
+Детерминизм: один seed ⇒ та же конфигурация и то же расписание; `-v` — трасса операций
+диска и парных OSD, захватов операций при падении пиров и изменений etcd.

src/blockstore/v1/impl.h

@@ -218,6 +218,7 @@ class blockstore_impl_t: public blockstore_i
 
     // Journaling
     void prepare_journal_sector_write(int sector, blockstore_op_t *op);
+    void mark_not_dirty(int journal_sector);
     void handle_journal_write(ring_data_t *data, uint64_t flush_id);
     void disk_error_abort(const char *op, int retval, int expected);
 

src/blockstore/v1/write.cpp

@@ -102,7 +102,7 @@ bool blockstore_impl_t::enqueue_write(blockstore_op_t *op)
 {
     // Check or assign version number
     bool found = false, deleted = false, unsynced = false, is_del = (op->opcode == BS_OP_DELETE);
-    bool wait_big = false, wait_del = false;
+    bool wait_big = false, wait_del = false, in_open_sector = false;
     void *dyn = NULL;
     uint8_t *dyn_ptr = NULL;
     if (is_del)
@@ -148,6 +148,7 @@ bool blockstore_impl_t::enqueue_write(blockstore_op_t *op)
             wait_big = (dirty_it->second.state & BS_ST_TYPE_MASK) == BS_ST_BIG_WRITE
                 ? !IS_SYNCED(dirty_it->second.state)
                 : ((dirty_it->second.state & BS_ST_WORKFLOW_MASK) == BS_ST_WAIT_BIG);
+            in_open_sector = dirty_it->second.journal_sector == journal.sector_info[journal.cur_sector].offset;
             if (!is_del && !deleted)
             {
                 void *dyn_from = alloc_dyn_data
@@ -198,6 +199,26 @@ bool blockstore_impl_t::enqueue_write(blockstore_op_t *op)
             wait_del = true;
             PRIV(op)->real_version = op->version;
             op->version = version;
+            if (!unsynced && in_open_sector &&
+                journal.sector_info[journal.cur_sector].flush_count == 0)
+            {
+                // In exotic configurations without no_same_sector_overwrites, the deletion may reside
+                // in the current sector. In that case nothing advances dirty_start and the deletion
+                // becomes unflushable... Example is test_blockstore_chaos --seed 10901878
+                if (!journal.sector_info[journal.cur_sector].dirty)
+                {
+                    // If it's unmodified we can just move to the next sector
+                    // Probably unreachable currently due to "Forcibly move to the next sector" logic below
+                    journal.in_sector_pos = journal.block_size;
+                    mark_not_dirty(journal.cur_sector);
+                }
+                else
+                {
+                    // If it's modified we may add a SYNC so it writes the sector out
+                    assert(immediate_commit != IMMEDIATE_ALL);
+                    unsynced = true;
+                }
+            }
             if (unsynced)
             {
                 // Issue an additional sync so the delete reaches the journal
@@ -893,6 +914,20 @@ void blockstore_impl_t::handle_write_event(ring_data_t *data, blockstore_op_t *o
     }
 }
 
+void blockstore_impl_t::mark_not_dirty(int journal_sector)
+{
+    uint64_t new_ds = journal.sector_info[journal_sector].offset + journal.block_size;
+    if (new_ds >= journal.len)
+    {
+        new_ds = journal.block_size;
+    }
+    if ((journal.dirty_start + (journal.dirty_start >= journal.used_start ? 0 : journal.len)) <
+        (new_ds + (new_ds >= journal.used_start ? 0 : journal.len)))
+    {
+        journal.dirty_start = new_ds;
+    }
+}
+
 void blockstore_impl_t::release_journal_sectors(blockstore_op_t *op)
 {
     // Release flushed journal sectors
@@ -907,19 +942,11 @@ void blockstore_impl_t::release_journal_sectors(blockstore_op_t *op)
                 if (s == (1+journal.cur_sector))
                 {
                     // Forcibly move to the next sector and move dirty position
+                    // FIXME: This is the target behaviour, ok, but it seems to duplicate no_same_sector_overwrites...
                     journal.in_sector_pos = journal.block_size;
                 }
                 // We know for sure that we won't write into this sector anymore
-                uint64_t new_ds = journal.sector_info[s-1].offset + journal.block_size;
-                if (new_ds >= journal.len)
-                {
-                    new_ds = journal.block_size;
-                }
-                if ((journal.dirty_start + (journal.dirty_start >= journal.used_start ? 0 : journal.len)) <
-                    (new_ds + (new_ds >= journal.used_start ? 0 : journal.len)))
-                {
-                    journal.dirty_start = new_ds;
-                }
+                mark_not_dirty(s-1);
             }
             if (s == PRIV(op)->max_flushed_journal_sector)
                 break;

src/client/cluster_client.cpp

@@ -39,7 +39,7 @@ cluster_client_t::cluster_client_t(ring_loop_t *ringloop, timerfd_manager_t *tfd
         {
             // peer_osd just dropped connection
             // determine WHICH dirty_buffers are now obsolete and repeat them
-            if (wb->repeat_ops_for(this, peer_osd, 0, 0) > 0)
+            if (wb->repeat_ops_for(this, peer_osd) > 0)
             {
                 continue_ops();
             }
@@ -56,7 +56,7 @@ cluster_client_t::cluster_client_t(ring_loop_t *ringloop, timerfd_manager_t *tfd
 
     st_cli = std::move(st_cli_ptr);
     st_cli->on_load_config_hook = [this](json11::Json::object & cfg) { on_load_config_hook(cfg); };
-    st_cli->on_change_osd_state_hook = [this](uint64_t peer_osd) { on_change_osd_state_hook(peer_osd); };
+    st_cli->on_change_osd_state_hook = [this](osd_num_t peer_osd) { on_change_osd_state_hook(peer_osd); };
     st_cli->on_change_pool_config_hook = [this]() { on_change_pool_config_hook(); };
     st_cli->on_change_pg_config_hook = [this]() { on_change_pool_config_hook(); };
     st_cli->on_change_pg_state_hook = [this](pool_id_t pool_id, pg_num_t pg_num, osd_num_t prev_primary) { on_change_pg_state_hook(pool_id, pg_num, prev_primary); };
@@ -616,12 +616,6 @@ void cluster_client_t::on_change_pool_config_hook()
 
 void cluster_client_t::on_change_pg_state_hook(pool_id_t pool_id, pg_num_t pg_num, osd_num_t prev_primary)
 {
-    auto & pg_cfg = st_cli->pool_config[pool_id].pg_config[pg_num];
-    if (pg_cfg.cur_primary != prev_primary)
-    {
-        // Repeat this PG operations because an OSD which stopped being primary may not fsync operations
-        wb->repeat_ops_for(this, 0, pool_id, pg_num);
-    }
     // Always continue to resume operations hung because of lack of the primary OSD
     continue_ops();
     continue_lists();
@@ -640,7 +634,7 @@ bool cluster_client_t::get_immediate_commit(uint64_t inode)
     return pool_it->second.immediate_commit == IMMEDIATE_ALL;
 }
 
-void cluster_client_t::on_change_osd_state_hook(uint64_t peer_osd)
+void cluster_client_t::on_change_osd_state_hook(osd_num_t peer_osd)
 {
     osd_tree_metrics.erase(peer_osd);
     if (msgr.wanted_peers.find(peer_osd) != msgr.wanted_peers.end())
@@ -1426,58 +1420,6 @@ bool cluster_client_t::slice_rw(cluster_op_t *op)
     return true;
 }
 
-bool cluster_client_t::affects_pg(uint64_t inode, uint64_t offset, uint64_t len, pool_id_t pool_id, pg_num_t pg_num)
-{
-    if (INODE_POOL(inode) != pool_id)
-    {
-        return false;
-    }
-    auto pool_it = st_cli->pool_config.find(INODE_POOL(inode));
-    if (pool_it == st_cli->pool_config.end())
-    {
-        return false;
-    }
-    auto & pool_cfg = pool_it->second;
-    uint32_t pg_data_size = (pool_cfg.scheme == POOL_SCHEME_REPLICATED ? 1 : pool_cfg.pg_size-pool_cfg.parity_chunks);
-    uint64_t pg_block_size = pool_cfg.data_block_size * pg_data_size;
-    uint64_t first_stripe = (offset / pg_block_size) * pg_block_size;
-    uint64_t last_stripe = len > 0 ? ((offset + len - 1) / pg_block_size) * pg_block_size : first_stripe;
-    if ((last_stripe/pool_cfg.pg_stripe_size) - (first_stripe/pool_cfg.pg_stripe_size) + 1 >= pool_cfg.real_pg_count)
-    {
-        // All PGs are affected
-        return true;
-    }
-    pg_num_t first_pg_num = (first_stripe/pool_cfg.pg_stripe_size) % pool_cfg.real_pg_count + 1; // like map_to_pg()
-    pg_num_t last_pg_num = (last_stripe/pool_cfg.pg_stripe_size) % pool_cfg.real_pg_count + 1; // like map_to_pg()
-    return (first_pg_num <= last_pg_num
-        ? (pg_num >= first_pg_num && pg_num <= last_pg_num)
-        : (pg_num >= first_pg_num || pg_num <= last_pg_num));
-}
-
-bool cluster_client_t::affects_osd(uint64_t inode, uint64_t offset, uint64_t len, osd_num_t osd)
-{
-    auto pool_it = st_cli->pool_config.find(INODE_POOL(inode));
-    if (pool_it == st_cli->pool_config.end())
-    {
-        return false;
-    }
-    auto & pool_cfg = pool_it->second;
-    uint32_t pg_data_size = (pool_cfg.scheme == POOL_SCHEME_REPLICATED ? 1 : pool_cfg.pg_size-pool_cfg.parity_chunks);
-    uint64_t pg_block_size = pool_cfg.data_block_size * pg_data_size;
-    uint64_t first_stripe = (offset / pg_block_size) * pg_block_size;
-    uint64_t last_stripe = len > 0 ? ((offset + len - 1) / pg_block_size) * pg_block_size : first_stripe;
-    for (uint64_t stripe = first_stripe; stripe <= last_stripe; stripe += pg_block_size)
-    {
-        pg_num_t pg_num = (stripe/pool_cfg.pg_stripe_size) % pool_cfg.real_pg_count + 1; // like map_to_pg()
-        auto pg_it = pool_cfg.pg_config.find(pg_num);
-        if (pg_it != pool_cfg.pg_config.end() && pg_it->second.cur_primary == osd)
-        {
-            return true;
-        }
-    }
-    return false;
-}
-
 int cluster_client_t::try_send(cluster_op_t *op, int i, std::function<void(osd_op_t *op_part)> cb)
 {
     if (!msgr_initialized)
@@ -1613,6 +1555,8 @@ resume_1:
     }
     if (op->retval != 0)
     {
+        for (int i = 0; i < op->parts.size(); i++)
+            dirty_osds.insert(op->parts[i].osd_num);
         wb->fsync_error();
         if (op->retval == -EPIPE || op->retval == -EAGAIN || op->retval == -EIO || op->retval == -ENOSPC)
         {
@@ -1723,7 +1667,10 @@ void cluster_client_t::handle_op_part(cluster_op_part_t *part)
         // OK
         op->inflight_count--;
         if ((op->opcode == OSD_OP_WRITE || op->opcode == OSD_OP_DELETE) && !(op->flags & OP_IMMEDIATE_COMMIT))
+        {
+            wb->mark_flush_written_to(op->inode, part->op.req.rw.offset, part->op.req.rw.len, op->flush_id, part->osd_num);
             dirty_osds.insert(part->osd_num);
+        }
         part->flags |= PART_DONE;
         op->done_count++;
         if ((op->opcode == OSD_OP_READ || op->opcode == OSD_OP_READ_BITMAP || op->opcode == OSD_OP_READ_CHAIN_BITMAP)

src/client/cluster_client.h

@@ -210,14 +210,11 @@ protected:
     void vault_destroy();
     void vault_parse_secret(const std::string & key_id, const std::string & err, json11::Json data);
 
-    bool affects_osd(uint64_t inode, uint64_t offset, uint64_t len, osd_num_t osd);
-    bool affects_pg(uint64_t inode, uint64_t offset, uint64_t len, pool_id_t pool_id, pg_num_t pg_num);
-
     void on_load_config_hook(json11::Json::object & config);
     void on_load_pgs_hook(bool success);
     void on_change_pool_config_hook();
     void on_change_pg_state_hook(pool_id_t pool_id, pg_num_t pg_num, osd_num_t prev_primary);
-    void on_change_osd_state_hook(uint64_t peer_osd);
+    void on_change_osd_state_hook(osd_num_t peer_osd);
     void on_change_node_placement_hook();
     void on_change_inode_hook(uint64_t inode, bool removed);
 

src/client/cluster_client_impl.h

@@ -4,6 +4,7 @@
 #pragma once
 
 #include "cluster_client.h"
+#include "small_vector.h"
 
 #define PART_SENT 1
 #define PART_DONE 2
@@ -29,6 +30,11 @@ struct cluster_buffer_t
     int state;
     uint64_t flush_id;
     uint64_t *refcnt;
+    // Funny thing is that a buffer may be CACHE_REPEATING or even CACHE_DIRTY
+    // which means it's NOT written yet and have written_to_osd at the same time,
+    // because it may be left from the PREVIOUS write attempt which we MUST remember
+    // to correctly handle OSD failure retries.
+    small_vector_t<osd_num_t, 4> written_to_osd;
 };
 
 typedef std::map<object_id, cluster_buffer_t>::iterator dirty_buf_it_t;
@@ -52,11 +58,12 @@ public:
     bool is_left_merged(dirty_buf_it_t dirty_it);
     bool is_right_merged(dirty_buf_it_t dirty_it);
     void copy_write(cluster_op_t *op, int state, uint64_t new_flush_id = 0);
-    int repeat_ops_for(cluster_client_t *cli, osd_num_t peer_osd, pool_id_t pool_id, pg_num_t pg_num);
+    int repeat_ops_for(cluster_client_t *cli, osd_num_t peer_osd);
     void start_writebacks(cluster_client_t *cli, int count);
     bool read_from_cache(cluster_op_t *op, uint32_t bitmap_granularity);
     void flush_buffers(cluster_client_t *cli, dirty_buf_it_t from_it, dirty_buf_it_t to_it);
     void mark_flush_written(uint64_t inode, uint64_t offset, uint64_t len, uint64_t flush_id);
+    void mark_flush_written_to(uint64_t inode, uint64_t offset, uint64_t len, uint64_t flush_id, osd_num_t osd_num);
     void delete_flush(uint64_t inode, uint64_t offset, uint64_t len, uint64_t flush_id);
     void fsync_start();
     void fsync_error();

src/client/cluster_client_wb.cpp

@@ -93,6 +93,7 @@ void writeback_cache_t::copy_write(cluster_op_t *op, int state, uint64_t new_flu
         // approaches and fix everything accordingly when I decide to implement TRIM.
         return;
     }
+    small_vector_t<osd_num_t, 4> was_written_to;
     auto dirty_it = find_dirty(op->inode, op->offset);
     auto new_end = op->offset + op->len;
     while (dirty_it != dirty_buffers.end() &&
@@ -100,6 +101,11 @@ void writeback_cache_t::copy_write(cluster_op_t *op, int state, uint64_t new_flu
         dirty_it->first.stripe < op->offset+op->len)
     {
         assert(dirty_it->first.stripe + dirty_it->second.len > op->offset);
+        for (auto osd_num: dirty_it->second.written_to_osd)
+        {
+            if (std::find(was_written_to.begin(), was_written_to.end(), osd_num) == was_written_to.end())
+                was_written_to.push_back(osd_num);
+        }
         // Remove overlapping part(s) of buffers
         auto old_end = dirty_it->first.stripe + dirty_it->second.len;
         if (dirty_it->first.stripe < op->offset)
@@ -117,6 +123,7 @@ void writeback_cache_t::copy_write(cluster_op_t *op, int state, uint64_t new_flu
                     .state = dirty_it->second.state,
                     .flush_id = dirty_it->second.flush_id,
                     .refcnt = dirty_it->second.refcnt,
+                    .written_to_osd = dirty_it->second.written_to_osd,
                 });
                 if (dirty_it->second.buf)
                 {
@@ -175,6 +182,7 @@ void writeback_cache_t::copy_write(cluster_op_t *op, int state, uint64_t new_flu
                 .state = dirty_it->second.state,
                 .flush_id = dirty_it->second.flush_id,
                 .refcnt = dirty_it->second.refcnt,
+                .written_to_osd = dirty_it->second.written_to_osd,
             });
             dirty_buffers.erase(dirty_it);
             dirty_it = new_dirty_it;
@@ -225,6 +233,7 @@ void writeback_cache_t::copy_write(cluster_op_t *op, int state, uint64_t new_flu
         .state = state,
         .flush_id = new_flush_id,
         .refcnt = refcnt,
+        .written_to_osd = std::move(was_written_to),
     });
     if (state == CACHE_DIRTY)
     {
@@ -263,7 +272,7 @@ void writeback_cache_t::copy_write(cluster_op_t *op, int state, uint64_t new_flu
     }
 }
 
-int writeback_cache_t::repeat_ops_for(cluster_client_t *cli, osd_num_t peer_osd, pool_id_t pool_id, pg_num_t pg_num)
+int writeback_cache_t::repeat_ops_for(cluster_client_t *cli, osd_num_t peer_osd)
 {
     int repeated = 0;
     if (dirty_buffers.size())
@@ -273,11 +282,10 @@ int writeback_cache_t::repeat_ops_for(cluster_client_t *cli, osd_num_t peer_osd,
         for (auto wr_it = dirty_buffers.begin(), flush_it = wr_it, last_it = wr_it; ; )
         {
             bool end = wr_it == dirty_buffers.end();
-            bool flush_this = !end && wr_it->second.state != CACHE_REPEATING && wr_it->second.state != CACHE_DIRTY;
-            if (peer_osd)
-                flush_this = flush_this && cli->affects_osd(wr_it->first.inode, wr_it->first.stripe, wr_it->second.len, peer_osd);
-            if (pool_id && pg_num)
-                flush_this = flush_this && cli->affects_pg(wr_it->first.inode, wr_it->first.stripe, wr_it->second.len, pool_id, pg_num);
+            auto & wr_to = wr_it->second.written_to_osd;
+            bool flush_this = std::find(wr_to.begin(), wr_to.end(), peer_osd) != wr_to.end();
+            if (flush_this)
+                wr_to.clear();
             if (flush_it != wr_it && (end || !flush_this ||
                 wr_it->first.inode != flush_it->first.inode ||
                 wr_it->first.stripe != last_it->first.stripe+last_it->second.len))
@@ -309,28 +317,28 @@ void writeback_cache_t::flush_buffers(cluster_client_t *cli, dirty_buf_it_t from
     op->offset = from_it->first.stripe;
     op->len = prev_it->first.stripe + prev_it->second.len - from_it->first.stripe;
     uint32_t calc_len = 0;
-    uint64_t flush_id = ++last_flush_id;
+    op->flush_id = ++last_flush_id;
     for (auto it = from_it; it != to_it; it++)
     {
         it->second.state = CACHE_REPEATING;
-        it->second.flush_id = flush_id;
+        it->second.flush_id = op->flush_id;
         if (it->second.buf)
         {
             (*it->second.refcnt)++;
-            flushed_buffers.emplace(flush_id, it->second.refcnt);
+            flushed_buffers.emplace(op->flush_id, it->second.refcnt);
             op->iov.push_back(it->second.buf, it->second.len);
         }
         calc_len += it->second.len;
     }
     assert(calc_len == op->len);
     writebacks_active++;
-    op->callback = [this, flush_id](cluster_op_t* op)
+    op->callback = [this](cluster_op_t* op)
     {
         // Buffer flushes are always retried, regardless of the error,
         // so they should never result in an error here
         assert(op->retval == op->len);
-        for (auto fl_it = flushed_buffers.find(flush_id);
-            fl_it != flushed_buffers.end() && fl_it->first == flush_id; )
+        for (auto fl_it = flushed_buffers.lower_bound(op->flush_id);
+            fl_it != flushed_buffers.end() && fl_it->first == op->flush_id; )
         {
             if (!--(*fl_it->second)) // refcnt
             {
@@ -340,11 +348,7 @@ void writeback_cache_t::flush_buffers(cluster_client_t *cli, dirty_buf_it_t from
         }
         if (op->flags & OP_IMMEDIATE_COMMIT)
         {
-            delete_flush(op->inode, op->offset, op->len, flush_id);
-        }
-        else
-        {
-            mark_flush_written(op->inode, op->offset, op->len, flush_id);
+            delete_flush(op->inode, op->offset, op->len, op->flush_id);
         }
         delete op;
         writebacks_active--;
@@ -396,6 +400,21 @@ void writeback_cache_t::mark_flush_written(uint64_t inode, uint64_t offset, uint
     }
 }
 
+void writeback_cache_t::mark_flush_written_to(uint64_t inode, uint64_t offset, uint64_t len, uint64_t flush_id, osd_num_t osd_num)
+{
+    for (auto dirty_it = find_dirty(inode, offset);
+        dirty_it != dirty_buffers.end() && dirty_it->first.inode == inode &&
+        dirty_it->first.stripe < offset+len; dirty_it++)
+    {
+        if (dirty_it->second.flush_id == flush_id && dirty_it->second.state == CACHE_REPEATING)
+        {
+            auto & wr_to = dirty_it->second.written_to_osd;
+            if (std::find(wr_to.begin(), wr_to.end(), osd_num) == wr_to.end())
+                wr_to.push_back(osd_num);
+        }
+    }
+}
+
 void writeback_cache_t::start_writebacks(cluster_client_t *cli, int count)
 {
     if (!writeback_queue.size())

src/client/etcd_state_client.h

@@ -159,7 +159,7 @@ public:
     uint32_t global_bitmap_granularity = DEFAULT_BITMAP_GRANULARITY;
     uint32_t global_immediate_commit = IMMEDIATE_NONE;
 
-    uint64_t osd_num = 0;
+    osd_num_t osd_num = 0;
     std::string etcd_prefix;
     std::string etcd_client_cert;
     std::string etcd_client_key;

src/client/etcd_state_client_mock.cpp

@@ -6,82 +6,57 @@
 #include "etcd_state_client_mock.h"
 #include "str_util.h"
 
-etcd_state_client_mock_t::etcd_state_client_mock_t()
+etcd_mock_t::etcd_mock_t()
 {
     timespec tv;
     clock_gettime(CLOCK_REALTIME, &tv);
     srand48(tv.tv_sec*1000000000 + tv.tv_nsec);
 }
 
-void etcd_state_client_mock_t::etcd_add_watch(json11::Json watch)
+void etcd_mock_t::register_client(etcd_state_client_mock_t *client)
 {
+    clients.push_back(client);
 }
 
-std::string etcd_state_client_mock_t::get_username()
+void etcd_mock_t::unregister_client(etcd_state_client_mock_t *client)
 {
-    return username;
-}
-
-void etcd_state_client_mock_t::etcd_call_oneshot(const std::string & etcd_address, const std::string & api, json11::Json payload,
-    int timeout, std::function<void(std::string, json11::Json)> callback)
-{
-}
-
-void etcd_state_client_mock_t::pause()
-{
-    paused = true;
-}
-
-void etcd_state_client_mock_t::resume(size_t n)
-{
-    std::vector<etcd_mock_request_t> ran;
-    if (!n)
+    for (size_t i = 0; i < clients.size(); i++)
     {
-        paused = false;
-        ran = std::move(queue);
-    }
-    else if (n >= queue.size())
-        ran = std::move(queue);
-    else
-    {
-        for (size_t i = 0; i < n; i++)
-            queue.push_back(std::move(queue[i]));
-        queue.erase(queue.begin(), queue.begin() + n);
-    }
-    for (auto& req: ran)
-    {
-        etcd_call_nopause(req.api, req.payload, req.timeout, req.retries, req.interval, req.callback);
+        if (clients[i] == client)
+        {
+            clients.erase(clients.begin() + i);
+            break;
+        }
     }
 }
 
-void etcd_state_client_mock_t::set(const std::string& key, json11::Json data, uint64_t mod_revision, uint64_t lease_id)
+void etcd_mock_t::notify(std::map<std::string, etcd_kv_t> & changes)
+{
+    auto clients_copy = clients;
+    for (auto & client: clients_copy)
+    {
+        client->mock_apply_changes(changes);
+    }
+}
+
+void etcd_mock_t::set(const std::string& key, const std::string& value, uint64_t mod_revision, uint64_t lease_id)
 {
     if (!mod_revision)
         mod_revision = ++this->mod_revision;
-    this->data[key] = (etcd_mock_key_data_t){ .value = data.dump(), .mod_revision = mod_revision, .lease_id = lease_id };
-    etcd_kv_t kv = (etcd_kv_t){ .key = key, .value = data, .mod_revision = mod_revision };
-    parse_state(kv);
-    if (on_change_hook != NULL)
+    this->data[key] = (etcd_mock_key_data_t){ .value = value, .mod_revision = mod_revision, .lease_id = lease_id };
+    std::string err;
+    json11::Json json_value = json11::Json::parse(value, err);
+    if (err != "")
     {
-        std::map<std::string, etcd_kv_t> changes;
-        changes[key] = std::move(kv);
-        on_change_hook(changes);
+        fprintf(stderr, "Invalid JSON in etcd key %s during test: %s\n", key.c_str(), value.c_str());
+        exit(1);
     }
+    std::map<std::string, etcd_kv_t> changes;
+    changes[key] = { .key = key, .value = json_value, .mod_revision = mod_revision };
+    notify(changes);
 }
 
-void etcd_state_client_mock_t::etcd_call(const std::string & api, json11::Json payload, int timeout,
-    int retries, int interval, std::function<void(std::string, json11::Json)> callback)
-{
-    if (paused)
-    {
-        queue.push_back({ api, payload, timeout, retries, interval, callback });
-        return;
-    }
-    etcd_call_nopause(api, payload, timeout, retries, interval, callback);
-}
-
-void etcd_state_client_mock_t::etcd_call_nopause(std::string api, json11::Json payload, int timeout,
-    int retries, int interval, std::function<void(std::string, json11::Json)> callback)
+void etcd_mock_t::etcd_call(const std::string & api, json11::Json payload, std::function<void(std::string, json11::Json)> callback, etcd_state_client_mock_t *from)
 {
     printf("+ etcd: %s\n", api.c_str());
     if (api == "/kv/txn")
@@ -207,14 +182,25 @@ void etcd_state_client_mock_t::etcd_call_nopause(std::string api, json11::Json p
             { "succeeded", ok },
             { "responses", responses }
         });
-        // Push changes to watcher
+        // Deliver changes to the other clients' views, like an etcd watch
         if (changes.size())
+            notify(changes);
+    }
+    else if (api == "/kv/lease/revoke")
+    {
+        // An OSD going down takes its keys with it, like in real etcd
+        uint64_t lease_id = strtoull(payload["ID"].string_value().c_str(), NULL, 10);
+        for (auto it = data.begin(); it != data.end(); )
         {
-            for (auto & kv: changes)
-                parse_state(kv.second);
-            if (on_change_hook != NULL)
-                on_change_hook(changes);
+            if (it->second.lease_id == lease_id)
+                data.erase(it++);
+            else
+                it++;
         }
+        leases.erase(lease_id);
+        callback("", json11::Json::object{
+            { "header", json11::Json::object{ { "revision", mod_revision } } }
+        });
     }
     else if (api == "/lease/grant")
     {
@@ -222,10 +208,102 @@ void etcd_state_client_mock_t::etcd_call_nopause(std::string api, json11::Json p
         leases[lease_id] = payload["TTL"].uint64_value();
         callback("", json11::Json::object{ { "ID", std::to_string(lease_id) } });
     }
+    else if (api == "/lease/keepalive")
+    {
+        // A lease never expires in the mock; a TTL of 0 would kill the OSD
+        uint64_t lease_id = payload["ID"].uint64_value();
+        std::string lid = payload["ID"].string_value();
+        if (lid != "")
+            lease_id = strtoull(lid.c_str(), NULL, 10);
+        uint64_t ttl = leases.find(lease_id) != leases.end() ? leases[lease_id] : 0;
+        callback("", json11::Json::object{ { "result", json11::Json::object{ { "TTL", (double)ttl } } } });
+    }
     else
         callback("Unsupported", json11::Json());
 }
 
+etcd_state_client_mock_t::etcd_state_client_mock_t(etcd_mock_t & store)
+{
+    this->store = &store;
+    store.register_client(this);
+}
+
+etcd_state_client_mock_t::~etcd_state_client_mock_t()
+{
+    store->unregister_client(this);
+}
+
+void etcd_state_client_mock_t::etcd_add_watch(json11::Json watch)
+{
+}
+
+std::string etcd_state_client_mock_t::get_username()
+{
+    return username;
+}
+
+void etcd_state_client_mock_t::etcd_call_oneshot(const std::string & etcd_address, const std::string & api, json11::Json payload,
+    int timeout, std::function<void(std::string, json11::Json)> callback)
+{
+}
+
+void etcd_state_client_mock_t::pause()
+{
+    paused = true;
+}
+
+void etcd_state_client_mock_t::resume(size_t n)
+{
+    std::vector<etcd_mock_request_t> ran;
+    if (!n)
+    {
+        paused = false;
+        ran = std::move(queue);
+    }
+    else if (n >= queue.size())
+        ran = std::move(queue);
+    else
+    {
+        for (size_t i = 0; i < n; i++)
+            ran.push_back(std::move(queue[i]));
+        queue.erase(queue.begin(), queue.begin() + n);
+    }
+    for (auto& req: ran)
+    {
+        etcd_call_nopause(req.api, req.payload, req.timeout, req.retries, req.interval, req.callback);
+    }
+}
+
+void etcd_state_client_mock_t::set(const std::string& key, json11::Json data, uint64_t mod_revision, uint64_t lease_id)
+{
+    store->set(key, data.dump(), mod_revision, lease_id);
+}
+
+void etcd_state_client_mock_t::etcd_call(const std::string & api, json11::Json payload, int timeout,
+    int retries, int interval, std::function<void(std::string, json11::Json)> callback)
+{
+    if (paused)
+    {
+        queue.push_back({ api, payload, timeout, retries, interval, callback });
+        return;
+    }
+    etcd_call_nopause(api, payload, timeout, retries, interval, callback);
+}
+
+void etcd_state_client_mock_t::etcd_call_nopause(std::string api, json11::Json payload, int timeout,
+    int retries, int interval, std::function<void(std::string, json11::Json)> callback)
+{
+    store->etcd_call(api, payload, callback, this);
+}
+
+void etcd_state_client_mock_t::mock_apply_changes(std::map<std::string, etcd_kv_t> & changes)
+{
+    for (auto & kv: changes)
+        parse_state(kv.second);
+    if (on_change_hook != NULL)
+        on_change_hook(changes);
+}
+
 void etcd_state_client_mock_t::load_global_config()
 {
     etcd_state_client_t::load_global_config([this](const std::string & err) {});

src/client/etcd_state_client_mock.h

@@ -22,22 +22,44 @@ struct etcd_mock_request_t
     std::function<void(std::string, json11::Json)> callback;
 };
 
-struct etcd_state_client_mock_t: public etcd_state_client_t
+struct etcd_state_client_mock_t;
+
+// Mock etcd to connect multiple etcd_state_client_mock_t's to.
+struct etcd_mock_t
 {
     uint64_t mod_revision = 0;
-    bool paused = false;
-    std::vector<etcd_mock_request_t> queue;
-public:
     std::map<uint64_t, uint64_t> leases;
     std::map<std::string, etcd_mock_key_data_t> data;
+    std::vector<etcd_state_client_mock_t*> clients;
+
+    etcd_mock_t();
+    void register_client(etcd_state_client_mock_t *client);
+    void unregister_client(etcd_state_client_mock_t *client);
+    // Write a key and notify watchers. Raw value, like a real put would.
+    void set(const std::string& key, const std::string& value, uint64_t mod_revision = 0, uint64_t lease_id = 0);
+    // Execute a /kv/txn or /lease/grant request against the data
+    void etcd_call(const std::string & api_name, json11::Json payload, std::function<void(std::string, json11::Json)> callback, etcd_state_client_mock_t *from = NULL);
+    // Deliver changes to all watchers
+    void notify(std::map<std::string, etcd_kv_t> & changes);
+};
+
+struct etcd_state_client_mock_t: public etcd_state_client_t
+{
+    // The client doesn't own the store; the test does
+    etcd_mock_t *store = NULL;
+    bool paused = false;
+    std::vector<etcd_mock_request_t> queue;
     std::string username;
-    etcd_state_client_mock_t();
+    etcd_state_client_mock_t(etcd_mock_t & store);
+    ~etcd_state_client_mock_t();
     void set(const std::string& key, json11::Json data, uint64_t mod_revision = 0, uint64_t lease_id = 0);
     void pause();
     void resume(size_t n = 0);
     void etcd_call_oneshot(const std::string & etcd_address, const std::string & api, json11::Json payload, int timeout, std::function<void(std::string, json11::Json)> callback) override;
-    void etcd_call(const std::string & api, json11::Json payload, int timeout, int retries, int interval, std::function<void(std::string, json11::Json)> callback) override;
+    void etcd_call(const std::string & api_name, json11::Json payload, int timeout, int retries, int interval, std::function<void(std::string, json11::Json)> callback) override;
     void etcd_call_nopause(std::string api, json11::Json payload, int timeout, int retries, int interval, std::function<void(std::string, json11::Json)> callback);
+    // The store applies a change to the client's own view of the cluster
+    void mock_apply_changes(std::map<std::string, etcd_kv_t> & changes);
     void etcd_add_watch(json11::Json watch) override;
     std::string get_username() override;
     void load_global_config() override;

src/client/messenger.cpp

@@ -338,7 +338,7 @@ void osd_messenger_t::parse_config(const json11::Json & config, bool init)
     }
 }
 
-void osd_messenger_t::connect_peer(uint64_t peer_osd, json11::Json peer_state)
+void osd_messenger_t::connect_peer(osd_num_t peer_osd, json11::Json peer_state)
 {
     if (wanted_peers[peer_osd].raw_address_list != peer_state["addresses"])
     {
@@ -385,7 +385,7 @@ void osd_messenger_t::connect_peer(uint64_t peer_osd, json11::Json peer_state)
     try_connect_peer(peer_osd);
 }
 
-void osd_messenger_t::try_connect_peer(uint64_t peer_osd)
+void osd_messenger_t::try_connect_peer(osd_num_t peer_osd)
 {
     auto wp_it = wanted_peers.find(peer_osd);
     if (wp_it == wanted_peers.end() || wp_it->second.connecting ||

src/client/messenger.h

@@ -291,7 +291,7 @@ public:
     uint32_t bs_block_size = 0;
     uint32_t max_write_request_size = 0;
     robin_hood::unordered_flat_map<uint64_t, osd_client_t*> clients;
-    robin_hood::unordered_flat_map<uint64_t, osd_client_t*> osd_peers;
+    robin_hood::unordered_flat_map<osd_num_t, osd_client_t*> osd_peers;
     robin_hood::unordered_flat_map<int, osd_client_t*> clients_by_fd;
     robin_hood::unordered_flat_map<osd_num_t, osd_wanted_peer_t> wanted_peers;
     std::vector<std::string> osd_networks;
@@ -308,7 +308,7 @@ public:
     void init();
     void init_iothreads();
     void parse_config(const json11::Json & config, bool init);
-    void connect_peer(uint64_t osd_num, json11::Json peer_state);
+    void connect_peer(osd_num_t osd_num, json11::Json peer_state);
     void stop_client(uint64_t client_id, bool force_delete = false);
     void destroy_client(osd_client_t *cl);
     void outbox_push(osd_op_t *cur_op);
@@ -344,7 +344,7 @@ public:
     void measure_exec(osd_op_t *cur_op);
 
 protected:
-    void try_connect_peer(uint64_t osd_num);
+    void try_connect_peer(osd_num_t osd_num);
     void try_connect_peer_tcp(osd_num_t peer_osd, const char *peer_host, int peer_port);
     void handle_peer_epoll(int peer_fd, int epoll_events);
     void handle_connect_epoll(int peer_fd);
@@ -392,7 +392,7 @@ protected:
     msgr_rdma_context_t* rdmacm_get_context(ibv_context *verbs);
     msgr_rdma_context_t* rdmacm_create_qp(rdma_cm_id *cmid);
     void rdmacm_accept(rdma_cm_event *ev);
-    void rdmacm_try_connect_peer(uint64_t peer_osd, const std::string & addr, int rdmacm_port, int fallback_tcp_port);
+    void rdmacm_try_connect_peer(osd_num_t peer_osd, const std::string & addr, int rdmacm_port, int fallback_tcp_port);
     void rdmacm_set_conn_timeout(rdmacm_connecting_t *conn);
     void rdmacm_on_connect_peer_error(rdma_cm_id *cmid, int res);
     void rdmacm_address_resolved(rdma_cm_event *ev);

src/client/msgr_rdmacm.cpp

@@ -354,7 +354,7 @@ void osd_messenger_t::rdmacm_on_connect_peer_error(rdma_cm_id *cmid, int res)
     }
 }
 
-void osd_messenger_t::rdmacm_try_connect_peer(uint64_t peer_osd, const std::string & addr, int rdmacm_port, int fallback_tcp_port)
+void osd_messenger_t::rdmacm_try_connect_peer(osd_num_t peer_osd, const std::string & addr, int rdmacm_port, int fallback_tcp_port)
 {
     struct sockaddr_storage sa = {};
     if (!string_to_addr(addr, false, rdmacm_port, &sa))

src/client/msgr_send.cpp

@@ -444,7 +444,9 @@ void osd_messenger_t::outbox_push(osd_op_t *cur_op)
     {
         clock_gettime(CLOCK_REALTIME, &cur_op->tv_begin);
         cur_op->req.hdr.id = ++cl->send_op_id;
-        cl->sent_ops[cur_op->req.hdr.id] = cur_op;
+        auto & sent_op = cl->sent_ops[cur_op->req.hdr.id];
+        assert(!sent_op);
+        sent_op = cur_op;
     }
     else
     {

src/client/osd_ops.h

@@ -323,7 +323,7 @@ struct __attribute__((__packed__)) osd_reply_describe_item_t
     uint64_t stripe;
     uint32_t role;      // part number: 0 for replicas, 0..pg_size-1 for EC
     uint32_t loc_bad;   // LOC_OUTDATED / LOC_CORRUPTED / LOC_INCONSISTENT
-    osd_num_t osd_num;  // OSD number
+    uint64_t osd_num;  // OSD number (wire)
 };
 
 // lock/unlock PG for use by a primary OSD

src/cmd/cli_describe.cpp

@@ -38,7 +38,7 @@ struct cli_describe_t
     uint64_t object_state = 0;
     pool_id_t only_pool = 0;
     pg_num_t only_pg = 0;
-    std::vector<uint64_t> only_osds;
+    std::vector<osd_num_t> only_osds;
     uint64_t min_inode = 0, max_inode = 0;
     uint64_t min_offset = 0, max_offset = 0;
 

src/cmd/cli_fix.cpp

@@ -187,7 +187,7 @@ struct cli_fix_t
                             (part == -1 || items[i].role == part))
                         {
                             // Remove
-                            uint64_t rm_osd_num = items[i].osd_num;
+                            osd_num_t rm_osd_num = items[i].osd_num;
                             osd_op_t *rm_op = new osd_op_t;
                             rm_op->req = (osd_any_op_t){
                                 .sec_del = {

src/cmd/cli_modify_osd.cpp

@@ -12,7 +12,7 @@ struct osd_changer_t
 {
     cli_tool_t *parent;
 
-    uint64_t osd_num = 0;
+    osd_num_t osd_num = 0;
     bool set_tags = false;
     std::vector<std::string> new_tags;
     bool set_reweight = false;

src/cmd/cli_raw_ls.cpp

@@ -18,7 +18,7 @@ struct cli_raw_ls_t
     pg_num_t pg_num = 0;
     pg_num_t pg_count = 0;
     uint32_t pg_stripe_size = 0;
-    std::set<uint64_t> osds;
+    std::set<osd_num_t> osds;
     std::vector<osd_num_t> osd_list;
     uint64_t min_inode = 0, max_inode = 0, min_offset = 0, max_offset = 0;
     bool offset_set = false;

src/cmd/cli_rm_osd.cpp

@@ -18,7 +18,7 @@ struct rm_osd_t
     bool dry_run, force_warning, force_dataloss, allow_up;
     uint64_t etcd_tx_retry_ms = 500;
     uint64_t etcd_tx_retries = 10000;
-    std::vector<uint64_t> osd_ids;
+    std::vector<osd_num_t> osd_ids;
 
     int state = 0;
     cli_result_t result;

src/cmd/cli_status.cpp

@@ -136,7 +136,7 @@ resume_2:
         }
         int osd_count = 0, osd_up = 0;
         uint64_t total_raw = 0, free_raw = 0, free_down_raw = 0, down_raw = 0;
-        std::vector<uint64_t> slow_op_primary_osds, slow_op_secondary_osds;
+        std::vector<osd_num_t> slow_op_primary_osds, slow_op_secondary_osds;
         parent->iterate_kvs_1(osd_stats, "/osd/stats/", [&](uint64_t stat_osd_num, json11::Json value)
         {
             osd_count++;

src/disk_tool/disk_tool_udev.cpp

@@ -429,7 +429,7 @@ int disk_tool_t::clear_osd_superblock(const std::string & dev)
 
 int disk_tool_t::purge_devices(const std::vector<std::string> & devices)
 {
-    std::set<uint64_t> osd_numbers;
+    std::set<osd_num_t> osd_numbers;
     json11::Json::array superblocks;
     for (auto & device: devices)
     {

src/osd/CMakeLists.txt

@@ -45,6 +45,25 @@ target_link_libraries(osd_test
 add_test(NAME osd_test COMMAND osd_test)
 add_dependencies(build_tests osd_test)
 
+# test_pg_chaos
+add_executable(test_pg_chaos
+	EXCLUDE_FROM_ALL
+	../test/test_pg_chaos.cpp
+	../client/etcd_state_client_mock.cpp
+	../test/mock/messenger.cpp
+	../test/ringloop_mock.cpp
+)
+target_link_libraries(test_pg_chaos
+	osd
+	vitastor_blk
+	vitastor_common
+	Jerasure
+	${ISAL_LIBRARIES}
+	${LIBURING_LIBRARIES}
+)
+add_test(NAME test_pg_chaos COMMAND test_pg_chaos)
+add_dependencies(build_tests test_pg_chaos)
+
 # osd_rmw_test
 add_executable(osd_rmw_test EXCLUDE_FROM_ALL osd_rmw_test.cpp)
 target_link_libraries(osd_rmw_test Jerasure ${ISAL_LIBRARIES} util)

src/osd/osd.cpp

@@ -109,6 +109,15 @@ osd_t::~osd_t()
     }
     ringloop->unregister_consumer(&consumer);
     ringloop->unregister_consumer(&init_consumer);
+    // Tear down clients first: destroying a client cancels its operations and the
+    // cancellation callbacks (continue_primary_*) still use the blockstore and the
+    // PGs. The messenger is a member, so its own destructor would otherwise cancel
+    // any still-bound operations after this body already cleared pgs. Destroy the
+    // clients directly rather than through stop_client(): the stop protocol would
+    // fire repeer and PG-report hooks on the way out of a dying OSD
+    in_destruction = true;
+    while (!msgr.clients.empty())
+        msgr.destroy_client(msgr.clients.begin()->second);
     if (bs)
         delete bs;
 #ifdef WITH_RDMACM

src/osd/osd.h

@@ -99,11 +99,18 @@ struct osd_pg_lock_t
     uint64_t state = 0;
 };
 
+struct osd_stab_ref_t
+{
+    uint32_t refs = 0;
+    osd_op_t *waiting_op = NULL;
+};
+
 struct osd_test_fixture_t;
 
 class osd_t
 {
     friend struct osd_test_fixture_t;
+    friend struct chaos_t;
 
     // config
 
@@ -169,6 +176,9 @@ class osd_t
     std::set<pool_pg_num_t> pg_state_dirty;
     bool etcd_global_config_loaded = false;
     bool pg_config_applied = false;
+    // Set in ~osd_t before clients start dying: cancellation callbacks of unfinished
+    // operations must not drive the state machines on half-destroyed internals
+    bool in_destruction = false;
     bool etcd_reporting_pg_state = false;
     bool etcd_reporting_stats = false;
     int print_stats_timer_id = -1, slow_log_timer_id = -1;
@@ -208,6 +218,13 @@ class osd_t
     uint64_t unstable_write_count = 0;
     std::map<osd_object_id_t, uint64_t> unstable_writes;
     std::deque<osd_op_t*> syncs_in_progress;
+    std::vector<osd_op_t*> delete_wakeups;
+    // Objects covered by stabilize snapshots of SYNCs that are already dispatched, with a
+    // refcount. A delete of a covered object waits until the SYNC finishes
+    robin_hood::unordered_map<object_id, osd_stab_ref_t> stabilize_refs;
+    void add_stabilize_refs(obj_ver_id *unstable_writes, int count);
+    void remove_stabilize_refs(obj_ver_id *unstable_writes, int count);
+    bool delete_purge_unstable(osd_op_t *cur_op);
 
     // client & peer I/O
 
@@ -364,20 +381,22 @@ class osd_t
     void handle_primary_bs_subop(osd_op_t *subop);
     void add_bs_subop_stats(osd_op_t *subop, bool recovery_related = false);
     void pg_cancel_write_queue(pg_t & pg, osd_op_t *first_op, object_id oid, int retval);
+    void pg_continue_queued_op(osd_op_t *next_op);
+    void finish_read_op(osd_op_t *cur_op, int64_t retval);
 
-    void submit_primary_subops(int submit_type, uint64_t op_version, const uint64_t* osd_set, osd_op_t *cur_op);
+    void submit_primary_subops(int submit_type, uint64_t op_version, const osd_num_t* osd_set, osd_op_t *cur_op);
     int submit_primary_subop_batch(int submit_type, inode_t inode, uint64_t op_version,
-        osd_rmw_stripe_t *stripes, const uint64_t* osd_set, osd_op_t *cur_op, int subop_idx, int zero_read);
+        osd_rmw_stripe_t *stripes, const osd_num_t* osd_set, osd_op_t *cur_op, int subop_idx, int zero_read);
     void submit_primary_subop(osd_op_t *cur_op, osd_op_t *subop,
         osd_rmw_stripe_t *si, bool wr, inode_t inode, uint64_t op_version);
     bool submit_to_osd(osd_op_t *subop, osd_num_t osd_num);
-    void submit_primary_del_subops(osd_op_t *cur_op, uint64_t *cur_set, uint64_t set_size, pg_osd_set_t & loc_set);
+    void submit_primary_del_subops(osd_op_t *cur_op, osd_num_t *cur_set, uint64_t set_size, pg_osd_set_t & loc_set);
     void submit_primary_del_batch(osd_op_t *cur_op, obj_ver_osd_t *chunks_to_delete, int chunks_to_delete_count);
     void submit_primary_sync_subops(osd_op_t *cur_op);
     void submit_primary_stab_subops(osd_op_t *cur_op);
-    void submit_primary_rollback_subops(osd_op_t *cur_op, const uint64_t* osd_set);
+    void submit_primary_rollback_subops(osd_op_t *cur_op, const osd_num_t* osd_set);
 
-    uint64_t* get_object_osd_set(pg_t &pg, object_id &oid, pg_osd_set_state_t **object_state);
+    osd_num_t* get_object_osd_set(pg_t &pg, object_id &oid, pg_osd_set_state_t **object_state);
 
     void continue_chained_read(osd_op_t *cur_op);
     int submit_chained_read_requests(pg_t *pg, osd_op_t *cur_op);

src/osd/osd_flush.cpp

@@ -67,6 +67,11 @@ void osd_t::submit_pg_flush_ops(pg_t & pg)
 
 void osd_t::handle_flush_op(bool rollback, pool_id_t pool_id, pg_num_t pg_num, pg_flush_batch_t *fb, osd_num_t peer_osd, int retval)
 {
+    if (in_destruction)
+    {
+        // Teardown: don't complete flush batches (and never repeer) on a dying OSD
+        return;
+    }
     if (log_level > 2)
     {
         printf("[PG %u/%u] flush batch %jx completed on OSD %ju with result %d\n",
@@ -149,7 +154,7 @@ void osd_t::handle_flush_op(bool rollback, pool_id_t pool_id, pg_num_t pg_num, p
         }
         for (osd_op_t *op: continue_ops)
         {
-            continue_primary_write(op);
+            pg_continue_queued_op(op);
         }
         continue_pg(pg);
     }

src/osd/osd_peering.cpp

@@ -157,13 +157,11 @@ void osd_t::reset_pg(pg_t & pg)
     pg.flush_batch = NULL;
     // Protected by inflight - reset_pg is impossible with active inflight operations
     assert(!pg.write_queue.size());
-    for (auto it = unstable_writes.begin(); it != unstable_writes.end(); )
+    // Just check that unstable_writes are cleared on stop/repeer and not re-added
+    for (auto uw: unstable_writes)
     {
-        // Forget this PG's unstable writes
-        if (INODE_POOL(it->first.oid.inode) == pg.pool_id && map_to_pg(it->first.oid) == pg.pg_num)
-            unstable_writes.erase(it++);
-        else
-            it++;
+        bool this_pg = INODE_POOL(uw.first.oid.inode) == pg.pool_id && map_to_pg(uw.first.oid) == pg.pg_num;
+        assert(!this_pg);
     }
 }
 
@@ -172,6 +170,15 @@ void osd_t::drop_dirty_pg_connections(pool_pg_num_t pg)
 {
     if (immediate_commit != IMMEDIATE_ALL)
     {
+        // Forget this PG's unstable_writes
+        // If we don't do that we may create a race with STABILIZE after LIST
+        for (auto it = unstable_writes.begin(); it != unstable_writes.end();)
+        {
+            if (INODE_POOL(it->first.oid.inode) == pg.pool_id && map_to_pg(it->first.oid) == pg.pg_num)
+                unstable_writes.erase(it++);
+            else
+                it++;
+        }
         std::vector<uint64_t> to_stop;
         for (auto & cp: msgr.clients)
         {
@@ -326,6 +333,11 @@ void osd_t::start_pg_peering(pg_t & pg)
 
 bool osd_t::continue_pg_peering(pg_t & pg)
 {
+    if (in_destruction)
+    {
+        // Teardown: cancelled peering subops must not resume the peering machine
+        return false;
+    }
     if (pg.peering_state->locked)
     {
         pg.peering_state->lists_done = true;

src/osd/osd_peering_pg.cpp

@@ -8,7 +8,7 @@ struct obj_ver_role
 {
     object_id oid;
     uint64_t version;
-    uint64_t osd_num;
+    osd_num_t osd_num;
     bool is_stable;
 };
 
@@ -110,9 +110,12 @@ void pg_obj_state_check_t::start_object()
 
 void pg_obj_state_check_t::handle_version()
 {
-    if (!target_ver && last_ver != list[list_pos].version && (n_stable > 0 || n_roles >= pg->pg_data_size))
+    // A version becomes target only if it's stable (successfully written to the previous set)
+    // or unstable but full. We could target any recoverable version, but in this case some
+    // client's writes written to OSD set A may actually land on a smaller subset of A.
+    // And then client can't be sure that the pg_minsize guarantee actually holds.
+    if (!target_ver && last_ver != list[list_pos].version && (n_stable > 0 || n_roles >= pg->pg_size))
     {
-        // Version is either stable or recoverable
         target_ver = last_ver;
         ver_end = list_pos;
     }
@@ -175,9 +178,9 @@ void pg_obj_state_check_t::handle_version()
 
 void pg_obj_state_check_t::finish_object()
 {
-    if (!target_ver && (n_stable > 0 || n_roles >= pg->pg_data_size))
+    if (!target_ver && (n_stable > 0 || n_roles >= pg->pg_size))
     {
-        // Version is either stable or recoverable
+        // Stable (possibly acked) or fully materialized unstable - see handle_version()
         target_ver = last_ver;
         ver_end = list_pos;
     }

src/osd/osd_peering_pg.h

@@ -57,11 +57,12 @@ struct pg_peering_state_t
 struct obj_piece_id_t
 {
     object_id oid;
-    uint64_t osd_num;
+    osd_num_t osd_num;
 };
 
 struct obj_ver_osd_t
 {
+    // crosses the wire (SEC_DELETE payload) - keep uint64
     uint64_t osd_num;
     object_id oid;
     uint64_t version;
@@ -106,7 +107,7 @@ struct pg_t
     // target_set is the "correct" peer OSD set for this PG
     std::vector<osd_num_t> target_set;
     // cur_set is the current set of connected peer OSDs for this PG
-    // cur_set = (role => osd_num or UINT64_MAX if missing). role numbers begin with zero
+    // cur_set = (role => osd_num or 0 if missing). role numbers begin with zero
     std::vector<osd_num_t> cur_set;
     // locked peer list => pg state reported to the peer
     std::map<osd_num_t, uint64_t> lock_peers;

src/osd/osd_peering_pg_test.cpp

@@ -35,7 +35,7 @@ int main(int argc, char *argv[])
         .cur_set = { 1, 2, 3 },
         .peering_state = new pg_peering_state_t(),
     };
-    for (uint64_t osd_num = 1; osd_num <= 3; osd_num++)
+    for (osd_num_t osd_num = 1; osd_num <= 3; osd_num++)
     {
         pg_list_result_t r = {
             .buf = (obj_ver_id*)malloc_or_die(sizeof(obj_ver_id) * 1024*1024*8),

src/osd/osd_primary.cpp

@@ -213,7 +213,7 @@ bool osd_t::prepare_primary_rw(osd_op_t *cur_op)
     return true;
 }
 
-uint64_t* osd_t::get_object_osd_set(pg_t &pg, object_id &oid, pg_osd_set_state_t **object_state)
+osd_num_t* osd_t::get_object_osd_set(pg_t &pg, object_id &oid, pg_osd_set_state_t **object_state)
 {
     if (!(pg.state & (PG_HAS_INCOMPLETE | PG_HAS_DEGRADED | PG_HAS_MISPLACED | PG_HAS_INCONSISTENT)))
     {
@@ -248,6 +248,29 @@ uint64_t* osd_t::get_object_osd_set(pg_t &pg, object_id &oid, pg_osd_set_state_t
     return pg.cur_set.data();
 }
 
+// Finish a read and continue the operation queued behind it. A read that hit a
+// raw ENOENT from a racing write/delete joins the write_queue and keeps its
+// entry as the queue head while it retries, so its finish must drain the queue
+// the same way a write or a delete does
+void osd_t::finish_read_op(osd_op_t *cur_op, int64_t retval)
+{
+    osd_primary_op_data_t *op_data = cur_op->op_data;
+    osd_op_t *next_op = NULL;
+    if (op_data && op_data->pg)
+    {
+        auto next_it = op_data->pg->write_queue.find(op_data->oid);
+        if (next_it != op_data->pg->write_queue.end() && next_it->second == cur_op)
+        {
+            op_data->pg->write_queue.erase(next_it++);
+            if (next_it != op_data->pg->write_queue.end() && next_it->first == op_data->oid)
+                next_op = next_it->second;
+        }
+    }
+    finish_op(cur_op, retval);
+    if (next_op)
+        pg_continue_queued_op(next_op);
+}
+
 void osd_t::continue_primary_read(osd_op_t *cur_op)
 {
     if (!cur_op->op_data && !prepare_primary_rw(cur_op))
@@ -265,6 +288,8 @@ void osd_t::continue_primary_read(osd_op_t *cur_op)
         goto resume_1;
     else if (op_data->st == 2)
         goto resume_2;
+    else if (op_data->st == 3)
+        goto resume_3;
 resume_0:
     cur_op->reply.rw.bitmap_len = 0;
     {
@@ -278,9 +303,18 @@ resume_0:
             op_data->stripes[role].read_start = 0;
             op_data->stripes[role].read_end = 0;
             op_data->stripes[role].missing = false;
+            // A retry must not carry error marks of the previous attempt: a stale
+            // not_exists would erase a chunk from the object set that answered fine now
+            op_data->stripes[role].read_error = false;
+            op_data->stripes[role].not_exists = false;
+            op_data->stripes[role].osd_num = 0;
         }
-        // OP_DATA_DEGRADED may be left from a previous degraded attempt
-        op_data->flags &= ~(uint32_t)OP_DATA_DEGRADED;
+        // OP_DATA_DEGRADED may be left from a previous degraded attempt, and
+        // OP_DATA_ENOENT_OK is decided again below from the current object state
+        op_data->flags &= ~(uint32_t)(OP_DATA_DEGRADED | OP_DATA_ENOENT_OK);
+        // Counters of the previous attempt must not leak into the retry verdict
+        op_data->done = op_data->errors = op_data->drops = op_data->errcode = 0;
+        op_data->fact_ver = 0;
         if (cur_op->req.rw.len == 0)
         {
             // len=0 => bitmap read
@@ -321,7 +355,7 @@ resume_0:
             if (pg && pg->scheme == POOL_SCHEME_REPLICATED &&
                 op_data->object_state && (op_data->object_state->state & OBJ_INCOMPLETE))
             {
-                finish_op(cur_op, -EIO);
+                finish_read_op(cur_op, -EIO);
                 return;
             }
             assert(!cur_op->buf);
@@ -332,7 +366,7 @@ resume_0:
         {
             if (extend_missing_stripes(op_data->stripes, op_data->prev_set, pg->pg_data_size, pg->pg_size) < 0)
             {
-                finish_op(cur_op, -EIO);
+                finish_read_op(cur_op, -EIO);
                 return;
             }
             // Submit reads
@@ -353,6 +387,30 @@ resume_2:
     {
         if (pg && (op_data->errcode == -EIO || op_data->errcode == -EDOM))
         {
+            // A raw -ENOENT behind the -EIO means a copy was removed while the read
+            // was in flight - a delete or a cleanup beat it. A read racing a delete
+            // legally sees both "before" and "after", so this is not corruption: join
+            // the write_queue behind the running write/delete to this object and
+            // retry when it's done. The queue holds the in-flight head operation, so
+            // a queue entry that isn't this read is exactly "an op is running here"
+            int enoent = 0;
+            for (int role = 0; role < op_data->stripe_count; role++)
+            {
+                if (op_data->stripes[role].not_exists)
+                {
+                    enoent = 1;
+                    break;
+                }
+            }
+            auto wq_it = pg->write_queue.find(op_data->oid);
+            if (enoent && wq_it != pg->write_queue.end() && wq_it->second != cur_op)
+            {
+                free(cur_op->buf);
+                cur_op->buf = NULL;
+                op_data->st = 3;
+                pg->write_queue.emplace(op_data->oid, cur_op);
+                return;
+            }
             // I/O or checksum error
             // FIXME: ref = true ideally... because new_state != state is not necessarily true if it's freed and recreated
             auto new_object_state = mark_object_corrupted(*pg, op_data->oid, op_data->object_state, op_data->stripes, false);
@@ -363,7 +421,7 @@ resume_2:
                 goto resume_0;
             }
         }
-        finish_op(cur_op, op_data->errcode);
+        finish_read_op(cur_op, op_data->errcode);
         return;
     }
     cur_op->reply.rw.version = op_data->fact_ver;
@@ -397,7 +455,12 @@ resume_2:
     {
         cur_op->iov.push_back(cur_op->buf, cur_op->req.rw.len);
     }
-    finish_op(cur_op, cur_op->req.rw.len);
+    finish_read_op(cur_op, cur_op->req.rw.len);
+    return;
+resume_3:
+    // The write/delete this read hit an ENOENT from is done and drained the queue
+    // to us; our entry is the head now and the read keeps it until it finishes
+    goto resume_0;
 }
 
 pg_osd_set_state_t *osd_t::mark_object(pg_t & pg, object_id oid, pg_osd_set_state_t *prev_object_state, bool ref,
@@ -809,6 +872,75 @@ void osd_t::deref_object_state(pg_t & pg, pg_osd_set_state_t **object_state, boo
     }
 }
 
+// A SYNC counts every entry of its stabilize snapshot so that deletes of those objects
+// wait while it's in flight; counts are removed when it finishes. Entries are keyed by
+// object with role bits stripped, so a count is just bumped per entry and released per
+// entry over the same array
+void osd_t::add_stabilize_refs(obj_ver_id *unstable_writes, int count)
+{
+    for (int i = 0; i < count; i++)
+    {
+        stabilize_refs[(object_id){
+            .inode = unstable_writes[i].oid.inode,
+            .stripe = unstable_writes[i].oid.stripe & ~STRIPE_MASK,
+        }].refs++;
+    }
+}
+
+void osd_t::remove_stabilize_refs(obj_ver_id *unstable_writes, int count)
+{
+    for (int i = 0; i < count; i++)
+    {
+        auto it = stabilize_refs.find((object_id){
+            .inode = unstable_writes[i].oid.inode,
+            .stripe = unstable_writes[i].oid.stripe & ~STRIPE_MASK,
+        });
+        assert(it != stabilize_refs.end());
+        if (--it->second.refs == 0)
+        {
+            if (it->second.waiting_op)
+                delete_wakeups.push_back(it->second.waiting_op);
+            stabilize_refs.erase(it);
+        }
+    }
+}
+
+// Delete supersedes the whole version chain below it. Forget the object in the global
+// unstable write set, then wait while some dispatched SYNC still has it in its snapshot.
+// Forgotten entries are not restored on a failed delete: failures repeer the PG.
+bool osd_t::delete_purge_unstable(osd_op_t *cur_op)
+{
+    osd_primary_op_data_t *op_data = cur_op->op_data;
+    uint64_t base_stripe = op_data->oid.stripe & ~STRIPE_MASK;
+    // A delete can't be the head of the write queue without a PG
+    pg_t & pg = *op_data->pg;
+    for (auto & chunk: pg.cur_loc_set)
+    {
+        object_id chunk_oid = {
+            .inode = op_data->oid.inode,
+            .stripe = base_stripe | (pg.scheme != POOL_SCHEME_REPLICATED ? (uint64_t)chunk.role : 0),
+        };
+        auto it = unstable_writes.find((osd_object_id_t){ .osd_num = chunk.osd_num, .oid = chunk_oid });
+        if (it == unstable_writes.end())
+            continue;
+        unstable_writes.erase(it);
+    }
+    object_id ref_key = {
+        .inode = op_data->oid.inode,
+        .stripe = op_data->oid.stripe & ~STRIPE_MASK,
+    };
+    auto stab_it = stabilize_refs.find(ref_key);
+    if (stab_it != stabilize_refs.end())
+    {
+        // A SYNC may be about to stabilize this object (SEC_STABILIZE can be requeued by the
+        // blockstore on EBUSY and overtake SEC_DELETE), so wait until the object leaves the
+        // snapshot. Deletes are serialized in write_queue, the wait is bounded
+        stab_it->second.waiting_op = cur_op;
+        return false;
+    }
+    return true;
+}
+
 void osd_t::continue_primary_del(osd_op_t *cur_op)
 {
     if (!cur_op->op_data && !prepare_primary_rw(cur_op))
@@ -873,8 +1005,15 @@ resume_3:
     // If not already deleted:
     if (op_data->fact_ver)
     {
-        // Save version override for parallel reads
-        pg.ver_override[op_data->oid] = op_data->fact_ver;
+        // Forget unstable writes of this object so that no next SYNC picks them up,
+        // then wait while some running SYNC still has the object in its stabilize
+        // snapshot. Purge and wait must stay atomic and must precede the SEC_DELETE
+        // dispatch: a SEC_STABILIZE requeued by the blockstore on EBUSY could
+        // otherwise overtake the delete and stabilize a version it removes
+        if (!delete_purge_unstable(cur_op))
+        {
+            return;
+        }
         // Submit deletes
         op_data->fact_ver++;
         submit_primary_del_subops(cur_op, NULL, 0, op_data->object_state ? op_data->object_state->osd_set : pg.cur_loc_set);
@@ -960,13 +1099,7 @@ continue_others:
     finish_op(cur_op, cur_op->reply.hdr.retval);
     if (next_op)
     {
-        // Continue next write to the same object
-        if (next_op->req.hdr.opcode == OSD_OP_WRITE)
-            continue_primary_write(next_op);
-        else
-        {
-            assert(next_op->req.hdr.opcode == OSD_OP_DELETE);
-            continue_primary_del(next_op);
-        }
+        // Continue next operation on the same object
+        pg_continue_queued_op(next_op);
     }
 }

src/osd/osd_primary.h

@@ -33,7 +33,7 @@ struct osd_primary_op_data_t
     osd_rmw_stripe_t *stripes = NULL;
     pg_t *pg = NULL;
     osd_op_t *subops = NULL;
-    uint64_t *prev_set = NULL;
+    osd_num_t *prev_set = NULL;
     pg_osd_set_state_t *object_state = NULL;
 
     union

src/osd/osd_primary_chain.cpp

@@ -168,7 +168,7 @@ int osd_t::collect_bitmap_requests(osd_op_t *cur_op, pg_t *pg, std::vector<bitma
         object_id cur_oid = { .inode = op_data->read_chain[chain_num], .stripe = op_data->oid.stripe };
         auto vo_it = pg->ver_override.find(cur_oid);
         uint64_t target_version = vo_it != pg->ver_override.end() ? vo_it->second : UINT64_MAX;
-        uint64_t* cur_set = get_object_osd_set(*pg, cur_oid, &op_data->chain_states[chain_num]);
+        osd_num_t* cur_set = get_object_osd_set(*pg, cur_oid, &op_data->chain_states[chain_num]);
         if (pg->scheme == POOL_SCHEME_REPLICATED)
         {
             osd_num_t read_target = 0;
@@ -433,7 +433,7 @@ int osd_t::submit_chained_read_requests(pg_t *pg, osd_op_t *cur_op)
             stripes[role].read_start = stripes[role].req_start;
             stripes[role].read_end = stripes[role].req_end;
         }
-        uint64_t *cur_set = pg ? pg->cur_set.data() : &this->osd_num;
+        osd_num_t *cur_set = pg ? pg->cur_set.data() : &this->osd_num;
         if (pg && pg->state != PG_ACTIVE)
         {
             cur_set = get_object_osd_set(*pg, cur_oid, &op_data->chain_states[chain_reads[cri].chain_pos]);
@@ -517,7 +517,7 @@ int osd_t::submit_chained_read_requests(pg_t *pg, osd_op_t *cur_op)
             target_ver = vo_it != pg->ver_override.end() ? vo_it->second : UINT64_MAX;
         }
         auto cur_state = op_data->chain_states[chain_reads[cri].chain_pos];
-        uint64_t *cur_set = (!pg ? &this->osd_num : (pg->state != PG_ACTIVE && cur_state ? cur_state->read_target.data() : pg->cur_set.data()));
+        osd_num_t *cur_set = (!pg ? &this->osd_num : (pg->state != PG_ACTIVE && cur_state ? cur_state->read_target.data() : pg->cur_set.data()));
         int zero_read = -1;
         if (!pg || pg->scheme == POOL_SCHEME_REPLICATED)
         {

src/osd/osd_primary_subops.cpp

@@ -105,7 +105,7 @@ void osd_t::finish_op(osd_op_t *cur_op, int retval)
     }
 }
 
-void osd_t::submit_primary_subops(int submit_type, uint64_t op_version, const uint64_t* osd_set, osd_op_t *cur_op)
+void osd_t::submit_primary_subops(int submit_type, uint64_t op_version, const osd_num_t* osd_set, osd_op_t *cur_op)
 {
     bool wr = submit_type == SUBMIT_WRITE;
     osd_primary_op_data_t *op_data = cur_op->op_data;
@@ -140,7 +140,7 @@ void osd_t::submit_primary_subops(int submit_type, uint64_t op_version, const ui
 }
 
 int osd_t::submit_primary_subop_batch(int submit_type, inode_t inode, uint64_t op_version,
-    osd_rmw_stripe_t *stripes, const uint64_t* osd_set, osd_op_t *cur_op, int subop_idx, int zero_read)
+    osd_rmw_stripe_t *stripes, const osd_num_t* osd_set, osd_op_t *cur_op, int subop_idx, int zero_read)
 {
     bool rep = !cur_op->op_data->pg || cur_op->op_data->pg->scheme == POOL_SCHEME_REPLICATED;
     bool wr = submit_type == SUBMIT_WRITE;
@@ -354,6 +354,14 @@ void osd_t::add_bs_subop_stats(osd_op_t *subop, bool recovery_related)
 
 void osd_t::handle_primary_subop(osd_op_t *subop, osd_op_t *cur_op)
 {
+    if (in_destruction)
+    {
+        // The OSD is being destroyed (a crash or shutdown cancelled this subop).
+        // Leave the whole op graph alone: some subops are still in flight in the
+        // blockstore, the primary state machine must not run on internals that are
+        // already half-gone, and all of this memory is going away with the OSD
+        return;
+    }
     uint64_t opcode = subop->req.hdr.opcode;
     int retval = subop->reply.hdr.retval;
     int expected;
@@ -548,7 +556,14 @@ void osd_t::submit_primary_del_subops(osd_op_t *cur_op, osd_num_t *cur_set, uint
 {
     osd_primary_op_data_t *op_data = cur_op->op_data;
     bool rep = op_data->pg->scheme == POOL_SCHEME_REPLICATED;
-    obj_ver_osd_t extra_chunks[loc_set.size()];
+    // A primary delete (cur_set == NULL) also has to clear copies the object state
+    // doesn't know about: peering drops copies of a *current* role owner that lag
+    // behind target, so they are invisible to object_state, never deleted, and they
+    // resurrect old bytes under a recreated object (blockstore read-merge walks every
+    // version below the newest one).  Deleting from every live member is safe: a
+    // delete of an absent copy is a 0 no-op.
+    int n_extra_roles = cur_set || !op_data->pg ? 0 : op_data->pg->pg_size;
+    obj_ver_osd_t extra_chunks[loc_set.size() + (size_t)n_extra_roles];
     int chunks_to_del = 0;
     for (auto & chunk: loc_set)
     {
@@ -568,6 +583,38 @@ void osd_t::submit_primary_del_subops(osd_op_t *cur_op, osd_num_t *cur_set, uint
             };
         }
     }
+    if (!cur_set && op_data->pg)
+    {
+        pg_t & pg = *op_data->pg;
+        for (int role = 0; role < (int)pg.cur_set.size(); role++)
+        {
+            if (pg.cur_set[role] == 0)
+            {
+                continue;
+            }
+            object_id role_oid = {
+                .inode = op_data->oid.inode,
+                .stripe = op_data->oid.stripe | (rep ? 0 : (uint64_t)role),
+            };
+            int j;
+            for (j = 0; j < chunks_to_del; j++)
+            {
+                if (extra_chunks[j].osd_num == pg.cur_set[role] &&
+                    extra_chunks[j].oid == role_oid)
+                {
+                    break;
+                }
+            }
+            if (j >= chunks_to_del)
+            {
+                extra_chunks[chunks_to_del++] = (obj_ver_osd_t){
+                    .osd_num = pg.cur_set[role],
+                    .oid = role_oid,
+                    .version = op_data->fact_ver,
+                };
+            }
+        }
+    }
     submit_primary_del_batch(cur_op, extra_chunks, chunks_to_del);
 }
 
@@ -635,7 +682,7 @@ void osd_t::submit_primary_sync_subops(osd_op_t *cur_op)
     }
     osd_op_t *subops = new osd_op_t[n_osds];
     op_data->subops = subops;
-    robin_hood::unordered_flat_map<uint64_t, osd_client_t*>::iterator peer_it;
+    robin_hood::unordered_flat_map<osd_num_t, osd_client_t*>::iterator peer_it;
     int subop_idx = 0;
     for (int i = 0; i < n_osds; i++)
     {
@@ -737,7 +784,7 @@ void osd_t::submit_primary_stab_subops(osd_op_t *cur_op)
     }
 }
 
-void osd_t::submit_primary_rollback_subops(osd_op_t *cur_op, const uint64_t* osd_set)
+void osd_t::submit_primary_rollback_subops(osd_op_t *cur_op, const osd_num_t* osd_set)
 {
     osd_primary_op_data_t *op_data = cur_op->op_data;
     osd_rmw_stripe_t *stripes = op_data->stripes;
@@ -828,8 +875,8 @@ void osd_t::submit_primary_rollback_subops(osd_op_t *cur_op, const uint64_t* osd
 
 void osd_t::pg_cancel_write_queue(pg_t & pg, osd_op_t *first_op, object_id oid, int retval)
 {
-    auto st_it = pg.write_queue.find(oid), it = st_it;
-    if (it == pg.write_queue.end() || it->second != first_op)
+    auto st_it = pg.write_queue.lower_bound(oid), it = st_it;
+    if (it == pg.write_queue.end() || it->first != oid || it->second != first_op)
     {
         // Write queue doesn't match the first operation.
         // first_op is a leftover operation from the previous peering of the same PG.
@@ -854,3 +901,16 @@ void osd_t::pg_cancel_write_queue(pg_t & pg, osd_op_t *first_op, object_id oid,
         }
     }
 }
+
+// Continue the next queued operation of an object. The queue holds writes and
+// deletes, and reads that hit a raw ENOENT from an op racing them, so the
+// drained operation has to be dispatched by its opcode
+void osd_t::pg_continue_queued_op(osd_op_t *next_op)
+{
+    if (next_op->req.hdr.opcode == OSD_OP_READ)
+        continue_primary_read(next_op);
+    else if (next_op->req.hdr.opcode == OSD_OP_WRITE)
+        continue_primary_write(next_op);
+    else
+        continue_primary_del(next_op);
+}

src/osd/osd_primary_sync.cpp

@@ -138,6 +138,13 @@ resume_2:
         int last_start = 0, last_end = 0;
         for (auto it = this->unstable_writes.begin(); it != this->unstable_writes.end(); it++)
         {
+            pool_pg_num_t wpg = {
+                .pool_id = INODE_POOL(it->first.oid.inode),
+                .pg_num = map_to_pg(it->first.oid),
+            };
+            // Should already be protected by dirty_pgs - any object in unstable_writes
+            // should also be represented by an entry in dirty_pgs
+            assert(pgs.at(wpg).state & PG_ACTIVE);
             if (last_osd != it->first.osd_num)
             {
                 if (last_osd != 0)
@@ -155,6 +162,8 @@ resume_2:
                 .oid = it->first.oid,
                 .version = it->second,
             };
+            // Deletes of these objects must wait while this snapshot is in flight
+            add_stabilize_refs(&(op_data->unstable_writes[last_end]), 1);
             last_end++;
         }
         if (last_osd != 0)
@@ -303,6 +312,11 @@ finish:
     }
     if (op_data->unstable_writes)
     {
+        // Releases the delete waits held by this snapshot
+        for (auto & run: *(op_data->unstable_write_osds))
+        {
+            remove_stabilize_refs(op_data->unstable_writes + run.start, run.len);
+        }
         delete op_data->unstable_write_osds;
         delete[] op_data->unstable_writes;
         op_data->unstable_writes = NULL;
@@ -316,6 +330,12 @@ finish:
     finish_op(cur_op, op_data->errcode);
     assert(syncs_in_progress.front() == cur_op);
     syncs_in_progress.pop_front();
+    std::vector<osd_op_t*> delete_wakeups_local;
+    delete_wakeups_local.swap(delete_wakeups);
+    for (osd_op_t *op: delete_wakeups_local)
+    {
+        continue_primary_del(op);
+    }
     if (syncs_in_progress.size() > 0)
     {
         cur_op = syncs_in_progress.front();

src/osd/osd_primary_write.cpp

@@ -76,6 +76,17 @@ resume_1:
         op_data->object_state->ref_count++;
     }
 retry_1:
+    // A retry of a mark-corrupted round must not carry the marks and counters of
+    // the failed one: a stale not_exists would erase a chunk that answers fine now
+    for (int role = 0; role < op_data->stripe_count; role++)
+    {
+        op_data->stripes[role].read_error = false;
+        op_data->stripes[role].not_exists = false;
+        op_data->stripes[role].osd_num = 0;
+    }
+    op_data->done = op_data->errors = op_data->drops = op_data->errcode = 0;
+    op_data->fact_ver = 0;
+    op_data->flags &= ~(uint32_t)OP_DATA_ENOENT_OK;
     if (pg.scheme == POOL_SCHEME_REPLICATED)
     {
         // Simplified algorithm
@@ -460,14 +471,8 @@ continue_others:
     }
     if (next_op)
     {
-        // Continue next write to the same object
-        if (next_op->req.hdr.opcode == OSD_OP_WRITE)
-            continue_primary_write(next_op);
-        else
-        {
-            assert(next_op->req.hdr.opcode == OSD_OP_DELETE);
-            continue_primary_del(next_op);
-        }
+        // Continue next operation on the same object
+        pg_continue_queued_op(next_op);
     }
 }
 
@@ -526,6 +531,7 @@ bool osd_t::remember_unstable_write(osd_op_t *cur_op, pg_t & pg, pg_osd_set_t &
     {
         goto resume_7;
     }
+    assert(pg.state & PG_ACTIVE);
     if (immediate_commit == IMMEDIATE_ALL)
     {
 immediate:
@@ -557,7 +563,7 @@ immediate:
 resume_6:
             if (op_data->n_subops > 0)
             {
-                op_data->st = 6;
+                op_data->st = base_state;
                 return false;
             }
 resume_7:

src/osd/osd_rmw.cpp

@@ -627,8 +627,8 @@ void* alloc_read_buffer(osd_rmw_stripe_t *stripes, int read_pg_size, uint64_t ad
     return buf;
 }
 
-void* calc_rmw(void *request_buf, osd_rmw_stripe_t *stripes, uint64_t *read_osd_set,
-    uint64_t pg_size, uint64_t pg_minsize, uint64_t pg_cursize, uint64_t *write_osd_set,
+void* calc_rmw(void *request_buf, osd_rmw_stripe_t *stripes, osd_num_t *read_osd_set,
+    uint64_t pg_size, uint64_t pg_minsize, uint64_t pg_cursize, osd_num_t *write_osd_set,
     uint64_t chunk_size, uint32_t bitmap_size)
 {
     // Generic parity modification (read-modify-write) algorithm
@@ -839,7 +839,7 @@ static void xor_multiple_buffers(buf_len_t *xor1, int n1, buf_len_t *xor2, int n
 }
 
 static void calc_rmw_parity_copy_mod(osd_rmw_stripe_t *stripes, int pg_size, int pg_minsize,
-    uint64_t *read_osd_set, uint64_t *write_osd_set, uint32_t chunk_size, uint32_t bitmap_granularity,
+    osd_num_t *read_osd_set, osd_num_t *write_osd_set, uint32_t chunk_size, uint32_t bitmap_granularity,
     uint32_t &start, uint32_t &end)
 {
     bool required = false;
@@ -904,7 +904,7 @@ static void calc_rmw_parity_copy_mod(osd_rmw_stripe_t *stripes, int pg_size, int
 }
 
 static void calc_rmw_parity_copy_parity(osd_rmw_stripe_t *stripes, int pg_size, int pg_minsize,
-    uint64_t *read_osd_set, uint64_t *write_osd_set, uint32_t chunk_size, uint32_t start, uint32_t end)
+    osd_num_t *read_osd_set, osd_num_t *write_osd_set, uint32_t chunk_size, uint32_t start, uint32_t end)
 {
     if (write_osd_set != read_osd_set && end != 0)
     {
@@ -942,7 +942,7 @@ static void calc_rmw_parity_copy_parity(osd_rmw_stripe_t *stripes, int pg_size,
 #endif
 }
 
-void calc_rmw_parity_xor(osd_rmw_stripe_t *stripes, int pg_size, uint64_t *read_osd_set, uint64_t *write_osd_set,
+void calc_rmw_parity_xor(osd_rmw_stripe_t *stripes, int pg_size, osd_num_t *read_osd_set, osd_num_t *write_osd_set,
     uint32_t chunk_size, uint32_t bitmap_size)
 {
     uint32_t bitmap_granularity = bitmap_size > 0 ? chunk_size / bitmap_size / 8 : 0;
@@ -984,7 +984,7 @@ void calc_rmw_parity_xor(osd_rmw_stripe_t *stripes, int pg_size, uint64_t *read_
 }
 
 void calc_rmw_parity_ec(osd_rmw_stripe_t *stripes, int pg_size, int pg_minsize,
-    uint64_t *read_osd_set, uint64_t *write_osd_set, uint32_t chunk_size, uint32_t bitmap_size)
+    osd_num_t *read_osd_set, osd_num_t *write_osd_set, uint32_t chunk_size, uint32_t bitmap_size)
 {
     uint32_t bitmap_granularity = bitmap_size > 0 ? chunk_size / bitmap_size / 8 : 0;
     reed_sol_matrix_t *matrix = get_ec_matrix(pg_size, pg_minsize);

src/osd/osd_rmw.h

@@ -38,11 +38,11 @@ int extend_missing_stripes(osd_rmw_stripe_t *stripes, osd_num_t *osd_set, int pg
 
 void* alloc_read_buffer(osd_rmw_stripe_t *stripes, int read_pg_size, uint64_t add_size);
 
-void* calc_rmw(void *request_buf, osd_rmw_stripe_t *stripes, uint64_t *read_osd_set,
-    uint64_t pg_size, uint64_t pg_minsize, uint64_t pg_cursize, uint64_t *write_osd_set,
+void* calc_rmw(void *request_buf, osd_rmw_stripe_t *stripes, osd_num_t *read_osd_set,
+    uint64_t pg_size, uint64_t pg_minsize, uint64_t pg_cursize, osd_num_t *write_osd_set,
     uint64_t chunk_size, uint32_t bitmap_size);
 
-void calc_rmw_parity_xor(osd_rmw_stripe_t *stripes, int pg_size, uint64_t *read_osd_set, uint64_t *write_osd_set,
+void calc_rmw_parity_xor(osd_rmw_stripe_t *stripes, int pg_size, osd_num_t *read_osd_set, osd_num_t *write_osd_set,
     uint32_t chunk_size, uint32_t bitmap_size);
 
 void use_ec(int pg_size, int pg_minsize, bool use);
@@ -50,7 +50,7 @@ void use_ec(int pg_size, int pg_minsize, bool use);
 void reconstruct_stripes_ec(osd_rmw_stripe_t *stripes, int pg_size, int pg_minsize, uint32_t bitmap_size);
 
 void calc_rmw_parity_ec(osd_rmw_stripe_t *stripes, int pg_size, int pg_minsize,
-    uint64_t *read_osd_set, uint64_t *write_osd_set, uint32_t chunk_size, uint32_t bitmap_size);
+    osd_num_t *read_osd_set, osd_num_t *write_osd_set, uint32_t chunk_size, uint32_t bitmap_size);
 
 std::vector<int> ec_find_good(osd_rmw_stripe_t *stripes, int stripe_count, int pg_size, int pg_minsize, bool is_xor,
     uint32_t chunk_size, uint32_t bitmap_size, uint64_t max_bruteforce, bool find_best);

src/test/CMakeLists.txt

@@ -81,10 +81,6 @@ add_dependencies(build_tests test_blockstore_chaos)
 target_link_libraries(test_blockstore_chaos vitastor_blk vitastor_common)
 add_test(NAME test_blockstore_chaos COMMAND test_blockstore_chaos)
 
-## test_shit
-#add_executable(test_shit test_shit.cpp osd_peering_pg.cpp)
-#target_link_libraries(test_shit ${LIBURING_LIBRARIES} m)
-
 # test_atomic
 add_executable(test_atomic test_atomic.cpp)
 target_link_libraries(test_atomic util)

src/test/mock/messenger.cpp

@@ -31,14 +31,16 @@ void osd_messenger_t::outbox_push(osd_op_t *cur_op)
     auto cl = cl_it->second;
     if (cur_op->op_type == OSD_OP_OUT)
         cur_op->req.hdr.id = ++cl->send_op_id;
-    cl->sent_ops[cur_op->req.hdr.id] = cur_op;
+    auto & sent_op = cl->sent_ops[cur_op->req.hdr.id];
+    assert(!sent_op);
+    sent_op = cur_op;
 }
 
 void osd_messenger_t::parse_config(const json11::Json & config, bool init)
 {
 }
 
-void osd_messenger_t::connect_peer(uint64_t peer_osd, json11::Json peer_state)
+void osd_messenger_t::connect_peer(osd_num_t peer_osd, json11::Json peer_state)
 {
     wanted_peers[peer_osd] = (osd_wanted_peer_t){
         .port = 1,

src/test/osd_test.cpp

@@ -2575,6 +2575,174 @@ void test_repeated_delete_is_noop()
     printf("test_repeated_delete_is_noop passed\n");
 }
 
+// A read whose copies were removed by a delete applied mid-flight is not
+// corruption: a read racing a delete legally sees both "before" and "after".
+// Such a read joins the write_queue behind the running delete and retries when
+// it's done - it may neither fail with -EIO nor leave the object marked
+// INCOMPLETE, because a concurrent write may be legitimately in flight too
+//
+// Layout: pool 1 replicated x2, PG set 1,2, primary = OSD 1 (us). Object v1
+// is on both members. A delete runs (applied locally, waiting for the peer),
+// a read races it and both its subops answer raw -ENOENT
+void test_read_enoent_waits_behind_running_delete()
+{
+    printf("test_read_enoent_waits_behind_running_delete\n");
+
+    osd_test_fixture_t f;
+    f.configure_replicated_pool(/*pool_id*/ 1, /*pg_size*/ 2, /*pg_minsize*/ 1, /*pg_count*/ 1,
+        { { 1, 2 } });
+    f.start(json11::Json::object {
+        { "osd_num", 1 },
+        { "etcd_address", "127.0.0.1:2379" },
+        { "immediate_commit", "all" },
+        { "no_recovery", true },
+        { "block_size", 131072 },
+        { "bitmap_granularity", 4096 },
+    });
+    f.connect_peer(2);
+
+    inode_t inode = INODE_WITH_POOL(1, 1);
+    object_id oid = { .inode = inode, .stripe = 0 };
+    f.reply_local_list({ { oid, 1 } }, 1);
+    // OSD 2 has nothing: the object is DEGRADED on OSD 1 alone, exactly the
+    // seed-5 shape where a raw ENOENT is not excused by OP_DATA_ENOENT_OK
+    f.reply_peer_list(2, {}, 0);
+    f.pump();
+    assert(f.pg(1, 1).state == (PG_ACTIVE|PG_HAS_DEGRADED));
+
+    // Delete v1: fact read, then local delete + peer SEC_DELETE. Hold both
+    int delete_retval = -1;
+    auto *del = new osd_op_t();
+    del->op_type = OSD_OP_IN;
+    del->client_id = 0;
+    del->req.rw.header.magic = SECONDARY_OSD_OP_MAGIC;
+    del->req.rw.header.id = 1;
+    del->req.rw.header.opcode = OSD_OP_DELETE;
+    del->req.rw.inode = inode;
+    del->req.rw.offset = 0;
+    del->req.rw.len = 0;
+    del->callback = [&delete_retval](osd_op_t *op) { delete_retval = op->reply.hdr.retval; };
+    f.exec(del);
+    f.bs_zero_read_ok(1);
+    assert(f.pg(1, 1).write_queue.count(oid) == 1);
+
+    // A read races the delete
+    int read_retval = -1;
+    uint64_t read_ver = 1;
+    auto *read = new osd_op_t();
+    read->op_type = OSD_OP_IN;
+    read->client_id = 0;
+    read->req.rw.header.magic = SECONDARY_OSD_OP_MAGIC;
+    read->req.rw.header.id = 2;
+    read->req.rw.header.opcode = OSD_OP_READ;
+    read->req.rw.inode = inode;
+    read->req.rw.offset = 0;
+    read->req.rw.len = 4096;
+    read->callback = [&read_retval, &read_ver](osd_op_t *op)
+    {
+        read_retval = op->reply.hdr.retval;
+        read_ver = op->reply.rw.version;
+    };
+    f.exec(read);
+
+    // The copy this read reads from is already deleted - the subop answers
+    // raw -ENOENT
+    auto *local_read = f.bs->take(BS_OP_READ);
+    assert(local_read->len == 4096);
+    local_read->retval = -ENOENT;
+    local_read->callback(local_read);
+
+    // THE KEY ASSERTION: the read neither fails nor rewrites the object set -
+    // it waits in the write_queue behind the running delete. Pre-fix it would
+    // have marked OSD 1 corrupted (falsely degrading the object) and retried
+    // against OSD 2
+    assert(read_retval == -1);
+    assert(f.pg(1, 1).state == (PG_ACTIVE|PG_HAS_DEGRADED));
+    assert(f.pg(1, 1).write_queue.count(oid) == 2);
+
+    // The delete lands; the queue drains to the read
+    auto *bd = f.bs->take(BS_OP_DELETE);
+    bd->version = 1;
+    bd->retval = 0;
+    bd->callback(bd);
+    f.peer_write_ok(2, OSD_OP_SEC_DELETE, 1);
+    assert(delete_retval == 0);
+
+    // The retry subops arrive. The object is really gone now, so the read has
+    // to resolve as non-existent: zeros with version 0
+    local_read = f.bs->take(BS_OP_READ);
+    assert(local_read->len == 4096);
+    local_read->retval = -ENOENT;
+    local_read->callback(local_read);
+
+    assert(read_retval == 4096);
+    assert(read_ver == 0);
+    assert(f.pg(1, 1).state == PG_ACTIVE);
+    assert(f.pg(1, 1).write_queue.count(oid) == 0);
+
+    delete del;
+    delete read;
+    printf("test_read_enoent_waits_behind_running_delete passed\n");
+}
+
+// The other side of the same branch: copies answering -ENOENT while NO
+// write/delete runs on the object is genuine data loss. It must still fail
+// with -EIO and mark the object INCOMPLETE - the ENOENT requeue must not
+// swallow it
+void test_read_enoent_without_racing_op_marks_corrupted()
+{
+    printf("test_read_enoent_without_racing_op_marks_corrupted\n");
+
+    osd_test_fixture_t f;
+    f.configure_replicated_pool(/*pool_id*/ 1, /*pg_size*/ 2, /*pg_minsize*/ 1, /*pg_count*/ 1,
+        { { 1, 2 } });
+    f.start(json11::Json::object {
+        { "osd_num", 1 },
+        { "etcd_address", "127.0.0.1:2379" },
+        { "immediate_commit", "all" },
+        { "block_size", 131072 },
+        { "bitmap_granularity", 4096 },
+        { "no_recovery", true },
+    });
+
+    f.connect_peer(2);
+
+    inode_t inode = INODE_WITH_POOL(1, 1);
+    object_id oid = { .inode = inode, .stripe = 0 };
+    f.reply_local_list({ { oid, 1 } }, 1);
+    // DEGRADED object (only OSD 1 has it): ENOENT is not excused, see above
+    f.reply_peer_list(2, {}, 0);
+    f.pump();
+    assert(f.pg(1, 1).state == (PG_ACTIVE|PG_HAS_DEGRADED));
+
+    int read_retval = 0;
+    auto *read = new osd_op_t();
+    read->op_type = OSD_OP_IN;
+    read->client_id = 0;
+    read->req.rw.header.magic = SECONDARY_OSD_OP_MAGIC;
+    read->req.rw.header.id = 2;
+    read->req.rw.header.opcode = OSD_OP_READ;
+    read->req.rw.inode = inode;
+    read->req.rw.offset = 0;
+    read->req.rw.len = 4096;
+    read->callback = [&read_retval](osd_op_t *op) { read_retval = op->reply.hdr.retval; };
+    f.exec(read);
+
+    auto *local_read = f.bs->take(BS_OP_READ);
+    assert(local_read->len == 4096);
+    local_read->retval = -ENOENT;
+    local_read->callback(local_read);
+    // (no peer subop: a replicated read goes to the one member holding the object)
+
+    // Nobody was running on the object - this is corruption, not a race
+    assert(read_retval == -EIO);
+    assert(f.pg(1, 1).state & PG_HAS_INCOMPLETE);
+    assert(f.pg(1, 1).write_queue.count(oid) == 0);
+
+    delete read;
+    printf("test_read_enoent_without_racing_op_marks_corrupted passed\n");
+}
+
 // Peering must not forget an outdated copy just because its OSD already holds
 // the object at another role. A stale piece kept by a set member after a set
 // change looks, by OSD number alone, like a copy that the target set already
@@ -2741,6 +2909,534 @@ void test_pg_epoch_bump_above_reported_epoch()
     printf("test_pg_epoch_bump_above_reported_epoch passed\n");
 }
 
+// A partial unstable chain left by a failed/rejected EC write must not become the
+// committed version even when it holds pg_data_size pieces: that would turn a
+// possibly-nonexistent write into committed data with zero redundancy. Peering must
+// fall back to the older stable version and roll the fragments back. A stable
+// fragment, on the other hand, is possibly-acked - the OSD can never know otherwise
+// - so it stays the target and is never rolled back, incomplete or not.
+// A partial chain of pg_data_size pieces is not committed either, even when a
+// degraded set holds a fragment per member: an EC 2+2 pair on the two data roles
+// of a 2-member set still cannot decode (subcase D: refused-write fragments of a
+// fresh object - peering must delete them, not commit).
+//
+// Layout: EC 2+2, PG 1/1, cur_set = role 0..3 => OSD 1..4. Object INODE_WITH_POOL(1,1)
+// is stable at v1 on all roles; a rejected write left v2 unstable on role 0 (OSD 1)
+// and role 2 (OSD 3) only. Peer lists put stable entries first (stable_count prefix).
+void test_peering_partial_unstable_ec_rolls_back()
+{
+    printf("test_peering_partial_unstable_ec_rolls_back\n");
+
+    inode_t inode = INODE_WITH_POOL(1, 1);
+    object_id oid = { .inode = inode, .stripe = 0 };
+
+    auto make_pg = []()
+    {
+        pg_t *pg = new pg_t();
+        pg->pool_id = 1;
+        pg->pg_num = 1;
+        pg->scheme = POOL_SCHEME_EC;
+        pg->pg_size = 4;
+        pg->pg_cursize = 4;
+        pg->pg_minsize = 2;
+        pg->pg_data_size = 2;
+        pg->target_set = { 1, 2, 3, 4 };
+        pg->cur_set = { 1, 2, 3, 4 };
+        pg->all_peers = { 1, 2, 3, 4 };
+        pg->cur_peers = { 1, 2, 3, 4 };
+        pg->peering_state = new pg_peering_state_t();
+        return pg;
+    };
+    // Peer list of (stripe, version) entries; the first `nstable` are stable
+    auto add_list = [&](pg_t *pg, osd_num_t osd_num, std::vector<std::pair<uint64_t, uint64_t>> items,
+        uint64_t nstable)
+    {
+        obj_ver_id *buf = (obj_ver_id*)malloc(sizeof(obj_ver_id)*items.size());
+        for (int i = 0; i < items.size(); i++)
+            buf[i] = { .oid = { .inode = inode, .stripe = items[i].first }, .version = items[i].second };
+        pg->peering_state->list_results[osd_num] = (pg_list_result_t){
+            .buf = buf, .total_count = items.size(), .stable_count = nstable,
+        };
+    };
+
+    // ============ A: partial unstable v2 falls back to stable v1 ============
+
+    pg_t *pg = make_pg();
+    add_list(pg, 1, { { 0, 1 }, { 0, 2 } }, 1);
+    add_list(pg, 2, { { 1, 1 } }, 1);
+    add_list(pg, 3, { { 2, 1 }, { 2, 2 } }, 1);
+    add_list(pg, 4, { { 3, 1 } }, 1);
+    pg->calc_object_states(0);
+
+    // v1 is the target: the object is clean at v1, reads see v1, fragments roll back
+    assert(pg->total_count == 1 && pg->clean_count == 1);
+    assert(!(pg->state & (PG_HAS_INCOMPLETE|PG_HAS_DEGRADED)));
+    assert(pg->state & PG_HAS_UNCLEAN);
+    assert(!find_object_state(*pg, oid));
+    assert(pg->ver_override.count(oid) && pg->ver_override[oid] == 1);
+    assert(pg->flush_actions.size() == 2);
+    for (auto & fa: pg->flush_actions)
+    {
+        assert(fa.second.rollback && !fa.second.make_stable && fa.second.rollback_to == 1);
+        assert((fa.first.oid.stripe & STRIPE_MASK) == (fa.first.osd_num == 1 ? 0u : 2u));
+    }
+    delete pg->peering_state;
+    delete pg;
+
+    // ============ B: fully materialized unstable v2 is committed as before ============
+
+    pg = make_pg();
+    add_list(pg, 1, { { 0, 1 }, { 0, 2 } }, 1);
+    add_list(pg, 2, { { 1, 1 }, { 1, 2 } }, 1);
+    add_list(pg, 3, { { 2, 1 }, { 2, 2 } }, 1);
+    add_list(pg, 4, { { 3, 1 }, { 3, 2 } }, 1);
+    pg->calc_object_states(0);
+
+    assert(pg->total_count == 1 && pg->clean_count == 1);
+    assert(pg->state & PG_HAS_UNCLEAN);
+    assert(!pg->ver_override.count(oid));
+    assert(pg->flush_actions.size() == 4);
+    for (auto & fa: pg->flush_actions)
+        assert(fa.second.make_stable && !fa.second.rollback && fa.second.stable_to == 2);
+    delete pg->peering_state;
+    delete pg;
+
+    // ============ C: a stable fragment is possibly-acked: target, never rolled back ============
+
+    pg = make_pg();
+    add_list(pg, 1, { { 0, 1 }, { 0, 2 } }, 2);
+    add_list(pg, 2, { { 1, 1 } }, 1);
+    add_list(pg, 3, { { 2, 1 }, { 2, 2 } }, 2);
+    add_list(pg, 4, { { 3, 1 } }, 1);
+    pg->calc_object_states(0);
+
+    // v2 (2 of 4 roles, stable) is degraded, not incomplete, and nothing is rolled back
+    assert(pg->state & PG_HAS_DEGRADED);
+    assert(!(pg->state & PG_HAS_INCOMPLETE));
+    assert(pg->flush_actions.empty());
+    delete pg->peering_state;
+    delete pg;
+
+    // ============ D: an unstable chain that was never whole. v2 exists on roles 0 and 1
+    // only - pg_data_size pieces, but EC 2+2 needs all 4 to decode, and the object never
+    // existed at any stable version: this is what a refused write to a fresh object
+    // leaves behind (seed 1996). The old rule (n_roles >= pg_data_size) targeted such a
+    // chain and peering-flush committed it; the new rule must refuse v2 and delete the
+    // fragments entirely ============
+
+    pg = make_pg();
+    pg->pg_cursize = 2;
+    pg->cur_set = { 1, 2 };
+    pg->cur_peers = { 1, 2 };
+    add_list(pg, 1, { { 0, 2 } }, 0);
+    add_list(pg, 2, { { 1, 2 } }, 0);
+    pg->calc_object_states(0);
+
+    // Nothing is committed: the object stays absent, both fragments are deleted
+    assert(pg->total_count == 0 && pg->clean_count == 0);
+    assert(pg->state & PG_HAS_UNCLEAN);
+    assert(!(pg->state & PG_HAS_INCOMPLETE));
+    assert(!pg->ver_override.count(oid));
+    assert(pg->flush_actions.size() == 2);
+    for (auto & fa: pg->flush_actions)
+    {
+        assert(fa.second.rollback && !fa.second.make_stable && fa.second.rollback_to == 0);
+        assert((fa.first.oid.stripe & STRIPE_MASK) == (fa.first.osd_num == 1 ? 0u : 1u));
+    }
+    delete pg->peering_state;
+    delete pg;
+
+    printf("test_peering_partial_unstable_ec_rolls_back passed\n");
+}
+
+// A dispatched SYNC holds its stabilize snapshot; a delete of a covered object must
+// wait after its fact read while the snapshot is in flight (SEC_STABILIZE of a version
+// the delete removes would answer -ENOENT) and must resume when the SYNC finishes.
+// Replicated pools have no unstable versions, so this is an EC-only race
+void test_delete_waits_for_stabilize_snapshot()
+{
+    printf("test_delete_waits_for_stabilize_snapshot\n");
+
+    osd_test_fixture_t f;
+    // EC 2+1, single PG on OSDs 1,2,3, OSD 1 is the primary
+    f.configure_ec_pool(/*data_chunks*/ 2, /*parity_chunks*/ 1, /*primary_osd*/ 1);
+    f.start(json11::Json::object {
+        { "osd_num", 1 },
+        { "etcd_address", "127.0.0.1:2379" },
+        { "immediate_commit", "none" },
+        { "block_size", 131072 },
+        { "bitmap_granularity", 4096 },
+    });
+    f.connect_peer(2);
+    f.connect_peer(3);
+    f.complete_peering_empty();
+    assert(f.pg(1, 1).state == PG_ACTIVE);
+
+    inode_t inode = INODE_WITH_POOL(1, 1);
+    uint64_t client_id = f.connect_client();
+
+    // One full-stripe lazy write: v1 is unstable on all three members
+    auto *wr = make_write_op(inode, 0, 256*1024, 0xab);
+    wr->client_id = client_id;
+    wr->req.hdr.id = 1;
+    f.exec(wr);
+    f.bs_zero_read_ok(0);
+    f.bs_write_ok(BS_OP_WRITE, 1);
+    {
+        auto *pw2 = f.peer_take(2, OSD_OP_SEC_WRITE);
+        pw2->reply.sec_rw.version = 1;
+        f.peer_complete(pw2, pw2->req.sec_rw.len);
+        auto *pw3 = f.peer_take(3, OSD_OP_SEC_WRITE);
+        pw3->reply.sec_rw.version = 1;
+        f.peer_complete(pw3, pw3->req.sec_rw.len);
+    }
+    assert(f.unstable_write_count_entries() == 3);
+
+    // Dispatch a SYNC; it takes its snapshot and waits for subop replies
+    auto *sync = new osd_op_t();
+    sync->op_type = OSD_OP_IN;
+    sync->client_id = client_id;
+    sync->req.hdr.magic = SECONDARY_OSD_OP_MAGIC;
+    sync->req.hdr.id = 43;
+    sync->req.hdr.opcode = OSD_OP_SYNC;
+    f.exec(sync);
+    // The snapshot covers the object's base stripe with all three role copies
+    assert(f.stabilize_ref_count() == 1);
+    assert(f.stabilize_ref_of((object_id){ .inode = inode, .stripe = 0 }) == 3);
+    assert(f.unstable_write_count_entries() == 0);
+
+    // The delete dispatches its fact read at once, but must park after the read is
+    // answered: purge+wait sit between the CAS check and the SEC_DELETE dispatch
+    int delete_retval = -1;
+    auto *del = new osd_op_t();
+    del->op_type = OSD_OP_IN;
+    del->client_id = 0;
+    del->req.rw.header.magic = SECONDARY_OSD_OP_MAGIC;
+    del->req.rw.header.id = 2;
+    del->req.rw.header.opcode = OSD_OP_DELETE;
+    del->req.rw.inode = inode;
+    del->req.rw.offset = 0;
+    del->req.rw.len = 0;
+    del->callback = [&delete_retval](osd_op_t *op) { delete_retval = op->reply.hdr.retval; };
+    f.exec(del);
+    f.pump();   // flush handle_peers so the assertions below are meaningful
+    assert(delete_retval == -1);
+    assert(f.pg(1, 1).write_queue.size() == 1);
+    // The fact read is in flight, no delete went anywhere yet
+    // (the SYNC's own local BS_OP_SYNC and peer SEC_SYNCs are still pending, of course)
+    assert(f.bs->take(BS_OP_DELETE, false) == NULL);
+    assert(!f.peer_has_sent_op(2, OSD_OP_SEC_DELETE) && !f.peer_has_sent_op(3, OSD_OP_SEC_DELETE));
+    // Answer the local fact read: v1 (the peer SEC_READs are answered by the mock)
+    f.bs_zero_read_ok(1);
+    assert(f.dels_awaiting_sync_count() == 1);
+    assert(f.bs->take(BS_OP_DELETE, false) == NULL);
+    assert(!f.peer_has_sent_op(2, OSD_OP_SEC_DELETE) && !f.peer_has_sent_op(3, OSD_OP_SEC_DELETE));
+
+    // Finish the SYNC: SYNC subops, then STABILIZE subops
+    {
+        auto *local = f.bs->take(BS_OP_SYNC);
+        local->retval = 0;
+        local->callback(local);
+        for (osd_num_t p = 2; p <= 3; p++)
+        {
+            auto *ps = f.peer_take(p, OSD_OP_SEC_SYNC);
+            f.peer_complete(ps, 0);
+        }
+        auto *local_st = f.bs->take(BS_OP_STABLE);
+        local_st->retval = 0;
+        local_st->callback(local_st);
+        for (osd_num_t p = 2; p <= 3; p++)
+        {
+            auto *pst = f.peer_take(p, OSD_OP_SEC_STABILIZE);
+            f.peer_complete(pst, 0);
+        }
+    }
+    assert(sync->reply.hdr.retval == 0);
+    // The snapshot released its refs and woke the waiting delete, which dispatched
+    // its deletes now that no snapshot covers the object
+    assert(f.stabilize_ref_count() == 0);
+    assert(f.dels_awaiting_sync_count() == 0);
+    assert(delete_retval == -1);
+    {
+        auto *bd = f.bs->take(BS_OP_DELETE);
+        bd->version = 2;
+        bd->retval = 0;
+        bd->callback(bd);
+        f.peer_write_ok(2, OSD_OP_SEC_DELETE, 2);
+        f.peer_write_ok(3, OSD_OP_SEC_DELETE, 2);
+    }
+    assert(delete_retval == 0);
+    delete del;
+}
+
+// A repeer or stop must remove the PG's versions from unstable_writes: that map is what a
+// client SYNC snapshot stabilizes, and a SYNC issued after the repeer would commit
+// versions the new peering round is about to re-decide (and roll back)
+void test_repeer_forgets_unstable_writes(bool stop_pg)
+{
+    printf("test_repeer_forgets_unstable_writes (stop_pg=%d)\n", stop_pg);
+    osd_test_fixture_t f;
+    // EC 2+1, single PG on OSDs 1,2,3, OSD 1 is the primary
+    f.configure_ec_pool(/*data_chunks*/ 2, /*parity_chunks*/ 1, /*primary_osd*/ 1);
+    f.start(json11::Json::object {
+        { "osd_num", 1 },
+        { "etcd_address", "127.0.0.1:2379" },
+        { "immediate_commit", "none" },
+        { "block_size", 131072 },
+        { "bitmap_granularity", 4096 },
+    });
+    f.connect_peer(2);
+    f.complete_peering_empty();
+    f.connect_peer(3);
+    f.complete_peering_empty();
+    assert(f.pg(1, 1).state == PG_ACTIVE);
+    uint64_t client_id = f.connect_client();
+
+    // A full lazy EC write makes one object unstable on all three role copies, and a
+    // SYNC from its writer covers it: the snapshot holds 3 stabilize refs and the
+    // SYNC waits for its subop answers with the PG inflight
+    inode_t inode_a = INODE_WITH_POOL(1, 1);
+    {
+        auto *wr = make_write_op(inode_a, 0, 256*1024, 0xab);
+        wr->client_id = client_id;
+        wr->req.hdr.id = 1;
+        f.exec(wr);
+        f.bs_zero_read_ok(0);
+        f.bs_write_ok(BS_OP_WRITE, 1);
+        auto *pw2 = f.peer_take(2, OSD_OP_SEC_WRITE);
+        pw2->reply.sec_rw.version = 1;
+        f.peer_complete(pw2, pw2->req.sec_rw.len);
+        auto *pw3 = f.peer_take(3, OSD_OP_SEC_WRITE);
+        pw3->reply.sec_rw.version = 1;
+        f.peer_complete(pw3, pw3->req.sec_rw.len);
+    }
+    assert(f.unstable_write_count_entries() == 3);
+    auto *sync1 = new osd_op_t();
+    sync1->op_type = OSD_OP_IN;
+    sync1->client_id = client_id;
+    sync1->req.hdr.magic = SECONDARY_OSD_OP_MAGIC;
+    sync1->req.hdr.id = 43;
+    sync1->req.hdr.opcode = OSD_OP_SYNC;
+    f.exec(sync1);
+    assert(f.stabilize_ref_of((object_id){ .inode = inode_a, .stripe = 0 }) == 3);
+    assert(f.unstable_write_count_entries() == 0);
+
+    // A second full lazy write while the SYNC is in flight: the SYNC snapshot does
+    // not cover it, so 3 new unstable versions are left pending the next SYNC
+    inode_t inode_b = INODE_WITH_POOL(1, 2);
+    {
+        auto *wr = make_write_op(inode_b, 0, 256*1024, 0xab);
+        wr->client_id = client_id;
+        wr->req.hdr.id = 2;
+        f.exec(wr);
+        f.bs_zero_read_ok(0);
+        f.bs_write_ok(BS_OP_WRITE, 1);
+        auto *pw2 = f.peer_take(2, OSD_OP_SEC_WRITE);
+        pw2->reply.sec_rw.version = 1;
+        f.peer_complete(pw2, pw2->req.sec_rw.len);
+        auto *pw3 = f.peer_take(3, OSD_OP_SEC_WRITE);
+        pw3->reply.sec_rw.version = 1;
+        f.peer_complete(pw3, pw3->req.sec_rw.len);
+    }
+    assert(f.unstable_write_count_entries() == 3);
+
+    if (!stop_pg)
+    {
+        // Drop a peer to repeer the PG. The in-flight SYNC keeps the PG busy, so it goes to REPEERING
+        f.fail_peer_connection(2);
+        assert(f.pg(1, 1).state & PG_REPEERING);
+    }
+    else
+    {
+        // Stop the PG like the monitor does it
+        f.st_cli->set("/vitastor/pg/config", json11::Json::object {
+            { "items", json11::Json::object{
+                { "1", json11::Json::object {
+                    { "1", json11::Json::object {
+                        { "osd_set", json11::Json::array { 1, 2, 3 } },
+                        { "primary", 2 },
+                    } },
+                } }
+            } },
+        });
+        f.ringloop->loop();
+        assert(f.pg(1, 1).state & PG_STOPPING);
+    }
+
+    // repeer_pg must forget the PG's unstable writes before anything else: a SYNC
+    // racing the re-peering would otherwise snapshot versions that the new peering
+    // round is about to re-decide (and possibly roll back)
+    assert(f.unstable_write_count_entries() == 0);
+    assert(!f.has_dirty_pg(1, 1));
+    assert(!f.has_client(client_id));
+
+    // Drain the SYNC: its subops now answer normally (the peers stay connected,
+    // only the writer's session died). The SYNC finishes with EPIPE, releases the
+    // snapshot refs and the PG inflight, and continue_pg completes the stop - the
+    // reset_pg inside finish_stop_pg also asserts that no unstable write survived
+    while (true)
+    {
+        auto *local = f.bs->take(BS_OP_SYNC, false);
+        if (local)
+        {
+            local->retval = 0;
+            local->callback(local);
+            continue;
+        }
+        bool advanced = false;
+        for (osd_num_t peer : { (osd_num_t)2, (osd_num_t)3 })
+        {
+            auto *ps = f.peer_take_opt(peer, OSD_OP_SEC_SYNC);
+            if (ps)
+            {
+                f.peer_complete(ps, 0);
+                advanced = true;
+                break;
+            }
+            auto *pst = f.peer_take_opt(peer, OSD_OP_SEC_STABILIZE);
+            if (pst)
+            {
+                f.peer_complete(pst, 0);
+                advanced = true;
+                break;
+            }
+        }
+        if (advanced)
+            continue;
+        auto *local_st = f.bs->take(BS_OP_STABLE, false);
+        if (local_st)
+        {
+            local_st->retval = 0;
+            local_st->callback(local_st);
+            continue;
+        }
+        break;
+    }
+    f.pump();
+    assert(f.stabilize_ref_count() == 0);
+
+    // The stop completed: the PG is OFFLINE (possibly already erased). Note that a
+    // SYNC racing the stop must not re-add the forgotten versions on its error
+    // path - that would defeat the forget and stabilize rolled-back data later
+    assert(f.unstable_write_count_entries() == 0);
+    if (f.has_pg(1, 1))
+        assert(!(f.pg(1, 1).state & PG_STOPPING));
+
+    // FIXME: Make this test pass without leaks without the following line:
+    f.complete_peering_empty();
+
+    printf("test_repeer_forgets_unstable_writes (stop_pg=%d) passed\n", stop_pg);
+}
+
+// A delete must only wait for objects that are actually covered: a fresh unstable
+// object written after the SYNC took its snapshot must delete without waiting
+void test_delete_waits_only_for_covered_objects()
+{
+    printf("test_delete_waits_only_for_covered_objects\n");
+
+    osd_test_fixture_t f;
+    f.configure_ec_pool(/*data_chunks*/ 2, /*parity_chunks*/ 1, /*primary_osd*/ 1);
+    f.start(json11::Json::object {
+        { "osd_num", 1 },
+        { "etcd_address", "127.0.0.1:2379" },
+        { "immediate_commit", "none" },
+        { "block_size", 131072 },
+        { "bitmap_granularity", 4096 },
+    });
+    f.connect_peer(2);
+    f.connect_peer(3);
+    f.complete_peering_empty();
+    assert(f.pg(1, 1).state == PG_ACTIVE);
+
+    uint64_t client_id = f.connect_client();
+    uint64_t op_id = 1;
+    auto ec_write = [&](inode_t inode)
+    {
+        auto *wr = make_write_op(inode, 0, 256*1024, 0xab);
+        wr->client_id = client_id;
+        wr->req.hdr.id = ++op_id;
+        f.exec(wr);
+        f.bs_zero_read_ok(0);
+        f.bs_write_ok(BS_OP_WRITE, 1);
+        for (osd_num_t p = 2; p <= 3; p++)
+        {
+            auto *pw = f.peer_take(p, OSD_OP_SEC_WRITE);
+            pw->reply.sec_rw.version = 1;
+            f.peer_complete(pw, pw->req.sec_rw.len);
+        }
+    };
+    ec_write(INODE_WITH_POOL(1, 1));
+    assert(f.unstable_write_count_entries() == 3);
+
+    auto *sync = new osd_op_t();
+    sync->op_type = OSD_OP_IN;
+    sync->client_id = client_id;
+    sync->req.hdr.magic = SECONDARY_OSD_OP_MAGIC;
+    sync->req.hdr.id = ++op_id;
+    sync->req.hdr.opcode = OSD_OP_SYNC;
+    f.exec(sync);
+    assert(f.stabilize_ref_count() == 1);
+
+    // A new write while the SYNC is in flight - its version isn't in the snapshot
+    ec_write(INODE_WITH_POOL(1, 2));
+    assert(f.unstable_write_count_entries() == 3);
+
+    // Delete of the fresh object must not wait: purge removes its entries, refs don't cover it
+    int delete_retval = -1;
+    auto *del = new osd_op_t();
+    del->op_type = OSD_OP_IN;
+    del->client_id = 0;
+    del->req.rw.header.magic = SECONDARY_OSD_OP_MAGIC;
+    del->req.rw.header.id = ++op_id;
+    del->req.rw.header.opcode = OSD_OP_DELETE;
+    del->req.rw.inode = INODE_WITH_POOL(1, 2);
+    del->req.rw.offset = 0;
+    del->req.rw.len = 0;
+    del->callback = [&delete_retval](osd_op_t *op) { delete_retval = op->reply.hdr.retval; };
+    f.exec(del);
+    f.pump();
+    // It wasn't blocked: answer the local fact read and it moves on with no wait
+    // record (the peer SEC_READs are answered by the mock)
+    f.bs_zero_read_ok(1);
+    assert(f.dels_awaiting_sync_count() == 0);
+    // Its unstable entries were purged and, unlike a covered delete, it moved on
+    assert(f.unstable_write_count_entries() == 0);
+    {
+        auto *bd = f.bs->take(BS_OP_DELETE);
+        bd->version = 2;
+        bd->retval = 0;
+        bd->callback(bd);
+        f.peer_write_ok(2, OSD_OP_SEC_DELETE, 2);
+        f.peer_write_ok(3, OSD_OP_SEC_DELETE, 2);
+    }
+    assert(delete_retval == 0);
+    delete del;
+
+    // Finish the SYNC - it stabilizes only the first object
+    {
+        auto *local = f.bs->take(BS_OP_SYNC);
+        local->retval = 0;
+        local->callback(local);
+        for (osd_num_t p = 2; p <= 3; p++)
+        {
+            auto *ps = f.peer_take(p, OSD_OP_SEC_SYNC);
+            f.peer_complete(ps, 0);
+        }
+        auto *local_st = f.bs->take(BS_OP_STABLE);
+        local_st->retval = 0;
+        local_st->callback(local_st);
+        for (osd_num_t p = 2; p <= 3; p++)
+        {
+            auto *pst = f.peer_take(p, OSD_OP_SEC_STABILIZE);
+            f.peer_complete(pst, 0);
+        }
+    }
+    assert(sync->reply.hdr.retval == 0);
+    assert(f.stabilize_ref_count() == 0);
+
+    printf("test_delete_waits_only_for_covered_objects passed\n");
+}
+
 int main(int narg, char *args[])
 {
     test_load_global_config();
@@ -2753,6 +3449,8 @@ int main(int narg, char *args[])
     test_ec33_chain_read_phantom_bitmap_source();
     test_chained_read_eio_retry();
     test_ec_degraded_read_retry_resets_parity();
+    test_repeer_forgets_unstable_writes(false);
+    test_repeer_forgets_unstable_writes(true);
     for (int i = 0; test_flush_error_pg_repeer(i); i++) {}
     test_sync_nothing_to_sync();
     test_sync_happy_path_replicated();
@@ -2769,9 +3467,14 @@ int main(int narg, char *args[])
     test_partial_write_to_misplaced_object();
     test_partial_write_object_state_refs();
     test_repeated_delete_is_noop();
+    test_read_enoent_waits_behind_running_delete();
+    test_read_enoent_without_racing_op_marks_corrupted();
     test_peering_outdated_copy_other_role();
     test_pg_epoch_base_is_reported_epoch();
     test_pg_epoch_bump_above_reported_epoch();
     test_chained_read_large_chain_info();
+    test_peering_partial_unstable_ec_rolls_back();
+    test_delete_waits_for_stabilize_snapshot();
+    test_delete_waits_only_for_covered_objects();
     return 0;
 }

src/test/osd_test_fixture.h

@@ -28,6 +28,7 @@ public:
         queued.push_back(op);
     }
 
+    // Pop the first queued op
     blockstore_op_t *take()
     {
         assert(!queued.empty());
@@ -72,6 +73,9 @@ struct osd_test_fixture_t
 {
     timerfd_manager_t *tfd = nullptr;
     ring_loop_mock_t *ringloop = nullptr;
+    // The etcd data outlives clients: tests may recreate clients (simulated
+    // restarts) or run several of them against one "etcd"
+    etcd_mock_t etcd_store;
     etcd_state_client_mock_t *st_cli = nullptr; // owned by osd_t after start()
     osd_t *osd = nullptr;
     capturing_bs_t *bs = nullptr;
@@ -80,7 +84,7 @@ struct osd_test_fixture_t
     {
         tfd = new timerfd_manager_t([](int, bool, std::function<void(int, int)>) {});
         ringloop = new ring_loop_mock_t(RINGLOOP_DEFAULT_SIZE, [](io_uring_sqe *) {});
-        st_cli = new etcd_state_client_mock_t();
+        st_cli = new etcd_state_client_mock_t(etcd_store);
     }
 
     ~osd_test_fixture_t()
@@ -189,6 +193,15 @@ struct osd_test_fixture_t
     void connect_peer(osd_num_t osd_num)
     {
         auto *msgr = &osd->msgr;
+        // A reconnect replaces the client behind osd_peers[osd_num]; stop the old
+        // one explicitly so a fresh session always starts with an empty queue
+        auto old = msgr->osd_peers.find(osd_num);
+        if (old != msgr->osd_peers.end())
+        {
+            uint64_t old_cid = old->second->client_id;
+            msgr->osd_peers.erase(old);
+            msgr->stop_client(old_cid);
+        }
         auto *cl = new osd_client_t();
         cl->client_id = msgr->next_client_id++;
         cl->osd_num = osd_num;
@@ -202,7 +215,18 @@ struct osd_test_fixture_t
 
     void disconnect_peer(osd_num_t osd_num)
     {
-        osd->msgr.stop_client(osd->msgr.osd_peers.at(osd_num)->client_id);
+        // stop_client() leaves the (now PEER_STOPPED) entry in osd_peers. Erase it:
+        // submit_to_osd looks peers up by osd_num, so the OSD can't push new subops
+        // into a dead session, and a reconnect can't silently re-adopt the dead
+        // client with its undelivered sent_ops queue (bytes of a closed socket
+        // don't get delivered to the new one)
+        auto it = osd->msgr.osd_peers.find(osd_num);
+        if (it != osd->msgr.osd_peers.end())
+        {
+            uint64_t cid = it->second->client_id;
+            osd->msgr.osd_peers.erase(it);
+            osd->msgr.stop_client(cid);
+        }
     }
 
     // Complete every outstanding peering LIST op (local BS_OP_LIST + peer
@@ -262,8 +286,9 @@ struct osd_test_fixture_t
         return buf;
     }
 
-    // Pop the first BS_OP_LIST from bs->queued and reply with the given
-    // object list. The peering callback consumes (and later frees) op->buf.
+    // Complete the self LIST of the peering round with the given object list. The
+    // peering callback frees the bs_op it was given, so pop it and only invoke the
+    // callback - the real blockstore owns and frees ops on completion
     void reply_local_list(const std::vector<obj_ver_id> & objects, uint64_t stable_count)
     {
         auto *op = bs->take(BS_OP_LIST);
@@ -319,8 +344,40 @@ struct osd_test_fixture_t
     {
         auto *cl = osd->msgr.clients.at(op->client_id);
         cl->sent_ops.erase(op->req.hdr.id);
+        op->reply.hdr.magic = SECONDARY_OSD_REPLY_MAGIC;
+        op->reply.hdr.id = op->req.hdr.id;
+        op->reply.hdr.opcode = op->req.hdr.opcode;
         op->reply.hdr.retval = retval;
-        op->callback(op);
+        std::function<void(osd_op_t*)>(op->callback)(op);
+    }
+
+    void repeer_pgs(osd_num_t osd_num)
+    {
+        osd->repeer_pgs(osd_num);
+    }
+
+    // peer_take, but NULL instead of abort when the op (or the peer) isn't there
+    osd_op_t *peer_take_opt(osd_num_t osd_num, uint64_t opcode)
+    {
+        auto it = osd->msgr.osd_peers.find(osd_num);
+        if (it == osd->msgr.osd_peers.end())
+            return NULL;
+        for (auto & kv: it->second->sent_ops)
+        {
+            if (kv.second->req.hdr.opcode == opcode)
+                return kv.second;
+        }
+        return NULL;
+    }
+
+    // Simulate a peer connection death exactly like the real messenger does when a
+    // socket closes: stop_client repeers PGs first and only then destroys the client,
+    // which completes its in-flight ops with -EPIPE
+    void fail_peer_connection(osd_num_t osd_num)
+    {
+        auto *cl = osd->msgr.osd_peers.at(osd_num);
+        uint64_t client_id = cl->client_id;
+        osd->msgr.stop_client(client_id);
     }
 
     // Drive ringloop to flush handle_peers and any pending lambdas.
@@ -339,7 +396,8 @@ struct osd_test_fixture_t
 
     osd_client_t *peer(osd_num_t osd_num)
     {
-        return osd->msgr.osd_peers.at(osd_num);
+        auto *cl = osd->msgr.osd_peers.at(osd_num);
+        return cl->peer_state == PEER_STOPPED ? NULL : cl;
     }
 
     // Create a regular (non-OSD) incoming client with no real socket. Replies
@@ -406,6 +464,17 @@ struct osd_test_fixture_t
         }
     }
 
+    bool peer_has_sent_op(osd_num_t osd_num, uint64_t opcode)
+    {
+        auto *cl = osd->msgr.osd_peers.at(osd_num);
+        for (auto & kv: cl->sent_ops)
+        {
+            if (kv.second->req.hdr.opcode == opcode)
+                return true;
+        }
+        return false;
+    }
+
     bool has_pg(pool_id_t pool, pg_num_t num)
     {
         return osd->pgs.count({ .pool_id = pool, .pg_num = num }) > 0;
@@ -426,14 +495,30 @@ struct osd_test_fixture_t
         return osd->syncs_in_progress.size();
     }
 
-    bool peer_has_sent_op(osd_num_t osd_num, uint64_t opcode)
+    size_t unstable_write_count_entries()
     {
-        auto *cl = osd->msgr.osd_peers.at(osd_num);
-        for (auto & kv: cl->sent_ops)
+        return osd->unstable_writes.size();
+    }
+
+    size_t stabilize_ref_count()
+    {
+        return osd->stabilize_refs.size();
+    }
+
+    uint32_t stabilize_ref_of(const object_id & oid)
+    {
+        auto it = osd->stabilize_refs.find(oid);
+        return it == osd->stabilize_refs.end() ? 0 : it->second.refs;
+    }
+
+    size_t dels_awaiting_sync_count()
+    {
+        size_t n = 0;
+        for (auto & ref: osd->stabilize_refs)
         {
-            if (kv.second->req.hdr.opcode == opcode)
-                return true;
+            if (ref.second.waiting_op)
+                n++;
         }
-        return false;
+        return n;
     }
 };

src/test/ringloop_mock.cpp

@@ -193,20 +193,40 @@ disk_mock_t::disk_mock_t(const std::string & name, size_t size, bool buffered)
     memset(this->data, 0, size);
 }
 
+// A process or OSD torn down without a power loss is a clean shutdown: the device's
+// volatile write cache has to reach the platter, exactly like the fsync a real disk
+// would get. After power_loss() the cache is already empty (it applied, discarded or
+// committed everything), so committing unconditionally is right in both cases.
 disk_mock_t::~disk_mock_t()
 {
-    discard_buffers(true, 0);
+    commit_buffers();
     free(data);
 }
 
-// Store a buffer covering [end-len, end). Frees whatever was registered under the same
-// key before, so that callers can't silently drop a buffer by overwriting the entry
+// Store a buffer covering [end-len, end). Entries are keyed by their end, so a different
+// range that happens to end at the same offset must not silently consume the existing
+// entry's bytes: a zero-length write carries nothing at all, and a shorter one must
+// leave the prefix of the longer entry it collides with as its own buffer.
 void disk_mock_t::set_buffer(uint64_t end, uint8_t *buf, uint64_t len)
 {
+    if (len == 0)
+    {
+        free(buf);
+        return;
+    }
     auto it = buffers.find(end);
     if (it != buffers.end())
     {
-        free(it->second.iov_base);
+        if (it->second.iov_len > len)
+        {
+            // The old entry reaches further down than the new one; keep the prefix
+            uint8_t *cs = (uint8_t*)realloc(it->second.iov_base, it->second.iov_len - len);
+            if (!cs)
+                throw std::bad_alloc();
+            buffers[end - len] = (iovec){ .iov_base = cs, .iov_len = it->second.iov_len - len };
+        }
+        else if (it->second.iov_len < len)
+            free(it->second.iov_base);
         it->second = (iovec){ .iov_base = buf, .iov_len = len };
     }
     else
@@ -677,9 +697,13 @@ bool io_sim_t::step()
     {
         // Nothing in flight, only timers can move things forward
         advance(idle_tick);
+        if (after_step)
+            after_step();
         return false;
     }
     advance(next > now_us ? next - now_us : 0);
+    if (after_step)
+        after_step();
     for (auto disk: disks)
         disk->complete_due(now_us);
     return true;

src/test/ringloop_mock.h

@@ -147,6 +147,10 @@ class io_sim_t
 public:
     uint64_t now_us = 0;
     std::mt19937 rnd;
+    // Called at the end of every step(), so harnesses can deliver what the mocked
+    // messenger buffered this step (replies only complete a client-side wait
+    // after delivery, so every pump path needs it)
+    std::function<void()> after_step;
     // How far to move the clock when nothing is in flight and only timers can make progress
     uint64_t idle_tick = 1000;
 

src/test/test_cluster_client.cpp

@@ -238,7 +238,8 @@ void test1()
 {
     json11::Json config;
     timerfd_manager_t *tfd = new timerfd_manager_t([](int fd, bool wr, std::function<void(int, int)> callback){});
-    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t();
+    etcd_mock_t etcd_store;
+    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t(etcd_store);
     mock->pause();
     cluster_client_t *cli = new cluster_client_t(NULL, tfd, config, std::unique_ptr<etcd_state_client_t>(mock));
 
@@ -335,9 +336,9 @@ void test1()
             assert(offset == op->req.rw.offset+op->req.rw.len);
             replay_ops.push_back(op);
         }
-        if (replay_start != 0 || replay_end != 0x10000)
+        if (replay_start != 0 || replay_end != 0x14000)
         {
-            printf("Write replay: range mismatch: 0x%jx-0x%jx (expected 0-0x10000)\n", replay_start, replay_end);
+            printf("Write replay: range mismatch: 0x%jx-0x%jx (expected 0-0x14000)\n", replay_start, replay_end);
             assert(0);
         }
         for (auto op: replay_ops)
@@ -453,7 +454,8 @@ void test_writeback()
         { "client_max_dirty_ops", 2 },
     };
     timerfd_manager_t *tfd = new timerfd_manager_t([](int fd, bool wr, std::function<void(int, int)> callback){});
-    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t();
+    etcd_mock_t etcd_store;
+    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t(etcd_store);
     mock->pause();
     cluster_client_t *cli = new cluster_client_t(NULL, tfd, config, std::unique_ptr<etcd_state_client_t>(mock));
 
@@ -694,7 +696,8 @@ void test_deoptimize_snapshot_read()
         { "client_max_dirty_ops", 4 },
     };
     timerfd_manager_t *tfd = new timerfd_manager_t([](int fd, bool wr, std::function<void(int, int)> callback){});
-    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t();
+    etcd_mock_t etcd_store;
+    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t(etcd_store);
     mock->pause();
     cluster_client_t *cli = new cluster_client_t(NULL, tfd, config, std::unique_ptr<etcd_state_client_t>(mock));
 
@@ -1013,7 +1016,8 @@ void cluster_client_test_t::test_vault()
     json11::Json::object config;
     config["vault_url"] = "http://vault";
     timerfd_manager_t *tfd = new timerfd_manager_t([](int fd, bool wr, std::function<void(int, int)> callback){});
-    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t();
+    etcd_mock_t etcd_store;
+    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t(etcd_store);
     mock->pause();
     cluster_client_t *cli = new cluster_client_t(NULL, tfd, config, std::unique_ptr<etcd_state_client_t>(mock));
 
@@ -1073,7 +1077,8 @@ void test_pg_config_disappears()
 {
     json11::Json config;
     timerfd_manager_t *tfd = new timerfd_manager_t([](int fd, bool wr, std::function<void(int, int)> callback){});
-    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t();
+    etcd_mock_t etcd_store;
+    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t(etcd_store);
     mock->pause();
     cluster_client_t *cli = new cluster_client_t(NULL, tfd, config, std::unique_ptr<etcd_state_client_t>(mock));
 
@@ -1098,7 +1103,8 @@ void test_snapshot_over_nonexistent_pool()
 {
     json11::Json config;
     timerfd_manager_t *tfd = new timerfd_manager_t([](int fd, bool wr, std::function<void(int, int)> callback){});
-    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t();
+    etcd_mock_t etcd_store;
+    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t(etcd_store);
     mock->pause();
     cluster_client_t *cli = new cluster_client_t(NULL, tfd, config, std::unique_ptr<etcd_state_client_t>(mock));
 
@@ -1122,6 +1128,167 @@ void test_snapshot_over_nonexistent_pool()
     printf("[ok] test_snapshot_over_nonexistent_pool\n");
 }
 
+// Data loss reproduction: dirty buffer re-flush after PG primary change.
+//
+// A write buffer is acknowledged by OSD 2 while OSD 2 is the PG primary.
+// Then the primary moves back to OSD 1 and OSD 2 goes down.  The only copy
+// of the data acknowledged by the client was on OSD 2, so the client must
+// re-flush the buffer to the current primary (OSD 1) before SYNC is allowed
+// to report success.
+//
+// Before the fix the repeat logic used affects_osd() which compares the
+// crashed OSD against the *current* pg_config.cur_primary.  Since the primary
+// has already moved back to OSD 1, affects_osd(2) returns false, the buffer
+// is never re-flushed, and SYNC completes with 0 without ever writing the
+// data to a live OSD - silent data loss.
+void test_primary_change_flush_loss()
+{
+    json11::Json config;
+    timerfd_manager_t *tfd = new timerfd_manager_t([](int fd, bool wr, std::function<void(int, int)> callback){});
+    etcd_mock_t etcd_store;
+    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t(etcd_store);
+    mock->pause();
+    cluster_client_t *cli = new cluster_client_t(NULL, tfd, config, std::unique_ptr<etcd_state_client_t>(mock));
+
+    // Pool: replicated, OSDs [1, 2], primary OSD 1
+    configure_single_pg_pool(mock);
+    mock->resume();
+    pretend_connected(cli, 1);
+    pretend_connected(cli, 2);
+    cluster_client_test_t::continue_ops(cli);
+
+    // First PG primary change: 1 -> 2 (before any write, so the change itself
+    // must not post anything to any OSD)
+    mock->set("/vitastor/pg/state/1/1", json11::Json::object {
+        { "peers", json11::Json::array { 1, 2 } },
+        { "primary", 2 },
+        { "state", json11::Json::array { "active" } },
+    });
+    cluster_client_test_t::continue_ops(cli);
+    check_op_count(cli, 1, 0);
+    check_op_count(cli, 2, 0);
+
+    // Post a write while OSD 2 is primary - it becomes a REPEATING flush to OSD 2
+    int *r1 = test_write(cli, 0x2000, 0x1000, 0x56);
+    can_complete(r1);
+    cluster_client_test_t::continue_ops(cli);
+    osd_op_t *flush_op = find_op(cli, 2, OSD_OP_WRITE, 0x2000, 0x1000);
+    assert(flush_op != NULL);
+
+    // Second PG primary change: 2 -> 1.
+    // The buffer is still REPEATING (in-flight), so repeat logic skips it -
+    // nothing is re-queued and the in-flight op stays pointed at OSD 2
+    mock->set("/vitastor/pg/state/1/1", json11::Json::object {
+        { "peers", json11::Json::array { 1, 2 } },
+        { "primary", 1 },
+        { "state", json11::Json::array { "active" } },
+    });
+    cluster_client_test_t::continue_ops(cli);
+    check_op_count(cli, 1, 0);
+    check_op_count(cli, 2, 1);
+
+    // OSD 2 acknowledges the flush.  Now the only copy of the acknowledged
+    // data exists on OSD 2, which is no longer the primary
+    pretend_op_completed(cli, flush_op, 0);
+    check_completed(r1);
+
+    // OSD 2 crashes and restarts.  repeer_pgs(2) fires the repeat logic
+    pretend_disconnected(cli, 2);
+    pretend_connected(cli, 2);
+    cluster_client_test_t::continue_ops(cli);
+
+    // Data safety invariant: the data whose last successful write went to
+    // the now-dead OSD 2 must be re-flushed to the current primary OSD 1.
+    // The pre-fix affects_osd(2) check sees cur_primary == 1 and skips it.
+    osd_op_t *reflush_op = find_op(cli, 1, OSD_OP_WRITE, 0x2000, 0x1000);
+    if (reflush_op == NULL)
+    {
+        printf("error: the buffer acknowledged by crashed OSD 2 was not re-flushed to the new primary OSD 1;\n"
+            "SYNC would complete with no live OSD holding the data\n");
+    }
+    assert(reflush_op != NULL);
+    pretend_op_completed(cli, reflush_op, 0);
+
+    // Only now SYNC may legitimately complete
+    int *r2 = test_sync(cli);
+    can_complete(r2);
+    cluster_client_test_t::continue_ops(cli);
+    osd_op_t *sync_op1 = find_op(cli, 1, OSD_OP_SYNC, 0, 0);
+    assert(sync_op1 != NULL);
+    osd_op_t *sync_op2 = find_op(cli, 2, OSD_OP_SYNC, 0, 0);
+    assert(sync_op2 != NULL);
+    pretend_op_completed(cli, sync_op1, 0);
+    pretend_op_completed(cli, sync_op2, 0);
+    check_completed(r2);
+
+    delete cli;
+    delete tfd;
+    printf("[ok] primary change flush loss test\n");
+}
+
+// Data loss reproduction: SYNC failure must keep dirty state.
+//
+// SYNC is posted to the dirty OSD, then fails with -EIO.  The client must
+// re-post the SYNC and only then complete the user operation.  Before the fix
+// dirty_osds was cleared when posting the SYNC and never restored on failure,
+// so the retry concluded "sync is not required", completed the user SYNC with
+// 0 without any OSD ever being synced, and every subsequent SYNC became a
+// silent no-op too.
+void test_sync_fail_dirty_state_loss()
+{
+    // Retry immediately on -EIO
+    json11::Json config = json11::Json::object {
+        { "client_eio_retry_interval", 0 },
+    };
+    timerfd_manager_t *tfd = new timerfd_manager_t([](int fd, bool wr, std::function<void(int, int)> callback){});
+    etcd_mock_t etcd_store;
+    etcd_state_client_mock_t *mock = new etcd_state_client_mock_t(etcd_store);
+    mock->pause();
+    cluster_client_t *cli = new cluster_client_t(NULL, tfd, config, std::unique_ptr<etcd_state_client_t>(mock));
+
+    configure_single_pg_pool(mock);
+    mock->resume();
+    pretend_connected(cli, 1);
+    cluster_client_test_t::continue_ops(cli);
+
+    // Write + complete on OSD 1 -> buffer is CACHE_WRITTEN, dirty_osds = {1}
+    int *r1 = test_write(cli, 0x1000, 0x1000, 0x55);
+    can_complete(r1);
+    cluster_client_test_t::continue_ops(cli);
+    pretend_op_completed(cli, find_op(cli, 1, OSD_OP_WRITE, 0x1000, 0x1000), 0);
+    check_completed(r1);
+
+    // Post SYNC; it goes to OSD 1 and fails with -EIO
+    int *r2 = test_sync(cli);
+    can_complete(r2);
+    cluster_client_test_t::continue_ops(cli);
+    osd_op_t *sync_op = find_op(cli, 1, OSD_OP_SYNC, 0, 0);
+    assert(sync_op != NULL);
+    pretend_op_completed(cli, sync_op, -EIO);
+    cluster_client_test_t::continue_ops(cli);
+
+    // Data safety invariant: the failed SYNC must be re-posted to OSD 1 and
+    // the user SYNC must not complete until a real SYNC succeeds
+    if (*r2 != -1001)
+    {
+        printf("error: SYNC completed successfully even though OSD 1 never acknowledged a sync\n");
+    }
+    assert(*r2 == -1001);
+    osd_op_t *sync_op2 = find_op(cli, 1, OSD_OP_SYNC, 0, 0);
+    if (sync_op2 == NULL)
+    {
+        printf("error: failed SYNC was not re-posted to OSD 1 (dirty_osds was lost)\n");
+    }
+    assert(sync_op2 != NULL);
+    pretend_op_completed(cli, sync_op2, 0);
+    check_completed(r2);
+
+    delete cli;
+    delete tfd;
+    printf("[ok] sync fail dirty state loss test\n");
+}
+
+
 int main(int narg, char *args[])
 {
     test1();
@@ -1132,6 +1299,8 @@ int main(int narg, char *args[])
     test_deoptimize_snapshot_read();
     test_msgr_encrypt();
     test_msgr_decrypt_chain();
+    test_primary_change_flush_loss();
+    test_sync_fail_dirty_state_loss();
     cluster_client_test_t::test_vault();
     test_pg_config_disappears();
     test_snapshot_over_nonexistent_pool();

src/test/test_shit.cpp

@@ -1,578 +0,0 @@
-// Copyright (c) Vitaliy Filippov, 2019+
-// License: VNPL-1.1 (see README.md for details)
-
-#ifndef _LARGEFILE64_SOURCE
-#define _LARGEFILE64_SOURCE
-#endif
-
-#include <sys/types.h>
-#include <sys/ioctl.h>
-#include <sys/stat.h>
-#include <sys/time.h>
-#include <fcntl.h>
-#include <unistd.h>
-#include <stdint.h>
-#include <malloc.h>
-#include <linux/fs.h>
-#include <string.h>
-#include <errno.h>
-#include <assert.h>
-#include <stdio.h>
-#include <liburing.h>
-#include <math.h>
-
-#include <sys/socket.h>
-#include <sys/epoll.h>
-#include <netinet/in.h>
-#include <arpa/inet.h>
-
-#include <map>
-#include <vector>
-#include <deque>
-#include <algorithm>
-
-#include "blockstore.h"
-#include "blockstore_impl.h"
-#include "osd_peering_pg.cpp"
-//#include "cpp-btree/btree_map.h"
-
-static int setup_context(unsigned entries, struct io_uring *ring)
-{
-    int ret = io_uring_queue_init(entries, ring, 0);
-    if (ret < 0)
-    {
-        fprintf(stderr, "queue_init: %s\n", strerror(-ret));
-        return -1;
-    }
-    return 0;
-}
-
-static void test_write(struct io_uring *ring, int fd)
-{
-    struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
-    assert(sqe);
-    uint8_t *buf = (uint8_t*)memalign(512, 1024*1024*1024);
-    struct iovec iov = { buf, 1024*1024*1024 };
-    io_uring_prep_writev(sqe, fd, &iov, 1, 0);
-    io_uring_sqe_set_data(sqe, buf);
-    io_uring_submit_and_wait(ring, 1);
-    struct io_uring_cqe *cqe;
-    io_uring_peek_cqe(ring, &cqe);
-    int ret = cqe->res;
-    //int ret = writev(fd, &iov, 1);
-    if (ret < 0)
-        printf("cqe failed: %d %s\n", ret, strerror(-ret));
-    else
-        printf("result: %d user_data: %lld -> %lld\n", ret, sqe->user_data, cqe->user_data);
-    io_uring_cqe_seen(ring, cqe);
-    free(buf);
-}
-
-int main00(int argc, char *argv[])
-{
-    // queue with random removal: vector is best :D
-    // deque: 8.1s
-    // vector: 6.6s
-    // list: 9.3s
-    for (int i = 0; i < 10000; i++)
-    {
-        std::list<int> q;
-        for (int i = 0; i < 20480; i++)
-        {
-            for (auto it = q.begin(); it != q.end();)
-            {
-                if (rand() < RAND_MAX/2)
-                {
-                    //q.erase(it); -> for deque and vector
-                    auto p = it++;
-                    q.erase(p);
-                }
-                else
-                    it++;
-            }
-            q.push_back(rand());
-        }
-    }
-    return 0;
-}
-
-int main01(int argc, char *argv[])
-{
-    // deque: 2.091s
-    // vector: 18.733s
-    // list: 5.216s
-    // good, at least in this test deque is fine
-    for (int i = 0; i < 10000; i++)
-    {
-        std::deque<int> q;
-        for (int i = 0; i < 20480; i++)
-        {
-            int r = rand();
-            if (r < RAND_MAX/4 && q.size() > 0)
-                q.pop_front();
-                //q.erase(q.begin());
-            else
-                q.push_back(r);
-        }
-    }
-    return 0;
-}
-
-int main_vec(int argc, char *argv[])
-{
-    // vector: 16 elements -> 0.047s, 256 elements -> 1.622s, 1024 elements -> 16.087s, 2048 elements -> 55.8s
-    for (int i = 0; i < 100000; i++)
-    {
-        std::vector<iovec> v;
-        for (int i = 0; i < 2048; i++)
-        {
-            int r = rand();
-            auto it = v.begin();
-            for (; it != v.end(); it++)
-                if (it->iov_len >= r)
-                    break;
-            v.insert(it, (iovec){ .iov_base = 0, .iov_len = (size_t)r });
-        }
-    }
-    return 0;
-}
-
-int main_map(int argc, char *argv[])
-{
-    // map: 16 elements -> 0.105s, 256 elements -> 2.634s, 1024 elements -> 12.55s, 2048 elements -> 27.475s
-    // conclustion: vector is better in fulfill_read
-    for (int i = 0; i < 100000; i++)
-    {
-        std::map<int,iovec> v;
-        for (int i = 0; i < 2048; i++)
-        {
-            int r = rand();
-            v[r] = (iovec){ .iov_base = 0, .iov_len = (size_t)r };
-        }
-    }
-    return 0;
-}
-
-int main0(int argc, char *argv[])
-{
-    // std::map 5M entries monotone -> 2.115s, random -> 8.782s
-    // btree_map 5M entries monotone -> 0.458s, random -> 5.429s
-    // absl::btree_map 5M entries random -> 5.09s
-    // sparse_hash_map 5M entries -> 2.193s, random -> 2.586s
-    btree::btree_map<obj_ver_id, dirty_entry> dirty_db;
-    //std::map<obj_ver_id, dirty_entry> dirty_db;
-    //spp::sparse_hash_map<obj_ver_id, dirty_entry, obj_ver_hash> dirty_db;
-    for (int i = 0; i < 5000000; i++)
-    {
-        dirty_db[(obj_ver_id){
-            .oid = (object_id){
-                .inode = (uint64_t)rand(),
-                .stripe = (uint64_t)i,
-            },
-            .version = 1,
-        }] = (dirty_entry){
-            .state = BS_ST_SYNCED | BS_ST_BIG_WRITE,
-            .flags = 0,
-            .location = (uint64_t)i << 17,
-            .offset = 0,
-            .len = 1 << 17,
-        };
-    }
-    return 0;
-}
-
-int main1(int argc, char *argv[])
-{
-    std::vector<uint64_t> v1, v2;
-    v1.reserve(10000);
-    v2.reserve(10000);
-    for (int i = 0; i < 10000; i++)
-    {
-        v1.push_back(i);
-        v2.push_back(i);
-    }
-    for (int i = 0; i < 100000; i++)
-    {
-        // haha (core i5-2500 | i7-6800HQ)
-        // vector 10000 items: 4.37/100000 | 3.66
-        // vector 100000 items: 9.68/10000 | 0.95
-        // deque 10000 items: 28.432/100000
-        // list 10000 items: 320.695/100000
-        std::vector<uint64_t> v3;
-        v3.insert(v3.end(), v1.begin(), v1.end());
-        v3.insert(v3.end(), v2.begin(), v2.end());
-    }
-    return 0;
-}
-
-int main02(int argc, char *argv[])
-{
-    std::map<int, std::string> strs;
-    strs.emplace(12, "str");
-    auto it = strs.upper_bound(13);
-    //printf("s = %d %s %d\n", it->first, it->second.c_str(), it == strs.begin());
-    it--;
-    printf("%d\n", it == strs.end());
-    //printf("s = %d %s\n", it->first, it->second.c_str());
-    struct io_uring ring;
-    int fd = open("/dev/loop0", O_RDWR | O_DIRECT, 0644);
-    if (fd < 0)
-    {
-        perror("open infile");
-        return 1;
-    }
-    if (setup_context(32, &ring))
-        return 1;
-    test_write(&ring, fd);
-    close(fd);
-    io_uring_queue_exit(&ring);
-    return 0;
-}
-
-int main03(int argc, char *argv[])
-{
-    int listen_fd = socket(AF_INET, SOCK_STREAM, 0), enable = 1;
-    assert(listen_fd >= 0);
-    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(enable));
-    struct sockaddr_in bind_addr;
-    assert(inet_pton(AF_INET, "0.0.0.0", &bind_addr.sin_addr) == 1);
-    bind_addr.sin_family = AF_INET;
-    bind_addr.sin_port = htons(13892);
-    int r = bind(listen_fd, (sockaddr*)&bind_addr, sizeof(bind_addr));
-    if (r)
-    {
-        perror("bind");
-        return 1;
-    }
-    assert(listen(listen_fd, 128) == 0);
-    struct sockaddr_in peer_addr;
-    socklen_t peer_addr_size = sizeof(peer_addr);
-    int peer_fd = accept(listen_fd, (sockaddr*)&peer_addr, &peer_addr_size);
-    assert(peer_fd >= 0);
-    //fcntl(peer_fd, F_SETFL, fcntl(listen_fd, F_GETFL, 0) | O_NONBLOCK);
-
-    struct io_uring ring;
-    assert(setup_context(32, &ring) == 0);
-    void *buf = memalign(512, 4096*1024);
-
-    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
-    assert(sqe);
-    struct iovec iov = { buf, 4096*1024 };
-    struct msghdr msg = { 0 };
-    msg.msg_iov = &iov;
-    msg.msg_iovlen = 1;
-    io_uring_prep_recvmsg(sqe, peer_fd, &msg, 0);
-    io_uring_sqe_set_data(sqe, buf);
-    io_uring_submit_and_wait(&ring, 1);
-    struct io_uring_cqe *cqe;
-    io_uring_peek_cqe(&ring, &cqe);
-    int ret = cqe->res;
-    printf("cqe result: %d\n", ret);
-    // ok, io_uring's sendmsg always reads as much data as is available and finishes
-    io_uring_cqe_seen(&ring, cqe);
-    close(peer_fd);
-    close(listen_fd);
-    io_uring_queue_exit(&ring);
-
-    return 0;
-}
-
-int main04(int argc, char *argv[])
-{
-    /*spp::sparse_hash_set<obj_ver_id> osd1, osd2;
-    // fill takes 18.9 s
-    for (int i = 0; i < 1024*1024*8*2; i++)
-    {
-        obj_ver_id ovid = { { rand() % 500, rand() }, rand() };
-        osd1.insert(ovid);
-        osd2.insert(ovid);
-    }
-    for (int i = 0; i < 50000; i++)
-    {
-        obj_ver_id ovid = { { rand() % 500, rand() }, rand() };
-        osd1.insert(ovid);
-        ovid = { { rand() % 500, rand() }, rand() };
-        osd2.insert(ovid);
-    }
-    // diff takes only 2.3 s
-    spp::sparse_hash_set<obj_ver_id> osd1diff;
-    for (obj_ver_id e: osd1)
-    {
-        auto it = osd2.find(e);
-        if (it != osd2.end())
-            osd2.erase(it);
-        else
-            osd1diff.insert(e);
-    }*/
-    // fill vector takes 2 s
-    std::vector<obj_ver_role> to_sort;
-    to_sort.resize(1024*1024*8*2*3);
-    printf("Filling\n");
-    for (int i = 0; i < 1024*1024*8*2*3; i++)
-    {
-        to_sort[i] = {
-            .oid = (object_id){
-                .inode = (uint64_t)(rand() % 500),
-                .stripe = (uint64_t)rand(),
-            },
-            .version = (uint64_t)rand(),
-            .osd_num = (uint64_t)(rand() % 16),
-        };
-    }
-    printf("Sorting\n");
-    // sorting the whole array takes 7 s
-    // sorting in 3 parts... almost the same, 6 s
-    std::sort(to_sort.begin(), to_sort.end());
-    return 0;
-}
-
-uint64_t jumphash(uint64_t key, int count)
-{
-    uint64_t b = 0;
-    uint64_t seed = key;
-    for (int j = 1; j < count; j++)
-    {
-        seed = 2862933555777941757ull*seed + 3037000493ull; // LCPRNG
-        if (seed < (UINT64_MAX / (j+1)))
-        {
-            b = j;
-        }
-    }
-    return b;
-}
-
-void jumphash_prepare(int count, uint64_t *out_weights, uint64_t *in_weights)
-{
-    if (count <= 0)
-    {
-        return;
-    }
-    uint64_t total_weight = in_weights[0];
-    out_weights[0] = UINT64_MAX;
-    for (int j = 1; j < count; j++)
-    {
-        total_weight += in_weights[j];
-        out_weights[j] = UINT64_MAX / total_weight * in_weights[j];
-    }
-}
-
-uint64_t jumphash_weights(uint64_t key, int count, uint64_t *prepared_weights)
-{
-    uint64_t b = 0;
-    uint64_t seed = key;
-    for (int j = 1; j < count; j++)
-    {
-        seed = 2862933555777941757ull*seed + 3037000493ull; // LCPRNG
-        if (seed < prepared_weights[j])
-        {
-            b = j;
-        }
-    }
-    return b;
-}
-
-void jumphash3(uint64_t key, int count, uint64_t *weights, uint64_t *r)
-{
-    r[0] = 0;
-    r[1] = 1;
-    r[2] = 2;
-    uint64_t total_weight = weights[0]+weights[1]+weights[2];
-    uint64_t seed = key;
-    for (int j = 3; j < count; j++)
-    {
-        seed = 2862933555777941757ull*seed + 3037000493ull; // LCPRNG
-        total_weight += weights[j];
-        if (seed < UINT64_MAX*1.0*weights[j]/total_weight)
-            r[0] = j;
-        else
-        {
-            seed = 2862933555777941757ull*seed + 3037000493ull; // LCPRNG
-            if (seed < UINT64_MAX*1.0*weights[j]/total_weight)
-                r[1] = j;
-            else
-            {
-                seed = 2862933555777941757ull*seed + 3037000493ull; // LCPRNG
-                if (seed < UINT64_MAX*1.0*weights[j]/total_weight)
-                    r[2] = j;
-            }
-        }
-    }
-}
-
-uint64_t crush(uint64_t key, int count, uint64_t *weights)
-{
-    uint64_t b = 0;
-    uint64_t seed = 0;
-    uint64_t max = 0;
-    for (int j = 0; j < count; j++)
-    {
-        seed = (key + 0xc6a4a7935bd1e995 + (seed << 6) + (seed >> 2));
-        seed ^= (j + 0xc6a4a7935bd1e995 + (seed << 6) + (seed >> 2));
-        seed = 2862933555777941757ull*seed + 3037000493ull; // LCPRNG
-        seed = -log(((double)seed) / ((uint64_t)1 << 32) / ((uint64_t)1 << 32)) * weights[j];
-        if (seed > max)
-        {
-            max = seed;
-            b = j;
-        }
-    }
-    return b;
-}
-
-void crush3(uint64_t key, int count, uint64_t *weights, uint64_t *r, uint64_t total_weight)
-{
-    uint64_t seed = 0;
-    uint64_t max = 0;
-    for (int k1 = 0; k1 < count; k1++)
-    {
-        for (int k2 = k1+1; k2 < count; k2++)
-        {
-            if (k2 == k1)
-            {
-                continue;
-            }
-            for (int k3 = k2+1; k3 < count; k3++)
-            {
-                if (k3 == k1 || k3 == k2)
-                {
-                    continue;
-                }
-                seed = (key + 0xc6a4a7935bd1e995 + (seed << 6) + (seed >> 2));
-                seed ^= (k1 + 0xc6a4a7935bd1e995 + (seed << 6) + (seed >> 2));
-                seed ^= (k2 + 0xc6a4a7935bd1e995 + (seed << 6) + (seed >> 2));
-                seed ^= (k3 + 0xc6a4a7935bd1e995 + (seed << 6) + (seed >> 2));
-                seed = 2862933555777941757ull*seed + 3037000493ull; // LCPRNG
-                //seed = ((double)seed) / ((uint64_t)1 << 32) / ((uint64_t)1 << 32) * (weights[k1] + weights[k2] + weights[k3]);
-                seed = ((double)seed) / ((uint64_t)1 << 32) / ((uint64_t)1 << 32) * (1 -
-                    (1 - 1.0*weights[k1]/total_weight)*
-                    (1 - 1.0*weights[k2]/total_weight)*
-                    (1 - 1.0*weights[k3]/total_weight)
-                ) * UINT64_MAX;
-                if (seed > max)
-                {
-                    r[0] = k1;
-                    r[1] = k2;
-                    r[2] = k3;
-                    max = seed;
-                }
-            }
-        }
-    }
-}
-
-int main(int argc, char *argv[])
-{
-    int host_count = 6;
-    uint64_t host_weights[] = {
-        34609*3,
-        34931*3,
-        35850+36387+35859,
-        36387,
-        36387*2,
-        36387,
-    };
-    /*int osd_count[] = { 3, 3, 3, 1, 2 };
-    uint64_t osd_weights[][3] = {
-        { 34609, 34609, 34609 },
-        { 34931, 34931, 34931 },
-        { 35850, 36387, 35859 },
-        { 36387 },
-        { 36387, 36387 },
-    };*/
-    uint64_t total_weight = 0;
-    for (int i = 0; i < host_count; i++)
-    {
-        total_weight += host_weights[i];
-    }
-    uint64_t host_weights_prepared[host_count];
-    jumphash_prepare(host_count, host_weights_prepared, host_weights);
-    uint64_t total_pgs[host_count] = { 0 };
-    int pg_count = 256;
-    double uniformity[pg_count] = { 0 };
-    for (uint64_t pg = 1; pg <= pg_count; pg++)
-    {
-        uint64_t r[3];
-
-/*
-        // Select first host
-        //r[0] = jumphash_weights(pg, host_count, host_weights_prepared);
-        r[0] = crush(pg, host_count, host_weights);
-        // Select second host
-        uint64_t seed = pg;
-        r[1] = r[0];
-        while (r[1] == r[0])
-        {
-            seed = 2862933555777941757ull*seed + 3037000493ull; // LCPRNG
-            //r[1] = jumphash_weights(seed, host_count, host_weights_prepared);
-            r[1] = crush(seed, host_count, host_weights);
-        }
-        // Select third host
-        seed = pg;
-        r[2] = r[0];
-        while (r[2] == r[0] || r[2] == r[1])
-        {
-            seed = 2862933555777941757ull*seed + 3037000493ull; // LCPRNG
-            //r[2] = jumphash_weights(seed, host_count, host_weights_prepared);
-            r[2] = crush(seed, host_count, host_weights);
-        }
-*/
-
-/*
-        // Select second host
-        uint64_t host_weights1[host_count];
-        for (int i = 0; i < r[0]; i++)
-            host_weights1[i] = host_weights[i];
-        for (int i = r[0]+1; i < host_count; i++)
-            host_weights1[i-1] = host_weights[i];
-        r[1] = crush(pg, host_count-1, host_weights1);
-        // Select third host
-        for (int i = r[1]+1; i < host_count-1; i++)
-            host_weights1[i-1] = host_weights[i];
-        r[2] = crush(pg, host_count-2, host_weights1);
-        // Transform numbers
-        r[2] = r[2] >= r[1] ? 1+r[2] : r[2];
-        r[2] = r[2] >= r[0] ? 1+r[2] : r[2];
-        r[1] = r[1] >= r[0] ? 1+r[1] : r[1];
-*/
-
-        crush3(pg, host_count, host_weights, r, total_weight);
-        uint64_t shift = (2862933555777941757ull*pg + 3037000493ull) % host_count;
-        if (shift == 1)
-        {
-            uint64_t tmp;
-            tmp = r[0];
-            r[0] = r[1];
-            r[1] = r[2];
-            r[2] = tmp;
-        }
-        else if (shift == 2)
-        {
-            uint64_t tmp;
-            tmp = r[0];
-            r[0] = r[2];
-            r[2] = r[1];
-            r[1] = tmp;
-        }
-
-        total_pgs[r[0]]++;
-        total_pgs[r[1]]++;
-        total_pgs[r[2]]++;
-
-        double u = 0;
-        for (int i = 0; i < host_count; i++)
-        {
-            double d = abs(1 - total_pgs[i]/3.0/pg * total_weight/host_weights[i]);
-            u += d;
-        }
-        uniformity[pg-1] = u/host_count;
-
-        printf("pg %ju: hosts %ju, %ju, %ju ; avg deviation = %.2f\n", pg, r[0], r[1], r[2], u/host_count);
-    }
-    printf("total PGs: ");
-    for (int i = 0; i < host_count; i++)
-    {
-        printf(i > 0 ? ", %ju (%.2f)" : "%ju (%.2f)", total_pgs[i], total_pgs[i]/3.0/pg_count * total_weight/host_weights[i]);
-    }
-    printf("\n");
-    return 0;
-}

src/util/small_vector.h

@@ -0,0 +1,176 @@
+// Copyright (c) Vitaliy Filippov, 2019+
+// License: VNPL-1.1 or GNU GPL-2.0+ (see README.md for details)
+
+#pragma once
+
+#include <stddef.h>
+#include <new>
+#include <utility>
+#include <type_traits>
+
+// Minimal vector with small-buffer optimization: up to INLINE elements live
+// inside the object itself, heap is only used beyond that
+template <typename T, size_t INLINE = 4>
+class small_vector_t
+{
+public:
+    typedef T* iterator;
+    typedef const T* const_iterator;
+
+    small_vector_t() : data_ptr(inline_data()), cap(INLINE)
+    {
+    }
+
+    ~small_vector_t()
+    {
+        clear();
+        if (cap > INLINE)
+            delete[] reinterpret_cast<char*>(data_ptr);
+    }
+
+    small_vector_t(const small_vector_t &other) : small_vector_t()
+    {
+        reserve(other.sz);
+        for (size_t i = 0; i < other.sz; i++)
+            new (data_ptr + i) T(other.data_ptr[i]);
+        sz = other.sz;
+    }
+
+    small_vector_t(small_vector_t &&other) noexcept : small_vector_t()
+    {
+        if (other.cap > INLINE)
+        {
+            data_ptr = other.data_ptr;
+            cap = other.cap;
+            sz = other.sz;
+            other.data_ptr = other.inline_data();
+            other.sz = 0;
+            other.cap = INLINE;
+        }
+        else
+        {
+            for (; sz < other.sz; sz++)
+                new (data_ptr + sz) T(std::move(other.data_ptr[sz]));
+            destroy_other(other);
+        }
+    }
+
+    small_vector_t& operator=(const small_vector_t &other)
+    {
+        if (this != &other)
+        {
+            clear();
+            reserve(other.sz);
+            for (size_t i = 0; i < other.sz; i++)
+                new (data_ptr + i) T(other.data_ptr[i]);
+            sz = other.sz;
+        }
+        return *this;
+    }
+
+    small_vector_t& operator=(small_vector_t &&other) noexcept
+    {
+        if (this != &other)
+        {
+            clear();
+            if (cap > INLINE)
+                delete[] reinterpret_cast<char*>(data_ptr);
+            if (other.cap > INLINE)
+            {
+                data_ptr = other.data_ptr;
+                cap = other.cap;
+                sz = other.sz;
+                other.data_ptr = other.inline_data();
+                other.sz = 0;
+                other.cap = INLINE;
+            }
+            else
+            {
+                data_ptr = inline_data();
+                cap = INLINE;
+                for (sz = 0; sz < other.sz; sz++)
+                    new (data_ptr + sz) T(std::move(other.data_ptr[sz]));
+                destroy_other(other);
+            }
+        }
+        return *this;
+    }
+
+    template <typename... Args>
+    T& emplace_back(Args&&... args)
+    {
+        if (sz == cap)
+            grow();
+        return *(new (data_ptr + sz++) T(std::forward<Args>(args)...));
+    }
+
+    void push_back(const T &v) { emplace_back(v); }
+    void push_back(T &&v) { emplace_back(std::move(v)); }
+
+    void pop_back() { data_ptr[--sz].~T(); }
+
+    T& operator[](size_t i) { return data_ptr[i]; }
+    const T& operator[](size_t i) const { return data_ptr[i]; }
+    T& front() { return data_ptr[0]; }
+    T& back() { return data_ptr[sz-1]; }
+    const T& front() const { return data_ptr[0]; }
+    const T& back() const { return data_ptr[sz-1]; }
+
+    T* data() { return data_ptr; }
+    const T* data() const { return data_ptr; }
+    size_t size() const { return sz; }
+    size_t capacity() const { return cap; }
+    bool empty() const { return sz == 0; }
+    bool is_inline() const { return cap <= INLINE; }
+
+    iterator begin() { return data_ptr; }
+    iterator end() { return data_ptr + sz; }
+    const_iterator begin() const { return data_ptr; }
+    const_iterator end() const { return data_ptr + sz; }
+
+    void clear()
+    {
+        for (size_t i = 0; i < sz; i++)
+            data_ptr[i].~T();
+        sz = 0;
+    }
+
+    // Grows only; inline storage is kept unless n exceeds INLINE
+    void reserve(size_t n)
+    {
+        if (n <= cap)
+            return;
+        size_t new_cap = n > cap*2 ? n : cap*2;
+        T *nd = reinterpret_cast<T*>(new char[sizeof(T)*new_cap]);
+        for (size_t i = 0; i < sz; i++)
+            new (nd + i) T(std::move(data_ptr[i]));
+        for (size_t i = 0; i < sz; i++)
+            data_ptr[i].~T();
+        if (cap > INLINE)
+            delete[] reinterpret_cast<char*>(data_ptr);
+        data_ptr = nd;
+        cap = new_cap;
+    }
+
+private:
+    T *inline_data() { return reinterpret_cast<T*>(inline_store); }
+
+    // Moved-from inline elements must be destroyed explicitly:
+    // the source stays alive with sz = 0 and its destructor won't reach them
+    void destroy_other(small_vector_t &other)
+    {
+        for (size_t i = 0; i < other.sz; i++)
+            other.data_ptr[i].~T();
+        other.sz = 0;
+    }
+
+    void grow()
+    {
+        reserve(cap*2 > INLINE ? cap*2 : INLINE+1);
+    }
+
+    T *data_ptr;
+    size_t sz = 0;
+    size_t cap;
+    alignas(T) char inline_store[INLINE * sizeof(T)];
+};

tests/common.sh

@@ -31,7 +31,9 @@ OSD_TLS=${OSD_TLS:-1}
 RDMA=${RDMA:-0}
 
 RAMDISK=/run/user/$(id -u)
-findmnt $RAMDISK >/dev/null || (sudo mkdir -p $RAMDISK && sudo mount -t tmpfs tmpfs $RAMDISK)
+findmnt $RAMDISK >/dev/null || \
+    findmnt /run >/dev/null && [[ -d $RAMDISK && -w $RAMDISK ]] >/dev/null || \
+    (sudo mkdir -p $RAMDISK && sudo mount -t tmpfs tmpfs $RAMDISK)
 
 if [[ -z "$KEEP_DATA" ]]; then
     rm -rf ./testdata

tests/test_heal.sh

@@ -58,7 +58,7 @@ qemu-img convert -S 4096 -p \
     -O raw ./testdata/bin/read.bin
 
 if ! diff -q ./testdata/bin/read.bin ./testdata/bin/mirror.bin; then
-    build/src/test/bindiff ./testdata/bin/read.bin ./testdata/bin/mirror.bin
+    build/src/test/bindiff ./testdata/bin/read.bin ./testdata/bin/mirror.bin > ./testdata/bindiff.log
     format_error Data lost during self-heal
 fi