Run the lost primary scenario on EC 3+2 with pg_minsize 4 and v2 written
to all five OSDs: OSD 1 stabilizes its part and loses its peers, the four
others roll v2 back with a new version, and the stable v2 of OSD 1 is
outdated when it returns. A pg_minsize margin above pg_data_size doesn't
protect an interrupted write, and this is the case that the current rule
(n_roles >= pg_size) gets wrong without rollback-write.
Let peer_ec21_all() take the EC scheme and pg_minsize, and
configure_ec_pool() an optional pg_minsize.
Run the primary on Store v1 with enable_rollback_write and let the fake
peers report the feature, so that peering flushes of EC objects roll back
with a new version. Fake peers apply a flagged SEC_ROLLBACK as a rollback
followed by a synced unstable empty version, and make the versions it is
based on durable along with it, like the store does.
A plain rollback leaves stable pieces of the rolled back version on OSDs
absent from peering untouched. When such an OSD comes back, its piece is
the newest version again and a later peering can see it as a lone stable
part and mark the object incomplete or bring the rolled back version back.
When all OSDs holding the pieces of the target version support it, roll
the object back with BS_OP_ROLLBACK_WRITE to a new version of the current
PG epoch on all of them, and stabilize that version once every
rollback-write has succeeded. The new version outdates every version
listed by the peering, so the pieces left on absent OSDs become outdated.
Like writes, every flush batch now waits until the epoch is reported with
the current OSD set, and a batch waiting for it holds a repeer or a stop
of the PG like a write does. Otherwise a repeer takes the same epoch again
and writes the new version to its smaller OSD set without reporting it.
An incomplete object is rolled back as before: a new version on its
fewer than pg_data_size pieces would outdate the pieces of the target
version left on absent OSDs. An object with no target version gets an
empty new version only if every current OSD has a piece of it, because
an OSD without one may have an unflushed deletion there.
Replicated pools keep plain rollbacks, because any copy there is a whole
object, and so do objects with a piece on an OSD without rollback-write.
An OSD whose store has BS_OP_ROLLBACK_WRITE enabled reports the
rollback_write feature in SHOW_CONFIG, and peers remember it for the
connection like pg_locks. SEC_ROLLBACK with the new OSD_OP_ROLLBACK_WRITE
flag and the new version in the previously unused bytes of the packet
becomes BS_OP_ROLLBACK_WRITE on the secondary. A flag on the existing op
keeps its payload, PG lock and statistics handling as they are, and it
must only be sent to peers which report the feature.
Test peers of osd_test now go through check_peer_config() with features
set by the test.
Dump BS_HEAP_ROLLBACK_WRITE entries as rollback_write or
rollback_write_big with rollback_to, and accept them in write-meta,
which then writes metadata format 4. Accept format 4 in dump-meta and
resize, and keep it on resize.
Roll the listed objects back to the given versions and add op->version
on top of them as a new unstable version with the content of the
version rolled back to, like Store v1 does.
Each object gets a single entry flagged BS_HEAP_ROLLBACK_WRITE, which
also rolls back all older entries above the version stored at its end:
a zero-length small write with the external bitmap of the version
rolled back to, or an empty big write when there is nothing to roll
back to. So the operation only writes metadata and both of its steps
are persisted at once.
Like in Store v1, the empty version takes over the data block of the
newest big write it rolls back. That write must not be rolled back
already: then it's unstable, and compaction can't free the block in the
meantime. Both entries refer to the block until they become garbage
together, and loading the metadata counts it once, the way marking them
as garbage already does. A block is only allocated when there is no
such write. All objects are checked, and these blocks allocated, before
any entry is added.
Older versions refuse such entries or, with skip_corrupted_meta_entries,
silently drop them. So the operation is only allowed with the new
enable_rollback_write option, which upgrades the metadata superblock to
format 4 at start, and older versions refuse to start with format 4.
is_rollback_write_enabled() reports the option to the OSD.
Enable rollback-write for Store v2 in test_blockstore_chaos too. Store
v2 two-phase seeds (seed % 24 in 0..5) replay differently from here on,
among them 255885411.
Accept version 3 journals in dump-journal and in everything that reads
the journal, keep version 3 in resize, and write version 3 in
write-journal when the entries include a rollback-write.
prepare puts enable_rollback_write into the superblock of new OSDs, on
unless --enable_rollback_write 0 is given. Existing OSDs keep running
without it until it's added with update-sb.
Older versions take an unknown journal entry for the end of the journal,
so after a downgrade they would silently drop everything after the first
JE_ROLLBACK_WRITE. Make BS_OP_ROLLBACK_WRITE opt-in with the new
enable_rollback_write option, off by default, and when it's on, upgrade
the journal to version 3 durably at start: older versions refuse to start
with an unknown journal version instead. A version 3 journal stays one,
also when it's trimmed and when the option is turned off again.
Keep the journal version in memory for the trim, which used to take it
from the superblock buffer, never read from disk without inmemory_journal.
Add blockstore_i::is_rollback_write_enabled() for the OSD to check for the
operation: false for Store v2, configurable in the mock.
Dump them and accept them in write-journal. Keep the data block of an
empty new version allocated in resize and trim, and remap it in resize
like the block of a big write.
When peering rolls an EC object back on some OSDs while another one has
already committed its part, that lone stable part leaves the object
incomplete forever. A rollback which leaves a new version under a newer
PG epoch makes such leftovers outdated instead.
BS_OP_ROLLBACK_WRITE rolls the listed objects back like BS_OP_ROLLBACK
and puts op->version on top of each of them as a synced, but unstable,
zero-length write. The new version refers to the data and the external
bitmap of the version rolled back to without copying them, so it needs
no journal space for data, and BS_OP_STABLE commits it like any write.
It is journaled as a new JE_ROLLBACK_WRITE entry with both versions, the
bitmap and, for an empty version (below), its block. Journal space for
the commit is only reserved for objects which aren't unstable already.
Store v2 rejects the operation with -EINVAL for now.
If there is nothing to roll back to, the new version is an empty big
write, like a zero-length write to an absent object. It takes over the
data block of the big write it rolls back, which the rollback frees
anyway: the first write after a deletion or to an absent object is
always big. So a full device doesn't fail the operation where a plain
rollback succeeds. A block is only allocated when there is no such write.
Store v1 two-phase chaos seeds (seed % 24 in 12..17) replay differently
from here on.
Peering commits or rolls back an unstable EC version by sending STABILIZE
or ROLLBACK to every OSD separately. If the primary completes its own part
and loses its peers before theirs arrive, the next peering, without it,
decides on the remaining unstable parts alone and may roll back a version
that is already stable on the lost OSD. When it returns, that lone stable
part is the target and the object stays incomplete.
The test runs this for a version written to 2 of 3 OSDs and to all 3.
The current condition (n_roles >= pg_size) handles the first case and
fails the second; the one before 368daa4b (n_roles >= pg_data_size)
does the reverse.