Vitaliy Filippov
ea73857450
Add asserts to catch "BUG: Attempt to overwrite used offset"
2024-02-20 15:56:48 +03:00
Vitaliy Filippov
ca34a6047a
Fix dynamic journal space reservation: include the new write itself, too
2024-02-11 13:42:51 +03:00
Vitaliy Filippov
77c10fd1f8
In fact, do not autosync blockstore when autosync_writes=0
Test / test_move_reappear (push) Successful in 19s
Details
Test / test_rm (push) Successful in 14s
Details
Test / test_snapshot_down (push) Successful in 24s
Details
Test / test_snapshot_down_ec (push) Successful in 25s
Details
Test / test_splitbrain (push) Successful in 17s
Details
Test / test_snapshot_chain (push) Successful in 1m57s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m41s
Details
Test / test_rebalance_verify (push) Successful in 3m5s
Details
Test / test_rebalance_verify_imm (push) Successful in 2m26s
Details
Test / test_switch_primary (push) Successful in 45s
Details
Test / test_write (push) Successful in 33s
Details
Test / test_write_xor (push) Successful in 33s
Details
Test / test_rebalance_verify_ec (push) Successful in 3m42s
Details
Test / test_write_no_same (push) Successful in 14s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m57s
Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m24s
Details
Test / test_heal_csum_32k_dj (push) Successful in 4m29s
Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m10s
Details
Test / test_heal_csum_32k (push) Successful in 5m13s
Details
Test / test_scrub (push) Successful in 1m5s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m1s
Details
Test / test_scrub_xor (push) Successful in 1m2s
Details
Test / test_heal_csum_4k_dj (push) Successful in 5m2s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 57s
Details
Test / test_scrub_ec (push) Successful in 50s
Details
Test / test_scrub_pg_size_3 (push) Successful in 2m1s
Details
Test / test_heal_csum_4k (push) Successful in 4m40s
Details
Test / test_interrupted_rebalance (push) Successful in 1m38s
Details
Test / test_heal_pg_size_2 (push) Successful in 4m2s
Details
Test / test_heal_ec (push) Successful in 5m17s
Details
2024-02-03 20:37:36 +03:00
Vitaliy Filippov
1cec62d25d
Sync only completed writes
...
Test / test_move_reappear (push) Successful in 21s
Details
Test / test_rm (push) Successful in 16s
Details
Test / test_snapshot_down (push) Successful in 25s
Details
Test / test_snapshot_down_ec (push) Successful in 35s
Details
Test / test_splitbrain (push) Successful in 24s
Details
Test / test_interrupted_rebalance (push) Successful in 5m14s
Details
Test / test_snapshot_chain (push) Successful in 2m50s
Details
Test / test_rebalance_verify_imm (push) Successful in 2m47s
Details
Test / test_rebalance_verify (push) Successful in 3m42s
Details
Test / test_switch_primary (push) Successful in 33s
Details
Test / test_write (push) Successful in 42s
Details
Test / test_write_xor (push) Successful in 44s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m52s
Details
Test / test_write_no_same (push) Successful in 15s
Details
Test / test_rebalance_verify_ec (push) Successful in 4m19s
Details
Test / test_heal_ec (push) Successful in 6m20s
Details
Test / test_heal_csum_32k (push) Successful in 3m29s
Details
Test / test_scrub (push) Successful in 1m24s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m11s
Details
Test / test_heal_csum_4k_dmj (push) Successful in 4m23s
Details
Test / test_scrub_xor (push) Successful in 1m9s
Details
Test / test_heal_csum_4k_dj (push) Successful in 5m29s
Details
Test / test_heal_csum_4k (push) Successful in 5m36s
Details
Test / test_scrub_pg_size_3 (push) Successful in 1m53s
Details
Test / test_scrub_ec (push) Successful in 29s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m9s
Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m13s
Details
Test / test_heal_csum_32k_dj (push) Successful in 4m17s
Details
Test / test_snapshot_chain_ec (push) Successful in 1m25s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Failing after 24s
Details
Should be a final remaining fix to EC + non-capacitor (non-immediate-commit) write hangs :).
First it was breaking non-EC ("instantly stable") writes because they sometimes
complete out of order which was leading to the following error:
terminate called after throwing an instance of 'std::runtime_error'
what(): BUG: Unexpected dirty_entry 1000000000001:29480000 v65540 unstable state during flush: 0x151
But it is easily fixed by scanning previous and next dirty_entries in mark_stable.
2024-01-27 15:17:22 +03:00
Vitaliy Filippov
a5e3dfbc5a
Oops, 1.3.0 needs a hotfix
Test / test_snapshot_ec (push) Successful in 30s
Details
Test / test_move_reappear (push) Successful in 19s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s
Details
Test / test_rm (push) Successful in 18s
Details
Test / test_snapshot_down (push) Successful in 30s
Details
Test / test_snapshot_down_ec (push) Successful in 31s
Details
Test / test_splitbrain (push) Successful in 27s
Details
Test / test_snapshot_chain (push) Successful in 2m13s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m56s
Details
Test / test_rebalance_verify_imm (push) Successful in 2m51s
Details
Test / test_rebalance_verify (push) Successful in 3m38s
Details
Test / test_write (push) Successful in 45s
Details
Test / test_write_no_same (push) Successful in 13s
Details
Test / test_rebalance_verify_ec (push) Successful in 4m5s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m22s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m51s
Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m39s
Details
Test / test_heal_ec (push) Successful in 6m39s
Details
Test / test_heal_csum_32k_dj (push) Successful in 5m55s
Details
Test / test_heal_csum_32k (push) Successful in 6m5s
Details
Test / test_scrub (push) Successful in 1m18s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m19s
Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m25s
Details
Test / test_scrub_xor (push) Successful in 50s
Details
Test / test_scrub_pg_size_3 (push) Successful in 1m46s
Details
Test / test_heal_csum_4k_dj (push) Successful in 6m17s
Details
Test / test_heal_csum_4k (push) Successful in 5m51s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m0s
Details
Test / test_scrub_ec (push) Successful in 21s
Details
Test / test_write_xor (push) Successful in 33s
Details
2023-12-04 13:45:54 +03:00
Vitaliy Filippov
cf36445359
Reserve journal space for stabilize requests dynamically to prevent stalls
2023-11-20 03:01:57 +03:00
Vitaliy Filippov
1f509cca77
Fix unused capture warnings and void* arithmetic (clang)
Test / test_minsize_1 (push) Successful in 14s
Details
Test / test_snapshot_ec (push) Successful in 40s
Details
Test / test_rm (push) Successful in 16s
Details
Test / test_move_reappear (push) Successful in 18s
Details
Test / test_snapshot_down (push) Successful in 31s
Details
Test / test_snapshot_down_ec (push) Successful in 33s
Details
Test / test_splitbrain (push) Successful in 22s
Details
Test / test_snapshot_chain (push) Successful in 2m31s
Details
Test / test_snapshot_chain_ec (push) Failing after 3m7s
Details
Test / test_rebalance_verify_imm (push) Successful in 3m6s
Details
Test / test_write (push) Successful in 39s
Details
Test / test_rebalance_verify (push) Successful in 4m7s
Details
Test / test_write_no_same (push) Successful in 13s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m23s
Details
Test / test_rebalance_verify_ec (push) Successful in 4m20s
Details
Test / test_write_xor (push) Failing after 3m9s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m55s
Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m44s
Details
Test / test_heal_csum_32k_dj (push) Successful in 6m8s
Details
Test / test_heal_csum_32k (push) Successful in 5m58s
Details
Test / test_heal_ec (push) Failing after 10m16s
Details
Test / test_heal_csum_4k_dmj (push) Successful in 5m57s
Details
Test / test_scrub (push) Successful in 1m8s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 59s
Details
Test / test_scrub_xor (push) Successful in 47s
Details
Test / test_heal_csum_4k_dj (push) Successful in 5m30s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m7s
Details
Test / test_scrub_pg_size_3 (push) Successful in 1m34s
Details
Test / test_heal_csum_4k (push) Successful in 5m21s
Details
Test / test_scrub_ec (push) Successful in 43s
Details
2023-11-04 14:55:12 +03:00
Vitaliy Filippov
5066e35a49
Fix write-over-delete failing for the very first entry in dirty_db
Test / test_minsize_1 (push) Successful in 17s
Details
Test / test_snapshot_ec (push) Successful in 38s
Details
Test / test_rm (push) Successful in 17s
Details
Test / test_snapshot_down (push) Failing after 27s
Details
Test / test_move_reappear (push) Failing after 53s
Details
Test / test_snapshot_down_ec (push) Failing after 27s
Details
Test / test_splitbrain (push) Successful in 25s
Details
Test / test_snapshot_chain (push) Successful in 2m28s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m59s
Details
Test / test_rebalance_verify (push) Successful in 3m22s
Details
Test / test_rebalance_verify_imm (push) Successful in 3m21s
Details
Test / test_write (push) Successful in 42s
Details
Test / test_write_xor (push) Successful in 48s
Details
Test / test_write_no_same (push) Successful in 14s
Details
Test / test_rebalance_verify_ec (push) Successful in 4m41s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m8s
Details
Test / test_heal_pg_size_2 (push) Successful in 4m17s
Details
Test / test_heal_ec (push) Successful in 4m33s
Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m23s
Details
Test / test_heal_csum_32k_dj (push) Successful in 6m18s
Details
Test / test_heal_csum_32k (push) Successful in 6m44s
Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m53s
Details
Test / test_scrub (push) Successful in 1m11s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m5s
Details
Test / test_scrub_xor (push) Successful in 1m1s
Details
Test / test_heal_csum_4k_dj (push) Successful in 6m58s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m2s
Details
Test / test_scrub_pg_size_3 (push) Successful in 1m41s
Details
Test / test_scrub_ec (push) Successful in 37s
Details
Test / test_heal_csum_4k (push) Successful in 6m4s
Details
2023-10-21 17:00:14 +03:00
Vitaliy Filippov
f600cc07b0
Autosync in blockstore every autosync_writes, too
2023-09-16 17:52:17 +03:00
Vitaliy Filippov
5cadb170b9
Fix possible OSD crash during sync due to missing min_flushed_journal_sector reset
2023-09-16 17:52:17 +03:00
Vitaliy Filippov
3c8e4c6b72
Use clean_dyn_size for space check
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
c5274f655b
...and partially remove the perversion with bitmap inlining
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
45e07d6294
Sadly we have to refcount dyn_data...
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
213a9ccb4d
Verify checksums during journal reads
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
7d532880c3
Implement large csum_block_size support (more than 4k) + refactor blockstore_flush
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
0b0405d115
Implement bitmap-granular (4k) metadata & data checksums
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
b7e4d0c9bf
Fix journal dirty_start position tracking and some debug prints
...
Fixes two bugs found during HDD testing :-)
1) OSD crashed with "BUG: Attempt to overwrite used offset of the journal" during
`fio -bs=900k -iodepth=128` test with 16 MB journal
2) OSD stalled during `fio -bs=512k -iodepth=128` test with 64 MB journal
2023-07-09 01:17:55 +03:00
Vitaliy Filippov
86b4682975
Put get_trim_pos into the "critical section". Fixes rare journal corruption issue
...
The consequence of this issue was that in some very rare cases (only reproduced
under load in CI when running 4+ tests in parallel) small write data written to
journal could overwrite journal entries.
Also add an assert-type safety check to be able to catch this issue in the
future again in case of a regression.
2023-06-17 00:06:42 +03:00
Vitaliy Filippov
4bfd994341
Sync unsynced deletes before overwriting them with a lower version
2023-05-20 23:19:39 +03:00
Vitaliy Filippov
a409598b16
Wait for free space again, but count on big_write flushes instead of just flusher activity
2023-05-10 01:51:02 +03:00
Vitaliy Filippov
73940adf07
Prioritize EC (non-instantly-stable) operations under journal pressure
...
This reduces the probability of hitting OSD stalls with EC due to "deadlocks"
where two parallel write operations wait for each other to complete
2023-01-03 00:05:45 +03:00
Vitaliy Filippov
e950c024d3
Do not sync peer OSDs before listing
...
Sync before listing was added to wait for all PG writes possibly left in queue
from the previous master to finish before listing it
But in fact it may block the cluster when EC is used and some unstable writes
are left in the queue - they block journal flushing, rollback/stabilize is
required to unblock them, but rollback/stabilize may only happen after PG is
peered. But peering needs listings, listings are requested only after sync, and
sync itself waits for currently blocked writes waiting in the queue
2023-01-03 00:05:45 +03:00
Vitaliy Filippov
71d6d9f868
Fix possible crash on ENOSPC during operation cancel in blockstore
2023-01-03 00:05:45 +03:00
Vitaliy Filippov
1d80bcc8d0
Fix blockstore returning garbage for unstable reads if there is an in-flight version
...
"In-flight" versions are added into dirty_db when writes are enqueued. And they
weren't ignored by subsequent reads even though they didn't have data location yet.
This bug was leading to test_heal.sh not passing sometimes with replicated setups.
2022-12-21 02:48:24 +03:00
Vitaliy Filippov
552e207d2b
Explicitly print errors about -EAGAIN in io_uring
2022-12-17 15:49:49 +03:00
Vitaliy Filippov
cb437913d3
Never try to wait for free space inside blockstore
2022-12-12 00:27:05 +03:00
Vitaliy Filippov
6e6f6ecbb0
Add missing journal space check for big_writes
...
Fixes possible `prefill_single_journal_entry(): Assertion `!journal.sector_info[journal.cur_sector].flush_count' failed` error
2022-11-20 00:50:13 +03:00
Vitaliy Filippov
b4edc79449
Fix possible segfault on ENOSPC
2022-11-12 11:59:43 +03:00
Vitaliy Filippov
d83580bd68
Fix "tried to overwrite non-zero metadata entry" when during a previous metadata
...
flush writing new entry is completed, but zeroing out an old one isn't
2022-08-21 00:31:18 +03:00
Vitaliy Filippov
dfd80626bd
Extract disk opening functions to separate module
2022-07-15 01:38:30 +03:00
Vitaliy Filippov
73a363bf92
Rename some variables and constants
2022-07-15 01:38:30 +03:00
Vitaliy Filippov
79ebda933e
Fix a write hang with throttling due to timer reenterability / triggerability
2022-03-28 01:42:06 +03:00
Vitaliy Filippov
839ec9e6e0
Shard clean_db by PGs to speedup listings
2022-02-20 00:21:24 +03:00
Vitaliy Filippov
2a0b881685
Respect max_write_iodepth for deletes
2022-01-23 22:05:23 +03:00
Vitaliy Filippov
7bdd92ca4f
Fix build under clang and some warnings
...
Build problems fixed:
- void* pointer arithmetic which is a GNU extension (works as byte*)
- "variable size object may not be initialized" which is OK under GCC
- nullptr_t related error in json11 (it lacks 'operator <' in clang)
Warnings fixed:
- empty nested struct initializer { 0 } replaced by {}
- removed several unused lambda captures
2022-01-16 00:02:54 +03:00
Vitaliy Filippov
f93491bc6c
Implement journal write batching and slightly refactor journal writes
...
Slightly reduces WA. For example, in 4K T1Q128 replicated randwrite tests
WA is reduced from ~3.6 to ~3.1, in T1Q64 from ~3.8 to ~3.4.
Only effective without no_same_sector_overwrites.
2021-12-16 00:27:17 +03:00
JiangYu
e86df4a2a2
fix BLOCKSTORE_DEBUG, error: ‘dirty_it’ was not declared in this scope
...
Signed-off-by: JiangYu <lnsyyj@hotmail.com>
2021-07-18 00:46:05 +08:00
Vitaliy Filippov
2ab423d4ef
Implement journaled write throttling for the SSD+HDD case
2021-04-10 17:44:12 +03:00
Vitaliy Filippov
591cad09c9
Fix bitmaps for objects larger than 128K
2021-04-10 17:44:12 +03:00
Vitaliy Filippov
b907ad50aa
Oops, forgot to add external bitmaps to blockstore in some places
2021-04-10 17:44:12 +03:00
Vitaliy Filippov
ab39ce2bbb
Use clean_entry_bitmap_size instead of entry_attr_size back because of changed bitmap handling
2021-04-10 17:44:12 +03:00
Vitaliy Filippov
d0c2e31312
Add a test for snapshots, fix bugs. Now the test passes
2021-04-10 17:44:12 +03:00
Vitaliy Filippov
004f265393
Remove cryptic bitmap inlining from bs_op_t and osd_op_t, use bitmap in primary OSD code
2021-04-10 17:44:12 +03:00
Vitaliy Filippov
6107a4d07b
Add "external" bitmap support to blockstore
2021-04-10 17:44:12 +03:00
Vitaliy Filippov
564d64e271
Add some details for debug prints
2021-03-25 11:00:10 +03:00
Vitaliy Filippov
b0ad1e1e6d
Remember writes as "unsynced" only after completing them
...
Previously BS_OP_SYNC could take unfinished writes and add them into the journal before
they were actually completed. This was leading to crashes with the message
"BUG: Unexpected dirty_entry 2000000000001:9f2a0000 v3 unstable state during flush: 338"
2021-03-25 02:41:58 +03:00
Vitaliy Filippov
299d7d7c95
Use common macro for get_sqe
2021-03-13 17:19:45 +03:00
Vitaliy Filippov
98f1e2c277
Rework write/sync ordering
...
Make syncs wait for all previous writes because it's the only way
to make sure that OSDs do not receive incomplete writes in LIST results
during peering when some writes are still in progress.
Also simplify blockstore submission queue logic.
2021-03-08 17:04:10 +03:00
Vitaliy Filippov
314b20437b
Do not break subsequent small writes badly when a big write is canceled
2021-03-08 17:04:10 +03:00
Vitaliy Filippov
41fd14e024
Fix deletes not increasing write_iodepth
2021-02-28 19:36:59 +03:00