Vitaliy Filippov
37d14e35f3
Save cached_read_* in superblock
2023-07-17 00:34:58 +03:00
Vitaliy Filippov
8c4a11b51c
Document cached_read_{data,meta,journal} parameters
2023-07-17 00:34:58 +03:00
Vitaliy Filippov
98d5849190
Support using Linux page cache for reads
2023-07-17 00:34:58 +03:00
Vitaliy Filippov
e4ea8a9514
Track used blocks, not object versions
Test / test_snapshot_ec (push) Successful in 19s
Details
Test / test_minsize_1 (push) Successful in 11s
Details
Test / test_move_reappear (push) Successful in 17s
Details
Test / test_rm (push) Successful in 12s
Details
Test / test_snapshot_chain (push) Successful in 1m5s
Details
Test / test_snapshot_chain_ec (push) Successful in 1m27s
Details
Test / test_snapshot_down (push) Successful in 21s
Details
Test / test_snapshot_down_ec (push) Successful in 20s
Details
Test / test_splitbrain (push) Successful in 12s
Details
Test / test_rebalance_verify (push) Successful in 3m25s
Details
Test / test_rebalance_verify_imm (push) Successful in 3m36s
Details
Test / test_rebalance_verify_ec (push) Successful in 3m55s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m52s
Details
Test / test_write (push) Successful in 36s
Details
Test / test_write_xor (push) Successful in 41s
Details
Test / test_write_no_same (push) Successful in 15s
Details
Test / test_heal_pg_size_2 (push) Successful in 5m32s
Details
Test / test_heal_ec (push) Successful in 7m7s
Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m46s
Details
Test / test_heal_csum_32k_dj (push) Successful in 5m44s
Details
Test / test_heal_csum_32k (push) Successful in 7m19s
Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m38s
Details
Test / test_heal_csum_4k_dj (push) Successful in 6m33s
Details
Test / test_heal_csum_4k (push) Successful in 6m21s
Details
Test / test_scrub (push) Successful in 58s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 56s
Details
Test / test_scrub_xor (push) Successful in 39s
Details
Test / test_scrub_pg_size_3 (push) Successful in 40s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 33s
Details
Test / test_scrub_ec (push) Successful in 34s
Details
2023-07-17 00:34:58 +03:00
Vitaliy Filippov
3c565e7b94
Document data_csum_type and csum_block_size parameters
2023-07-17 00:34:58 +03:00
Vitaliy Filippov
708918a4c7
Remove creepy "metadata copying" during overwrite
...
Instead of it, just do not verify checksums of currently mutated objects.
When clean data modification during flush runs in parallel to a read request,
that request may read a mix of old and new data. It may even read a mix of
multiple flushed versions if it lasts too long... And attempts to verify it
using temporary copies of metadata make the algorithm too complex and creepy.
2023-07-15 02:34:20 +03:00
Vitaliy Filippov
8e099c1d11
Support keeping checksums on disk (not in memory)
...
Definitely beneficial for SSD+HDD setups
2023-07-14 00:38:15 +03:00
Vitaliy Filippov
debb00a535
Check for "Checksum mismatch" and "BUG" messages during test_heal
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
c8891ab1d6
Use clean_dyn_size for space check
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
7062b73d87
Log more details about checksum mismatch in big_writes
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
02e24f5144
Use find_holes() in flusher for unification
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
55f506f6e0
Fill journal header to know checksum type & size when dumping journal with --all
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
e12dd9b82c
Fix journal read checksum verification with inmemory_journal=false
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
82d8848b8f
Call fill_partial_checksum_blocks() correctly in regard to COPY_BUF_CSUM_FILL
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
ef800408dc
Wait for journal reads before checking them in clear_incomplete_csum_block_bits
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
3b1150c478
Check for checksum mismatch absence in test_heal
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
ccdf87dc81
Use zero checksum size for zero-length writes
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
bede73d158
Fix journal data checksum mangling on corrupted block overwrite
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
8e35319a34
Check journal entry size when checking block checksums
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
dc6e88e2ca
Fix journal data checksum verification on start
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
4c3370220b
Add more details to "journal entry data is corrupt" messages
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
f02344c0a4
...and partially remove the perversion with bitmap inlining
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
b369032665
Sadly we have to refcount dyn_data...
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
319b0833eb
Fix clean block checksum read
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
0641b06fb1
Allow to forcibly set meta_format
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
3b9873b9a9
Rename meta_version to meta_format
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
eef97a0dc4
Support old metadata format in vitastor-disk dump-meta
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
5b16e5ab5b
Fix journal big_write simple reads after checksum changes
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
bb430fccd5
Verify checksums in test_heal in different combinations
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
bafadd5559
Fix bitmap-granular checksums
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
6ebca5fedc
Fix wait_journal_count not being zeroed
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
e3e2325ef5
Rewrite and fix find_holes into a more obvious version
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
9a908f3e66
Fix missing checksum read offset
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
89f6fef920
Add a test for checksums
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
63b9382067
Fix checksum verification in big_write journal reads
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
8a7dea9fa2
Verify checksums during journal reads
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
1adf77f8fb
Add backwards compatibility with non-checksum metadata and journal formats
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
7a530346a6
Implement large csum_block_size support (more than 4k) + refactor blockstore_flush
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
cc1f03971d
Implement bitmap-granular (4k) metadata & data checksums
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
a0aac7eb2a
Update drives
Test / test_interrupted_rebalance (push) Failing after 1m42s
Details
Test / test_interrupted_rebalance_imm (push) Failing after 1m37s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m48s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m25s
Details
Test / test_failure_domain (push) Successful in 19s
Details
Test / test_snapshot (push) Successful in 25s
Details
Test / test_snapshot_ec (push) Successful in 27s
Details
Test / test_minsize_1 (push) Successful in 15s
Details
Test / test_move_reappear (push) Failing after 49s
Details
Test / test_rm (push) Successful in 16s
Details
Test / test_snapshot_chain (push) Successful in 2m24s
Details
Test / test_snapshot_chain_ec (push) Successful in 3m0s
Details
Test / test_snapshot_down (push) Successful in 23s
Details
Test / test_snapshot_down_ec (push) Successful in 25s
Details
Test / test_splitbrain (push) Successful in 22s
Details
Test / test_rebalance_verify (push) Successful in 3m39s
Details
Test / test_rebalance_verify_imm (push) Successful in 3m38s
Details
Test / test_rebalance_verify_ec (push) Successful in 5m29s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m32s
Details
Test / test_write (push) Successful in 54s
Details
Test / test_write_xor (push) Successful in 1m5s
Details
Test / test_write_no_same (push) Successful in 15s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m51s
Details
Test / test_heal_ec (push) Successful in 4m16s
Details
Test / test_scrub (push) Successful in 35s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 32s
Details
Test / test_scrub_xor (push) Successful in 32s
Details
Test / test_scrub_pg_size_3 (push) Successful in 55s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 48s
Details
Test / test_scrub_ec (push) Successful in 27s
Details
2023-07-13 01:49:45 +03:00
Vitaliy Filippov
ac7b834af3
Disable journal_no_same_sector_overwrites by default for HDD-only
Test / test_interrupted_rebalance (push) Successful in 1m17s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m29s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m31s
Details
Test / test_failure_domain (push) Successful in 9s
Details
Test / test_snapshot (push) Successful in 20s
Details
Test / test_snapshot_ec (push) Successful in 21s
Details
Test / test_minsize_1 (push) Successful in 12s
Details
Test / test_move_reappear (push) Successful in 39s
Details
Test / test_rm (push) Successful in 12s
Details
Test / test_snapshot_chain (push) Successful in 1m10s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m9s
Details
Test / test_snapshot_down (push) Successful in 21s
Details
Test / test_snapshot_down_ec (push) Successful in 22s
Details
Test / test_splitbrain (push) Successful in 14s
Details
Test / test_rebalance_verify (push) Successful in 2m56s
Details
Test / test_rebalance_verify_imm (push) Successful in 2m53s
Details
Test / test_rebalance_verify_ec (push) Successful in 5m39s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m44s
Details
Test / test_write (push) Successful in 49s
Details
Test / test_write_xor (push) Successful in 59s
Details
Test / test_write_no_same (push) Successful in 13s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m41s
Details
Test / test_heal_ec (push) Successful in 3m49s
Details
Test / test_scrub (push) Successful in 33s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 30s
Details
Test / test_scrub_xor (push) Successful in 25s
Details
Test / test_scrub_pg_size_3 (push) Successful in 40s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 35s
Details
Test / test_scrub_ec (push) Successful in 33s
Details
2023-07-10 00:34:35 +03:00
Vitaliy Filippov
ee0c78fd74
Fix default HDD block size in docs (actual size is 1 MB)
2023-07-09 13:12:30 +03:00
Vitaliy Filippov
e6646a5b2f
Bump QEMU version to vitastor3
2023-07-09 13:01:04 +03:00
Vitaliy Filippov
ae69662b17
Add "Recommended drives"
2023-07-09 12:59:18 +03:00
Vitaliy Filippov
57ad4c3636
Add a note about HDD, enable throttling only for hybrid OSDs
Test / test_interrupted_rebalance (push) Successful in 2m4s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m30s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m45s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m23s
Details
Test / test_failure_domain (push) Successful in 9s
Details
Test / test_snapshot (push) Successful in 19s
Details
Test / test_snapshot_ec (push) Successful in 19s
Details
Test / test_minsize_1 (push) Successful in 12s
Details
Test / test_move_reappear (push) Failing after 1m32s
Details
Test / test_rm (push) Successful in 12s
Details
Test / test_snapshot_chain (push) Successful in 1m8s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m2s
Details
Test / test_snapshot_down (push) Successful in 22s
Details
Test / test_snapshot_down_ec (push) Successful in 21s
Details
Test / test_splitbrain (push) Successful in 13s
Details
Test / test_rebalance_verify (push) Successful in 2m52s
Details
Test / test_rebalance_verify_imm (push) Successful in 2m46s
Details
Test / test_rebalance_verify_ec (push) Successful in 5m9s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m5s
Details
Test / test_write (push) Successful in 40s
Details
Test / test_write_xor (push) Successful in 49s
Details
Test / test_write_no_same (push) Successful in 15s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s
Details
Test / test_heal_ec (push) Successful in 4m35s
Details
Test / test_scrub (push) Successful in 37s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 40s
Details
Test / test_scrub_xor (push) Successful in 31s
Details
Test / test_scrub_pg_size_3 (push) Successful in 40s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 26s
Details
Test / test_scrub_ec (push) Successful in 22s
Details
2023-07-09 12:45:11 +03:00
Vitaliy Filippov
b7e4d0c9bf
Fix journal dirty_start position tracking and some debug prints
...
Fixes two bugs found during HDD testing :-)
1) OSD crashed with "BUG: Attempt to overwrite used offset of the journal" during
`fio -bs=900k -iodepth=128` test with 16 MB journal
2) OSD stalled during `fio -bs=512k -iodepth=128` test with 64 MB journal
2023-07-09 01:17:55 +03:00
Vitaliy Filippov
161a23c966
Support reloading state when etcd says "revisions were compacted"
...
Test / test_interrupted_rebalance (push) Successful in 3m9s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m38s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m54s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m36s
Details
Test / test_failure_domain (push) Successful in 9s
Details
Test / test_snapshot (push) Successful in 23s
Details
Test / test_snapshot_ec (push) Successful in 22s
Details
Test / test_minsize_1 (push) Successful in 14s
Details
Test / test_move_reappear (push) Successful in 19s
Details
Test / test_rm (push) Successful in 12s
Details
Test / test_snapshot_chain (push) Successful in 2m2s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m38s
Details
Test / test_snapshot_down (push) Successful in 21s
Details
Test / test_snapshot_down_ec (push) Successful in 24s
Details
Test / test_splitbrain (push) Successful in 15s
Details
Test / test_rebalance_verify (push) Successful in 3m10s
Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s
Details
Test / test_rebalance_verify_ec (push) Successful in 3m27s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 6m2s
Details
Test / test_write (push) Successful in 35s
Details
Test / test_write_xor (push) Successful in 45s
Details
Test / test_write_no_same (push) Successful in 22s
Details
Test / test_heal_pg_size_2 (push) Successful in 4m0s
Details
Test / test_heal_ec (push) Successful in 3m52s
Details
Test / test_scrub (push) Successful in 1m1s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 42s
Details
Test / test_scrub_xor (push) Successful in 34s
Details
Test / test_scrub_pg_size_3 (push) Successful in 53s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 45s
Details
Test / test_scrub_ec (push) Successful in 26s
Details
Before this change, OSDs almost always died when one of the etcds was restarted,
even though the rest of them was still in quorum and the lease was still active
2023-07-07 01:33:48 +03:00
Vitaliy Filippov
2f999d8607
Reduce etcd memory usage
...
Test / test_etcd_fail (push) Successful in 1m34s
Details
Test / test_interrupted_rebalance (push) Successful in 1m52s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m4s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m13s
Details
Test / test_failure_domain (push) Successful in 12s
Details
Test / test_snapshot (push) Successful in 19s
Details
Test / test_snapshot_ec (push) Successful in 20s
Details
Test / test_minsize_1 (push) Successful in 13s
Details
Test / test_move_reappear (push) Successful in 19s
Details
Test / test_rm (push) Successful in 11s
Details
Test / test_snapshot_chain (push) Successful in 1m56s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m36s
Details
Test / test_snapshot_down (push) Successful in 22s
Details
Test / test_snapshot_down_ec (push) Successful in 23s
Details
Test / test_splitbrain (push) Successful in 15s
Details
Test / test_rebalance_verify (push) Successful in 3m6s
Details
Test / test_rebalance_verify_imm (push) Successful in 3m9s
Details
Test / test_rebalance_verify_ec (push) Successful in 3m27s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m30s
Details
Test / test_write (push) Successful in 35s
Details
Test / test_write_xor (push) Successful in 41s
Details
Test / test_write_no_same (push) Successful in 14s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m48s
Details
Test / test_heal_ec (push) Successful in 4m15s
Details
Test / test_scrub (push) Successful in 33s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 31s
Details
Test / test_scrub_xor (push) Successful in 28s
Details
Test / test_scrub_pg_size_3 (push) Successful in 53s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 35s
Details
Test / test_scrub_ec (push) Successful in 28s
Details
With default --snapshot-count 100000 and GOGC=100 it easily reaches 6.6 GB
even when we only store 1-2 MB of data in it
2023-07-06 00:46:26 +03:00
Vitaliy Filippov
d007a374f2
Delete extra /pool/stats/ keys for non-existing pools
Test / test_interrupted_rebalance (push) Failing after 10m5s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m29s
Details
Test / test_interrupted_rebalance_ec (push) Failing after 10m7s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m32s
Details
Test / test_failure_domain (push) Successful in 8s
Details
Test / test_snapshot (push) Successful in 19s
Details
Test / test_snapshot_ec (push) Successful in 19s
Details
Test / test_minsize_1 (push) Successful in 12s
Details
Test / test_move_reappear (push) Successful in 17s
Details
Test / test_rm (push) Successful in 11s
Details
Test / test_snapshot_chain (push) Successful in 1m1s
Details
Test / test_snapshot_chain_ec (push) Successful in 1m25s
Details
Test / test_snapshot_down (push) Successful in 20s
Details
Test / test_snapshot_down_ec (push) Successful in 19s
Details
Test / test_splitbrain (push) Successful in 12s
Details
Test / test_rebalance_verify (push) Successful in 3m1s
Details
Test / test_rebalance_verify_imm (push) Successful in 4m11s
Details
Test / test_rebalance_verify_ec (push) Successful in 4m19s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m51s
Details
Test / test_write (push) Successful in 31s
Details
Test / test_write_xor (push) Successful in 41s
Details
Test / test_write_no_same (push) Successful in 12s
Details
Test / test_heal_pg_size_2 (push) Successful in 4m10s
Details
Test / test_heal_ec (push) Failing after 10m11s
Details
Test / test_scrub (push) Successful in 43s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 36s
Details
Test / test_scrub_xor (push) Successful in 37s
Details
Test / test_scrub_pg_size_3 (push) Successful in 48s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 23s
Details
Test / test_scrub_ec (push) Successful in 22s
Details
2023-07-06 00:40:13 +03:00
Vitaliy Filippov
45c0694853
Clear etcd_local addresses on reload and also skip duplicates
2023-07-06 00:39:39 +03:00
Vitaliy Filippov