Vitaliy Filippov
c752b68167
Remove "without checksums" from docs :)
2023-07-29 12:19:03 +03:00
Vitaliy Filippov
564df2eb5d
Support using buffered I/O with O_SYNC instead of direct I/O
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
9a427dd70a
Allow to override OSD devices in tests
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
1a4ceb420d
Track used blocks, not object versions
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
21b5124a4b
Document data_csum_type and csum_block_size parameters
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
4181add1f4
Remove creepy "metadata copying" during overwrite
...
Instead of it, just do not verify checksums of currently mutated objects.
When clean data modification during flush runs in parallel to a read request,
that request may read a mix of old and new data. It may even read a mix of
multiple flushed versions if it lasts too long... And attempts to verify it
using temporary copies of metadata make the algorithm too complex and creepy.
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
a8464c19af
Support keeping checksums on disk (not in memory)
...
Definitely beneficial for SSD+HDD setups
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
819cb70cdd
Check for "Checksum mismatch" and "BUG" messages during test_heal
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
3c8e4c6b72
Use clean_dyn_size for space check
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
8ef4cf89dc
Log more details about checksum mismatch in big_writes
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
7bfb1639ea
Use find_holes() in flusher for unification
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
628e481c32
Fill journal header to know checksum type & size when dumping journal with --all
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
af6f2046fc
Fix journal read checksum verification with inmemory_journal=false
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
9357e5293e
Call fill_partial_checksum_blocks() correctly in regard to COPY_BUF_CSUM_FILL
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
12851dc07d
Wait for journal reads before checking them in clear_incomplete_csum_block_bits
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
a5753e35a3
Check for checksum mismatch absence in test_heal
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
d6ee1ca17c
Use zero checksum size for zero-length writes
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
71674d00cf
Fix journal data checksum mangling on corrupted block overwrite
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
ddb078d5a7
Check journal entry size when checking block checksums
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
d22d56f90a
Fix journal data checksum verification on start
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
eb1331a079
Add more details to "journal entry data is corrupt" messages
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
c5274f655b
...and partially remove the perversion with bitmap inlining
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
45e07d6294
Sadly we have to refcount dyn_data...
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
a8ee391e05
Fix clean block checksum read
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
de48fa3fd2
Allow to forcibly set meta_format
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
874a766b62
Rename meta_version to meta_format
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
384bd8e28f
Support old metadata format in vitastor-disk dump-meta
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
430994f48a
Fix journal big_write simple reads after checksum changes
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
3d7f838c59
Verify checksums in test_heal in different combinations
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
b909d81f41
Fix bitmap-granular checksums
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
e42975ffd1
Fix wait_journal_count not being zeroed
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
93778324e5
Rewrite and fix find_holes into a more obvious version
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
eeb6727170
Fix missing checksum read offset
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
7fe82c692e
Add a test for checksums
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
92c6e16eba
Fix checksum verification in big_write journal reads
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
213a9ccb4d
Verify checksums during journal reads
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
a166147110
Add backwards compatibility with non-checksum metadata and journal formats
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
7d532880c3
Implement large csum_block_size support (more than 4k) + refactor blockstore_flush
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
0b0405d115
Implement bitmap-granular (4k) metadata & data checksums
2023-07-29 12:17:18 +03:00
Vitaliy Filippov
e651c93a90
Release 0.9.6
...
Test / test_interrupted_rebalance (push) Successful in 1m59s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 3m41s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m29s
Details
Test / test_failure_domain (push) Successful in 50s
Details
Test / test_snapshot (push) Successful in 45s
Details
Test / test_snapshot_ec (push) Successful in 23s
Details
Test / test_minsize_1 (push) Successful in 15s
Details
Test / test_move_reappear (push) Successful in 18s
Details
Test / test_rm (push) Successful in 15s
Details
Test / test_snapshot_chain (push) Successful in 2m23s
Details
Test / test_snapshot_chain_ec (push) Successful in 3m1s
Details
Test / test_snapshot_down (push) Successful in 31s
Details
Test / test_snapshot_down_ec (push) Successful in 32s
Details
Test / test_splitbrain (push) Successful in 19s
Details
Test / test_rebalance_verify (push) Successful in 3m34s
Details
Test / test_rebalance_verify_imm (push) Successful in 3m31s
Details
Test / test_rebalance_verify_ec (push) Successful in 5m14s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m14s
Details
Test / test_write (push) Successful in 44s
Details
Test / test_write_xor (push) Successful in 54s
Details
Test / test_write_no_same (push) Successful in 15s
Details
Test / test_heal_pg_size_2 (push) Successful in 4m38s
Details
Test / test_heal_ec (push) Successful in 3m56s
Details
Test / test_scrub (push) Successful in 36s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 35s
Details
Test / test_scrub_xor (push) Successful in 31s
Details
Test / test_scrub_pg_size_3 (push) Successful in 46s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 25s
Details
Test / test_scrub_ec (push) Successful in 24s
Details
- Fix vitastor-disk partition zeroing (sometimes it was writing garbage instead of zeroes)
- Fix incorrect EC space statistics in `vitastor-cli status`
- Several bug fixes for NFS:
- Add . and .. in NFS directory listings
- Return FILE_SYNC from NFS writes if immediate_commit is enabled
- Return the same "verifier" in NFS COMMIT as in NFS WRITE
- Make parallel NFS extending writes work correctly, without conflicts
- Handle parallel NFS extending writes without imposing extra load on etcd
- Support UTF-8 in vitastor-cli table output
- Also allow "0" and "no" as false for inmemory_metadata and inmemory_journal
- Use HDD defaults for HDD-only in automatic `vitastor-disk prepare` mode
2023-07-29 10:54:00 +03:00
Vitaliy Filippov
988e90be69
Fix vitastor-disk partition zeroing (it was writing random garbage instead of zeroes :D)
Test / test_interrupted_rebalance (push) Successful in 1m59s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m38s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m12s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m18s
Details
Test / test_failure_domain (push) Successful in 8s
Details
Test / test_snapshot (push) Successful in 19s
Details
Test / test_snapshot_ec (push) Successful in 20s
Details
Test / test_minsize_1 (push) Successful in 12s
Details
Test / test_move_reappear (push) Successful in 18s
Details
Test / test_rm (push) Successful in 12s
Details
Test / test_snapshot_chain (push) Successful in 1m3s
Details
Test / test_snapshot_chain_ec (push) Successful in 1m25s
Details
Test / test_snapshot_down (push) Successful in 20s
Details
Test / test_snapshot_down_ec (push) Successful in 20s
Details
Test / test_splitbrain (push) Successful in 12s
Details
Test / test_rebalance_verify (push) Successful in 3m16s
Details
Test / test_rebalance_verify_imm (push) Successful in 3m40s
Details
Test / test_rebalance_verify_ec (push) Successful in 4m17s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m20s
Details
Test / test_write (push) Successful in 40s
Details
Test / test_write_xor (push) Successful in 33s
Details
Test / test_write_no_same (push) Successful in 13s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m54s
Details
Test / test_heal_ec (push) Successful in 4m7s
Details
Test / test_scrub (push) Successful in 58s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 53s
Details
Test / test_scrub_xor (push) Successful in 40s
Details
Test / test_scrub_pg_size_3 (push) Successful in 1m6s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 37s
Details
Test / test_scrub_ec (push) Successful in 39s
Details
2023-07-28 12:29:07 +03:00
Vitaliy Filippov
272a45ad63
Fix modprobe command in docs
2023-07-27 23:57:02 +03:00
Vitaliy Filippov
25a15d24cf
Fix incorrect EC space statistics in `vitastor-cli status`
Test / test_etcd_fail (push) Successful in 1m21s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m9s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m52s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m25s
Details
Test / test_failure_domain (push) Successful in 10s
Details
Test / test_snapshot (push) Successful in 28s
Details
Test / test_snapshot_ec (push) Successful in 30s
Details
Test / test_minsize_1 (push) Successful in 15s
Details
Test / test_move_reappear (push) Successful in 17s
Details
Test / test_rm (push) Successful in 11s
Details
Test / test_snapshot_chain (push) Successful in 2m1s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m41s
Details
Test / test_snapshot_down (push) Successful in 23s
Details
Test / test_snapshot_down_ec (push) Successful in 24s
Details
Test / test_splitbrain (push) Successful in 17s
Details
Test / test_rebalance_verify (push) Successful in 3m9s
Details
Test / test_rebalance_verify_imm (push) Successful in 3m9s
Details
Test / test_rebalance_verify_ec (push) Successful in 3m23s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m38s
Details
Test / test_write (push) Successful in 33s
Details
Test / test_write_xor (push) Successful in 43s
Details
Test / test_write_no_same (push) Successful in 14s
Details
Test / test_heal_pg_size_2 (push) Successful in 4m16s
Details
Test / test_heal_ec (push) Successful in 5m0s
Details
Test / test_scrub (push) Successful in 56s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 41s
Details
Test / test_scrub_xor (push) Successful in 32s
Details
Test / test_scrub_pg_size_3 (push) Successful in 53s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 45s
Details
Test / test_scrub_ec (push) Successful in 40s
Details
2023-07-27 02:26:17 +00:00
Vitaliy Filippov
700e0e9bff
Handle parallel NFS extending writes without imposing extra load on etcd
2023-07-27 02:26:17 +00:00
Vitaliy Filippov
ab0ca7c00f
Return FILE_SYNC from NFS writes if immediate_commit is enabled
2023-07-26 02:09:47 +03:00
Vitaliy Filippov
f153bc950b
Return the same "verifier" in NFS COMMIT as in NFS WRITE
...
This fixes buffered (not O_DIRECT) NFS writes in Linux - previously they were
hanging in an infinite loop because COMMIT didn't return the same verifier as
previous WRITEs, and NFS kernel client was infinitely retrying the same writes.
Also this probably allows for correct NFS failover, at least for the same
buffered writes, because NFS clients repeat all write requests until a COMMIT
confirms them.
2023-07-26 02:09:47 +03:00
Vitaliy Filippov
425ff8818d
Add . and .. in NFS directory listings
...
MC, for example, hangs with infinite listing retries without them
2023-07-26 02:09:47 +03:00
Vitaliy Filippov
9e287a7778
Handle extending writes correctly in NFS proxy
...
Previously, multiple parallel writes extending file size through NFS were
racing with each other and triggering deletions of part of the written data
I.e. if you mounted vitastor-nfs and just copied a file into it in MC then
you could end up with only a part of the file actually written
2023-07-26 02:09:43 +03:00
Vitaliy Filippov
f52f58b9e9
Support UTF-8 in vitastor-cli table output
2023-07-25 01:48:57 +00:00
Vitaliy Filippov
1fe6b0c0e2
Also allow "0" and "no" as false for inmemory_metadata and inmemory_journal
2023-07-25 01:48:57 +00:00
Vitaliy Filippov