Compare commits

..

33 Commits

Author SHA1 Message Date
Vitaliy Filippov 37915d8315 Fix eviction when random_pos selects the end
Test / test_rm (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_interrupted_rebalance (push) Successful in 4m51s Details
Test / test_snapshot_chain (push) Successful in 1m27s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_snapshot_down_ec (push) Successful in 26s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_snapshot_chain_ec (push) Successful in 2m26s Details
Test / test_rebalance_verify_imm (push) Successful in 2m13s Details
Test / test_rebalance_verify_ec (push) Successful in 3m1s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_xor (push) Successful in 36s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m58s Details
Test / test_interrupted_rebalance_ec (push) Failing after 10m10s Details
Test / test_rebalance_verify (push) Successful in 7m26s Details
Test / test_heal_pg_size_2 (push) Failing after 4m17s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m45s Details
Test / test_heal_ec (push) Successful in 5m35s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m19s Details
Test / test_heal_csum_32k (push) Successful in 5m53s Details
Test / test_scrub (push) Successful in 56s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m10s Details
Test / test_scrub_zero_osd_2 (push) Successful in 50s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m47s Details
Test / test_scrub_xor (push) Successful in 38s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m7s Details
Test / test_scrub_ec (push) Successful in 47s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m49s Details
Test / test_heal_csum_4k (push) Successful in 6m18s Details
2024-01-13 01:30:29 +03:00
Vitaliy Filippov 496d6122d0 Implement min/max list_count to make listings during performance test reasonable 2024-01-13 01:30:29 +03:00
Vitaliy Filippov ec5b761903 Fix and improve parallel allocation
- Do not try to allocate more DB blocks in an inode block until it's "confirmed" and "locked" by the first write
- Do not recheck for new zero DB blocks on first write into an inode block - a CAS failure means someone else is already writing into it
- Throw new allocation blocks away regardless of whether the known_version is 0 on a CAS failure
2024-01-13 01:30:29 +03:00
Vitaliy Filippov 2d7069abdb Implement key_prefix for K/V stress test 2024-01-13 01:30:29 +03:00
Vitaliy Filippov b032cb767a More fixes
- do not overwrite a block with older version if known version is newer
  (read may start before update and end after update)
- invalidated block versions can't be remembered and trusted
- right boundary for split blocks is right_half when diving down, not key_lt
- restart update also when block is "invalidated", not just on version mismatch
- copy callback in listings to avoid closure destruction bugs too
2024-01-13 01:30:29 +03:00
Vitaliy Filippov 6203b12203 Add logging and one more assert 2024-01-13 01:30:29 +03:00
Vitaliy Filippov c51644727b Make get_block() wait for updating when unrelated block is found along the path 2024-01-13 01:30:29 +03:00
Vitaliy Filippov af651aef24 Fix a race condition where changed blocks were parsed over existing cached blocks and getting a mix of data 2024-01-13 01:30:29 +03:00
Vitaliy Filippov b4962e636d Simplify code by removing an unneeded "optimisation" 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 2aaa7f13c4 Add kv_log_level, print warnings on level 1, trace ops on level 10 2024-01-13 01:30:29 +03:00
Vitaliy Filippov ba3eb5c0f5 Fix duplicate keys in listings on parallel updates -- do not rewind key "iterator position" 2024-01-13 01:30:29 +03:00
Vitaliy Filippov cac938e5b4 Implement key suffix to avoid collisions of multiple test workers 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 6771f766af Do not complain on empty first block 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 9d354ae4e5 Add JSON output for stress-tester 2024-01-13 01:30:29 +03:00
Vitaliy Filippov a6067124fc Print total stats 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 58b04953e3 Do not send more than op_count operations (fix segfault on finish) 2024-01-13 01:30:29 +03:00
Vitaliy Filippov da4f1def94 Add some more resiliency to serialize() 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 14a6ddcf3f Invalidate blocks being updated too 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 5e61f34f1e Change new block allocation method: make each writer choose multiple empty PG blocks and place blocks in them 2024-01-13 01:30:29 +03:00
Vitaliy Filippov d362602c17 Remove blocks from cache on unsuccessful updates 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 4b285437fe Allow to track multiple updates per block (it should never happen though) 2024-01-13 01:30:29 +03:00
Vitaliy Filippov b9d21348b3 Do not call stop_updating after failed write_new_block and after clear_block (both delete the item) 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 6865ffa9e7 Track versions of parent blocks and recheck if changed during update 2024-01-13 01:30:29 +03:00
Vitaliy Filippov e5cc6cb30e Fix resume_split condition (key_lt can also be "") 2024-01-13 01:30:29 +03:00
Vitaliy Filippov ca2a8fe464 Experiment: transform offsets for better sharding 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 1d2a6c20b4 More post-stress-test fixes
- Prevent _split types of new blocks
- Stop updating new blocks only after the whole update, otherwise pointers
  may become invalid
- Use recheck_none for updates initially
- Use UINT64_MAX as initial block version when postponing ops, otherwise the
  check fails when the block is initially empty. This for example leads to
  writing both leaf items & block pointers (which is incorrect) into the root
  block when starting stress-test with --parallelism 32
- Fix -EINTR comparison
2024-01-13 01:30:29 +03:00
Vitaliy Filippov c50b256cef Print operation statistics 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 1db4d8668d K/V fixes after stress-test :-)
- track block versions correctly - per inode block (128kb) instead of tree block (4kb)
- prevent multiple parallel CAS writes of the same inode block
- add logging for EILSEQ which means invalid data in the tree
- fix get_block updated flag which was true for blocks already in cache and was leading to infinite loops on "unrelated block" errors
- apply changes to blocks in cache only after successful writes (using "virtual changes")
- do not replace cached block with an older version from disk
- recheck "unrelated blocks" (read/update collisions) until data stops changing
- track tree path correctly - do not treat split block as parent of its right half
- correctly move blocks when finding new empty place on disk
- restart updates from the beginning when one of blocks is changed by a parallel update
- fix delete using SET opcode and setting key to the empty value instead
- prevent changing the same key more than 1 time in parallel
- fix listing verification
- resume continue_updates in update_find (required because it uses continue_update itself)
- add allow_old_cached parameter to get()
2024-01-13 01:30:29 +03:00
Vitaliy Filippov 447569cae6 Implement K/V DB stress tester 2024-01-13 01:30:29 +03:00
Vitaliy Filippov 811d6c71f8 Evict blocks based on memory limit & block usage 2024-01-13 01:30:29 +03:00
Vitaliy Filippov fca151336f Track blocks per level 2024-01-13 01:30:28 +03:00
Vitaliy Filippov 0ff803138a Track block level 2024-01-13 01:30:28 +03:00
Vitaliy Filippov 938991b102 Experimental B-Tree Vitastor embedded K/V database implementation! 2024-01-13 01:30:28 +03:00
3 changed files with 3 additions and 31 deletions

View File

@ -38,7 +38,7 @@ in the offer.
on behalf of third parties, including on behalf of his employer.
2. Subject of the Agreement. \
2.1. Subject of the Agreement shall be the Contributions sent to the Author by Contributors. \
2.1. Subject of the Agreement shall be the Contributions sent to the Author by Contributors.
2.2. The Contributor grants to the Author the right to use Contributions at his own
discretion and without any necessity to get a prior approval from Contributor or
any other third party in any way, under a simple (non-exclusive), royalty-free,
@ -86,7 +86,7 @@ in the offer.
of their provision to the Author. \
5.2. The Contributor represents and warrants that he legally owns exclusive
intellectual property rights to the Contributions. \
5.3. The Contributor represents and warrants that any further use of
5.3. The Contributor represents and warrants that any further use of \
Contributions by the Author as provided by Contributor under the terms
of the Agreement does not infringe on intellectual and other rights and
legitimate interests of third parties. \

@ -1 +1 @@
Subproject commit 8de8b467acbca50cfd8835c20e0e379110f3b32b
Subproject commit 45e6d1f13196a0824e2089a586c53b9de0283f17

View File

@ -1,28 +0,0 @@
name: Pull Request
about: Submit a pull request
body:
- type: textarea
id: description
attributes:
label: Description
description: Describe your pull request
placeholder: ""
value: ""
validations:
required: true
- type: input
id: author
attributes:
label: Contributor Name
description: Contributor Name or Company Details if the Contributor is a company
placeholder: ""
validations:
required: false
- type: checkboxes
id: terms
attributes:
label: CLA
description: By submitting this pull request, I accept [Vitastor CLA](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md)
options:
- label: "I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md"
required: true