Fix eviction when random_pos selects the end

Implement min/max list_count to make listings during performance test reasonable
Fix and improve parallel allocation
2024-01-15 22:24:54 +03:00 · 2024-01-15 22:24:54 +03:00 · 2024-01-15 22:24:54 +03:00 · 2024-01-15 22:24:54 +03:00 · 2024-01-15 22:24:54 +03:00 · 2024-01-15 22:24:54 +03:00
3 changed files with 31 additions and 3 deletions
--- a/CLA-en.md
+++ b/CLA-en.md
@ -38,7 +38,7 @@ in the offer.
        on behalf of third parties, including on behalf of his employer.
 2. Subject of the Agreement. \
-   2.1. Subject of the Agreement shall be the Contributions sent to the Author by Contributors.
+   2.1. Subject of the Agreement shall be the Contributions sent to the Author by Contributors. \
   2.2. The Contributor grants to the Author the right to use Contributions at his own
        discretion and without any necessity to get a prior approval from Contributor or
        any other third party in any way, under a simple (non-exclusive), royalty-free,
@ -86,7 +86,7 @@ in the offer.
        of their provision to the Author. \
   5.2. The Contributor represents and warrants that he legally owns exclusive
        intellectual property rights to the Contributions. \
-   5.3. The Contributor represents and warrants that any further use of \
+   5.3. The Contributor represents and warrants that any further use of
        Contributions by the Author as provided by Contributor under the terms
        of the Agreement does not infringe on intellectual and other rights and
        legitimate interests of third parties. \
--- a/2
+++ b/2
@ -1 +1 @@
-Subproject commit 45e6d1f13196a0824e2089a586c53b9de0283f17
+Subproject commit 8de8b467acbca50cfd8835c20e0e379110f3b32b
--- a/pull_request_template.yml
+++ b/pull_request_template.yml
@ -0,0 +1,28 @@
 name: Pull Request
 about: Submit a pull request
 body:
  - type: textarea
    id: description
    attributes:
      label: Description
      description: Describe your pull request
      placeholder: ""
      value: ""
    validations:
      required: true
  - type: input
    id: author
    attributes:
      label: Contributor Name
      description: Contributor Name or Company Details if the Contributor is a company
      placeholder: ""
    validations:
      required: false
  - type: checkboxes
    id: terms
    attributes:
      label: CLA
      description: By submitting this pull request, I accept [Vitastor CLA](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md)
      options:
        - label: "I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md"
          required: true
Author	SHA1	Message	Date
Vitaliy Filippov	cc20443627	Fix eviction when random_pos selects the end Test / test_move_reappear (push) Successful in 19s Details Test / test_rm (push) Successful in 15s Details Test / test_interrupted_rebalance (push) Successful in 5m57s Details Test / test_snapshot_down (push) Successful in 26s Details Test / test_snapshot_chain (push) Successful in 1m39s Details Test / test_snapshot_down_ec (push) Successful in 25s Details Test / test_splitbrain (push) Successful in 23s Details Test / test_snapshot_chain_ec (push) Successful in 2m45s Details Test / test_rebalance_verify_imm (push) Successful in 2m37s Details Test / test_rebalance_verify (push) Successful in 3m42s Details Test / test_write (push) Successful in 36s Details Test / test_rebalance_verify_ec (push) Successful in 3m57s Details Test / test_write_no_same (push) Successful in 14s Details Test / test_interrupted_rebalance_ec (push) Failing after 10m9s Details Test / test_rebalance_verify_ec_imm (push) Successful in 3m49s Details Test / test_write_xor (push) Failing after 3m11s Details Test / test_heal_pg_size_2 (push) Failing after 3m48s Details Test / test_heal_ec (push) Successful in 4m53s Details Test / test_heal_csum_32k_dmj (push) Successful in 4m25s Details Test / test_heal_csum_32k_dj (push) Successful in 5m36s Details Test / test_heal_csum_32k (push) Successful in 6m0s Details Test / test_scrub (push) Successful in 57s Details Test / test_heal_csum_4k_dj (push) Successful in 5m41s Details Test / test_scrub_zero_osd_2 (push) Successful in 50s Details Test / test_scrub_xor (push) Successful in 43s Details Test / test_heal_csum_4k_dmj (push) Successful in 7m9s Details Test / test_scrub_ec (push) Successful in 53s Details Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 57s Details Test / test_scrub_pg_size_3 (push) Successful in 2m9s Details Test / test_heal_csum_4k (push) Successful in 6m2s Details	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	a0890c58c1	Implement min/max list_count to make listings during performance test reasonable	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	23d3ff1cdf	Fix and improve parallel allocation - Do not try to allocate more DB blocks in an inode block until it's "confirmed" and "locked" by the first write - Do not recheck for new zero DB blocks on first write into an inode block - a CAS failure means someone else is already writing into it - Throw new allocation blocks away regardless of whether the known_version is 0 on a CAS failure	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	fc7e254375	Implement key_prefix for K/V stress test	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	f867c2f873	More fixes - do not overwrite a block with older version if known version is newer (read may start before update and end after update) - invalidated block versions can't be remembered and trusted - right boundary for split blocks is right_half when diving down, not key_lt - restart update also when block is "invalidated", not just on version mismatch - copy callback in listings to avoid closure destruction bugs too	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	ee6d9fcae3	Add logging and one more assert	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	8dfa5fd1be	Make get_block() wait for updating when unrelated block is found along the path	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	e184f31710	Fix a race condition where changed blocks were parsed over existing cached blocks and getting a mix of data	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	fd8d587271	Simplify code by removing an unneeded "optimisation"	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	389f209298	Add kv_log_level, print warnings on level 1, trace ops on level 10	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	fa28828369	Fix duplicate keys in listings on parallel updates -- do not rewind key "iterator position"	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	c91a14c67c	Implement key suffix to avoid collisions of multiple test workers	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	f779d9d3a7	Do not complain on empty first block	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	9361fa54c4	Add JSON output for stress-tester	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	f548683e88	Print total stats	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	6c69a66c96	Do not send more than op_count operations (fix segfault on finish)	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	9f8fbdbecf	Add some more resiliency to serialize()	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	b85ee1c970	Invalidate blocks being updated too	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	53d0722695	Change new block allocation method: make each writer choose multiple empty PG blocks and place blocks in them	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	31669d15e1	Remove blocks from cache on unsuccessful updates	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	6da085eae3	Allow to track multiple updates per block (it should never happen though)	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	73ffd06dbf	Do not call stop_updating after failed write_new_block and after clear_block (both delete the item)	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	bb809529b1	Track versions of parent blocks and recheck if changed during update	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	f5edc0d4d9	Fix resume_split condition (key_lt can also be "")	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	16cb28c52f	Experiment: transform offsets for better sharding	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	25a9d490da	More post-stress-test fixes - Prevent _split types of new blocks - Stop updating new blocks only after the whole update, otherwise pointers may become invalid - Use recheck_none for updates initially - Use UINT64_MAX as initial block version when postponing ops, otherwise the check fails when the block is initially empty. This for example leads to writing both leaf items & block pointers (which is incorrect) into the root block when starting stress-test with --parallelism 32 - Fix -EINTR comparison	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	f2e71af4fe	Print operation statistics	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	ae26df16ef	K/V fixes after stress-test :-) - track block versions correctly - per inode block (128kb) instead of tree block (4kb) - prevent multiple parallel CAS writes of the same inode block - add logging for EILSEQ which means invalid data in the tree - fix get_block updated flag which was true for blocks already in cache and was leading to infinite loops on "unrelated block" errors - apply changes to blocks in cache only after successful writes (using "virtual changes") - do not replace cached block with an older version from disk - recheck "unrelated blocks" (read/update collisions) until data stops changing - track tree path correctly - do not treat split block as parent of its right half - correctly move blocks when finding new empty place on disk - restart updates from the beginning when one of blocks is changed by a parallel update - fix delete using SET opcode and setting key to the empty value instead - prevent changing the same key more than 1 time in parallel - fix listing verification - resume continue_updates in update_find (required because it uses continue_update itself) - add allow_old_cached parameter to get()	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	1a913117e6	Implement K/V DB stress tester	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	b8b8ee5666	Evict blocks based on memory limit & block usage	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	a28f515214	Track blocks per level	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	961bb3b695	Track block level	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	31b9a8aa79	Experimental B-Tree Vitastor embedded K/V database implementation!	2024-01-15 22:24:54 +03:00
Vitaliy Filippov	2cf649eba6	Fix "partly outside array bounds" warnings for GCC 12 in cpp-btree Test / test_move_reappear (push) Successful in 19s Details Test / test_rm (push) Successful in 13s Details Test / test_snapshot_down (push) Successful in 25s Details Test / test_snapshot_down_ec (push) Successful in 25s Details Test / test_splitbrain (push) Successful in 19s Details Test / test_snapshot_chain (push) Successful in 2m13s Details Test / test_interrupted_rebalance (push) Successful in 7m36s Details Test / test_rebalance_verify (push) Successful in 3m35s Details Test / test_write (push) Successful in 1m0s Details Test / test_rebalance_verify_imm (push) Successful in 4m4s Details Test / test_rebalance_verify_ec (push) Successful in 3m13s Details Test / test_rebalance_verify_ec_imm (push) Successful in 2m35s Details Test / test_write_no_same (push) Successful in 14s Details Test / test_heal_pg_size_2 (push) Successful in 4m32s Details Test / test_heal_csum_32k_dmj (push) Successful in 4m29s Details Test / test_heal_ec (push) Successful in 5m47s Details Test / test_heal_csum_32k_dj (push) Successful in 5m47s Details Test / test_heal_csum_4k_dmj (push) Successful in 6m4s Details Test / test_heal_csum_32k (push) Successful in 6m19s Details Test / test_scrub (push) Successful in 56s Details Test / test_scrub_zero_osd_2 (push) Successful in 43s Details Test / test_heal_csum_4k_dj (push) Successful in 6m14s Details Test / test_scrub_xor (push) Successful in 53s Details Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 57s Details Test / test_scrub_ec (push) Successful in 47s Details Test / test_heal_csum_4k (push) Successful in 5m56s Details Test / test_minsize_1 (push) Successful in 14s Details Test / test_scrub_pg_size_3 (push) Successful in 46s Details Test / test_snapshot_chain_ec (push) Successful in 1m40s Details Test / test_write_xor (push) Failing after 3m6s Details	2024-01-15 03:04:33 +03:00
Vitaliy Filippov	5935640a4a	Add CLA PR form	2024-01-14 16:48:24 +03:00
		`@ -1 +1 @@`
			`Subproject commit 45e6d1f13196a0824e2089a586c53b9de0283f17`				`Subproject commit 8de8b467acbca50cfd8835c20e0e379110f3b32b`