Compare commits

..

37 Commits

Author SHA1 Message Date
Vitaliy Filippov 093c355e80 Move KV FS header into a separate file
Test / test_snapshot_chain_ec (push) Has been cancelled Details
Test / test_snapshot_down (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec (push) Has been cancelled Details
Test / test_snapshot_down_ec (push) Has been cancelled Details
Test / test_interrupted_rebalance (push) Has been cancelled Details
Test / test_interrupted_rebalance_imm (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_switch_primary (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec_imm (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
2024-02-03 20:31:08 +03:00
Vitaliy Filippov 8d9e596963 Implement packing small files into shared inodes 2024-02-03 20:31:08 +03:00
Vitaliy Filippov e462209130 Split new NFS proxy implementation into multiple files 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 5020aeb4eb WIP VitastorFS with metadata storage in VitastorKV 2024-02-03 20:31:08 +03:00
Vitaliy Filippov c211209fd6 Fix eviction when random_pos selects the end
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot_chain (push) Has been cancelled Details
Test / test_snapshot_chain_ec (push) Has been cancelled Details
Test / test_snapshot_down (push) Has been cancelled Details
Test / test_snapshot_down_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_switch_primary (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
2024-02-03 20:31:08 +03:00
Vitaliy Filippov fd7f961749 Implement min/max list_count to make listings during performance test reasonable 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 925837f12e Fix and improve parallel allocation
- Do not try to allocate more DB blocks in an inode block until it's "confirmed" and "locked" by the first write
- Do not recheck for new zero DB blocks on first write into an inode block - a CAS failure means someone else is already writing into it
- Throw new allocation blocks away regardless of whether the known_version is 0 on a CAS failure
2024-02-03 20:31:08 +03:00
Vitaliy Filippov 4c183742f1 Implement key_prefix for K/V stress test 2024-02-03 20:31:08 +03:00
Vitaliy Filippov e456b5d621 More fixes
- do not overwrite a block with older version if known version is newer
  (read may start before update and end after update)
- invalidated block versions can't be remembered and trusted
- right boundary for split blocks is right_half when diving down, not key_lt
- restart update also when block is "invalidated", not just on version mismatch
- copy callback in listings to avoid closure destruction bugs too
2024-02-03 20:31:08 +03:00
Vitaliy Filippov 79ca34a238 Add logging and one more assert 2024-02-03 20:31:08 +03:00
Vitaliy Filippov e38dd5b88d Make get_block() wait for updating when unrelated block is found along the path 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 562094fe12 Fix a race condition where changed blocks were parsed over existing cached blocks and getting a mix of data 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 1a4aa895d3 Simplify code by removing an unneeded "optimisation" 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 56303d3fec Add kv_log_level, print warnings on level 1, trace ops on level 10 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 094995fbf4 Fix duplicate keys in listings on parallel updates -- do not rewind key "iterator position" 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 7952b12ebc Implement key suffix to avoid collisions of multiple test workers 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 4d8c3d0b0e Do not complain on empty first block 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 4e5fb7f0ab Add JSON output for stress-tester 2024-02-03 20:31:08 +03:00
Vitaliy Filippov dd228fce7a Print total stats 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 7956eafa87 Do not send more than op_count operations (fix segfault on finish) 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 1ef1a74d45 Add some more resiliency to serialize() 2024-02-03 20:31:08 +03:00
Vitaliy Filippov fd317edfed Invalidate blocks being updated too 2024-02-03 20:31:08 +03:00
Vitaliy Filippov b1bc1d746c Change new block allocation method: make each writer choose multiple empty PG blocks and place blocks in them 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 1fce9a5229 Remove blocks from cache on unsuccessful updates 2024-02-03 20:31:08 +03:00
Vitaliy Filippov cb4192caa8 Allow to track multiple updates per block (it should never happen though) 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 3993971825 Do not call stop_updating after failed write_new_block and after clear_block (both delete the item) 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 87e600bf6f Track versions of parent blocks and recheck if changed during update 2024-02-03 20:31:08 +03:00
Vitaliy Filippov b9f857cf65 Fix resume_split condition (key_lt can also be "") 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 50ca29d627 Experiment: transform offsets for better sharding 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 5f453430df More post-stress-test fixes
- Prevent _split types of new blocks
- Stop updating new blocks only after the whole update, otherwise pointers
  may become invalid
- Use recheck_none for updates initially
- Use UINT64_MAX as initial block version when postponing ops, otherwise the
  check fails when the block is initially empty. This for example leads to
  writing both leaf items & block pointers (which is incorrect) into the root
  block when starting stress-test with --parallelism 32
- Fix -EINTR comparison
2024-02-03 20:31:08 +03:00
Vitaliy Filippov 5ba08ee21d Print operation statistics 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 91a2f90e83 K/V fixes after stress-test :-)
- track block versions correctly - per inode block (128kb) instead of tree block (4kb)
- prevent multiple parallel CAS writes of the same inode block
- add logging for EILSEQ which means invalid data in the tree
- fix get_block updated flag which was true for blocks already in cache and was leading to infinite loops on "unrelated block" errors
- apply changes to blocks in cache only after successful writes (using "virtual changes")
- do not replace cached block with an older version from disk
- recheck "unrelated blocks" (read/update collisions) until data stops changing
- track tree path correctly - do not treat split block as parent of its right half
- correctly move blocks when finding new empty place on disk
- restart updates from the beginning when one of blocks is changed by a parallel update
- fix delete using SET opcode and setting key to the empty value instead
- prevent changing the same key more than 1 time in parallel
- fix listing verification
- resume continue_updates in update_find (required because it uses continue_update itself)
- add allow_old_cached parameter to get()
2024-02-03 20:31:08 +03:00
Vitaliy Filippov 3df8147cf5 Implement K/V DB stress tester 2024-02-03 20:31:08 +03:00
Vitaliy Filippov 0788b31d79 Evict blocks based on memory limit & block usage 2024-02-03 20:31:08 +03:00
Vitaliy Filippov fae200dabd Track blocks per level 2024-02-03 20:31:08 +03:00
Vitaliy Filippov d8833e41da Track block level 2024-02-03 20:31:08 +03:00
Vitaliy Filippov c19c49dd21 Experimental B-Tree Vitastor embedded K/V database implementation! 2024-02-03 20:31:08 +03:00
1 changed files with 1 additions and 1 deletions

View File

@ -129,7 +129,7 @@ bool blockstore_impl_t::enqueue_write(blockstore_op_t *op)
}
bool imm = (op->len < dsk.data_block_size ? (immediate_commit != IMMEDIATE_NONE) : (immediate_commit == IMMEDIATE_ALL));
if (wait_big && !is_del && !deleted && op->len < dsk.data_block_size && !imm ||
!imm && autosync_writes && unsynced_queued_ops >= autosync_writes)
!imm && unsynced_queued_ops >= autosync_writes)
{
// Issue an additional sync so that the previous big write can reach the journal
blockstore_op_t *sync_op = new blockstore_op_t;