737 Commits (kv-update)

Author	SHA1	Message	Date
Vitaliy Filippov	a9ef9a86c0	Add update() API to kv_db	2024-01-06 17:14:42 +03:00
Vitaliy Filippov	25832cb7e4	Fix eviction when random_pos selects the end	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	e6326c6539	Implement min/max list_count to make listings during performance test reasonable	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	e32f382815	Fix and improve parallel allocation ... - Do not try to allocate more DB blocks in an inode block until it's "confirmed" and "locked" by the first write - Do not recheck for new zero DB blocks on first write into an inode block - a CAS failure means someone else is already writing into it - Throw new allocation blocks away regardless of whether the known_version is 0 on a CAS failure	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	fb23d94000	Implement key_prefix for K/V stress test	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	ee462c2dad	More fixes ... - do not overwrite a block with older version if known version is newer (read may start before update and end after update) - invalidated block versions can't be remembered and trusted - right boundary for split blocks is right_half when diving down, not key_lt - restart update also when block is "invalidated", not just on version mismatch - copy callback in listings to avoid closure destruction bugs too	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	16e4c767f1	Add logging and one more assert	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	9e2b677499	Make get_block() wait for updating when unrelated block is found along the path	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	fd57096d2d	Fix a race condition where changed blocks were parsed over existing cached blocks and getting a mix of data	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	e5ae907256	Simplify code by removing an unneeded "optimisation"	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	64fd6f1c56	Add kv_log_level, print warnings on level 1, trace ops on level 10	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	6e76e09d16	Fix duplicate keys in listings on parallel updates -- do not rewind key "iterator position"	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	0964aeebd2	Implement key suffix to avoid collisions of multiple test workers	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	facff20ca1	Do not complain on empty first block	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	16e09745f0	Add JSON output for stress-tester	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	442f44a64f	Print total stats	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	c67e3d56cb	Do not send more than op_count operations (fix segfault on finish)	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	de41e46335	Add some more resiliency to serialize()	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	bf9a279ff9	Invalidate blocks being updated too	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	b7a41e6394	Change new block allocation method: make each writer choose multiple empty PG blocks and place blocks in them	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	4175cb3720	Remove blocks from cache on unsuccessful updates	2024-01-02 13:24:15 +03:00
Vitaliy Filippov	3a4b71b0cd	Allow to track multiple updates per block (it should never happen though)	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	34969c5919	Do not call stop_updating after failed write_new_block and after clear_block (both delete the item)	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	02a8df6586	Track versions of parent blocks and recheck if changed during update	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	86c6482cf3	Fix resume_split condition (key_lt can also be "")	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	4dd68c543c	Experiment: transform offsets for better sharding	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	10ad96c56c	More post-stress-test fixes ... - Prevent _split types of new blocks - Stop updating new blocks only after the whole update, otherwise pointers may become invalid - Use recheck_none for updates initially - Use UINT64_MAX as initial block version when postponing ops, otherwise the check fails when the block is initially empty. This for example leads to writing both leaf items & block pointers (which is incorrect) into the root block when starting stress-test with --parallelism 32 - Fix -EINTR comparison	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	6e451117ce	Print operation statistics	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	85f35bdf30	K/V fixes after stress-test :-) ... - track block versions correctly - per inode block (128kb) instead of tree block (4kb) - prevent multiple parallel CAS writes of the same inode block - add logging for EILSEQ which means invalid data in the tree - fix get_block updated flag which was true for blocks already in cache and was leading to infinite loops on "unrelated block" errors - apply changes to blocks in cache only after successful writes (using "virtual changes") - do not replace cached block with an older version from disk - recheck "unrelated blocks" (read/update collisions) until data stops changing - track tree path correctly - do not treat split block as parent of its right half - correctly move blocks when finding new empty place on disk - restart updates from the beginning when one of blocks is changed by a parallel update - fix delete using SET opcode and setting key to the empty value instead - prevent changing the same key more than 1 time in parallel - fix listing verification - resume continue_updates in update_find (required because it uses continue_update itself) - add allow_old_cached parameter to get()	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	09adaf62fd	Implement K/V DB stress tester	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	af93f8323c	Evict blocks based on memory limit & block usage	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	3d29c76ff4	Track blocks per level	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	19275379c1	Track block level	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	96ad3c7c50	Experimental B-Tree Vitastor embedded K/V database implementation!	2024-01-02 13:24:14 +03:00
Vitaliy Filippov	2f228fa96a	Only treat data partitions as existing OSDs in vitastor-disk prepare	2023-12-31 11:46:47 +03:00
Vitaliy Filippov	a6ab54b1ba	Do not allow negative util_low/high	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	99ee8596ea	Rename min/max_util to util_low/high	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	c4928e6ecd	Protect from try_send completing the operation immediately ... Fixes a possible use-after-free in case of continue_ops() calling try_send(), then connect_peer() -> set_timer() -> trigger_nearest() -> handle_op_part() -> continue_ops() again	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	8b8c1179a7	Use a separate used_blocks counter for free space stats to hide possibly delayed on-flush deallocation	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	d5a6fa6dd7	Fix possible crash on print_slow when bs_op is NULL	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	1edf86ed26	Aggregate recovery delay using simple mean over last 10 observations (EWMA is shit)	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	5ca7cde612	Experiment/WIP: Try to track "secondary" recovery ops separately	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	751935ddd8	WIP Auto-tune recovery speed	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	d84dee7098	Track recovery op latencies + refactor into a structure	2023-12-31 01:23:17 +03:00
Vitaliy Filippov	f72f14e6a7	Clear old PG states, history, and OSD states on etcd state reload ... Also add protection from etcd watcher messages being split into multiple websocket messages - I'm not sure if etcd actually does that, but it's better to have extra protection anyway. Also check that all etcd watchers are started in the keepalive routine, otherwise it sometimes tries to revive etcd watchers starting with revision=1 which obviously always fails because this revision is nearly always compacted. All these changes should fix an old rarely reproduced bug where SOMETIMES OSDs didn't react to PG config changes which was leading to offline pools on node reboot. It happened on the full reload of state from etcd.	2023-12-24 02:02:13 +03:00
Vitaliy Filippov	178bb0e701	Prevent re-entry into timerfd set_nearest	2023-12-22 02:32:40 +03:00
Vitaliy Filippov	7239cfb91a	Parse log_level in cluster_client	2023-12-20 02:21:23 +03:00
Vitaliy Filippov	7cea642f4a	Fix vitastor-nbd image existence check not working because of non-zeroed inode_watch fields	2023-12-19 01:11:37 +03:00
Vitaliy Filippov	dc615403d9	Do not warn on EPIPE in client unless log_level is raised explicitly	2023-12-17 13:42:26 +03:00
Vitaliy Filippov	1a704e06ab	Allow multiple interfaces with the same IP address, for "simple routed" full mesh network	2023-12-17 13:25:56 +03:00