1572 Commits (kv-debug)

All Branches

Search

Author	SHA1	Message	Date
Vitaliy Filippov	06f4e0fcce	K/V control prints (for debug only) O:-)	2023-12-01 02:33:04 +03:00
Vitaliy Filippov	f285cfc483	Fix eviction when random_pos selects the end	2023-12-01 01:43:03 +03:00
Vitaliy Filippov	12b50b421d	Implement min/max list_count to make listings during performance test reasonable	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	9f6d09428d	Fix and improve parallel allocation ... - Do not try to allocate more DB blocks in an inode block until it's "confirmed" and "locked" by the first write - Do not recheck for new zero DB blocks on first write into an inode block - a CAS failure means someone else is already writing into it - Throw new allocation blocks away regardless of whether the known_version is 0 on a CAS failure	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	580025cfc9	Implement key_prefix for K/V stress test	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	13e2d3ce7c	More fixes ... - do not overwrite a block with older version if known version is newer (read may start before update and end after update) - invalidated block versions can't be remembered and trusted - right boundary for split blocks is right_half when diving down, not key_lt - restart update also when block is "invalidated", not just on version mismatch - copy callback in listings to avoid closure destruction bugs too	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	c5b00f897a	Add logging and one more assert	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	e847e26912	Make get_block() wait for updating when unrelated block is found along the path	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	3393463466	Fix a race condition where changed blocks were parsed over existing cached blocks and getting a mix of data	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	bd96a6194a	Simplify code by removing an unneeded "optimisation"	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	601fe10c28	Add kv_log_level, print warnings on level 1, trace ops on level 10	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	63dbc9ca85	Fix duplicate keys in listings on parallel updates -- do not rewind key "iterator position"	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	aa0c363c39	Implement key suffix to avoid collisions of multiple test workers	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	ce52c5589e	Do not complain on empty first block	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	aee20ab1ee	Add JSON output for stress-tester	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	bb81992fac	Print total stats	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	a28f401aff	Do not send more than op_count operations (fix segfault on finish)	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	4ac7e096fd	Add some more resiliency to serialize()	2023-12-01 01:17:04 +03:00
Vitaliy Filippov	b6171a4599	Invalidate blocks being updated too	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	28045f230c	Change new block allocation method: make each writer choose multiple empty PG blocks and place blocks in them	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	10e867880f	Remove blocks from cache on unsuccessful updates	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	012462171a	Allow to track multiple updates per block (it should never happen though)	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	904793cdab	Do not call stop_updating after failed write_new_block and after clear_block (both delete the item)	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	45c01db2de	Track versions of parent blocks and recheck if changed during update	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	8c9206cecd	Fix resume_split condition (key_lt can also be "")	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	e8c46ededa	Experiment: transform offsets for better sharding	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	e9b321a0e0	More post-stress-test fixes ... - Prevent _split types of new blocks - Stop updating new blocks only after the whole update, otherwise pointers may become invalid - Use recheck_none for updates initially - Use UINT64_MAX as initial block version when postponing ops, otherwise the check fails when the block is initially empty. This for example leads to writing both leaf items & block pointers (which is incorrect) into the root block when starting stress-test with --parallelism 32 - Fix -EINTR comparison	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	09a77991ae	Print operation statistics	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	29d8c9b6f3	K/V fixes after stress-test :-) ... - track block versions correctly - per inode block (128kb) instead of tree block (4kb) - prevent multiple parallel CAS writes of the same inode block - add logging for EILSEQ which means invalid data in the tree - fix get_block updated flag which was true for blocks already in cache and was leading to infinite loops on "unrelated block" errors - apply changes to blocks in cache only after successful writes (using "virtual changes") - do not replace cached block with an older version from disk - recheck "unrelated blocks" (read/update collisions) until data stops changing - track tree path correctly - do not treat split block as parent of its right half - correctly move blocks when finding new empty place on disk - restart updates from the beginning when one of blocks is changed by a parallel update - fix delete using SET opcode and setting key to the empty value instead - prevent changing the same key more than 1 time in parallel - fix listing verification - resume continue_updates in update_find (required because it uses continue_update itself) - add allow_old_cached parameter to get()	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	20321aaaef	Implement K/V DB stress tester	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	987b005356	Evict blocks based on memory limit & block usage	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	41754b748b	Track blocks per level	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	31913256f3	Track block level	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	0ee36baed7	Experimental B-Tree Vitastor embedded K/V database implementation!	2023-12-01 01:17:03 +03:00
Vitaliy Filippov	19e2d9d6fa	Fix crash on unknown long argument to vitastor-disk	2023-12-01 00:55:51 +03:00
Vitaliy Filippov	bfc7e61909	Add more notes + performance comparison about VDUSE	2023-11-25 02:25:56 +03:00
Vitaliy Filippov	7da4868b37	Fix monitor statistics aggregation in case of empty /osd/stats keys	2023-11-24 01:05:21 +03:00
Vitaliy Filippov	b5c020ce0b	Use io_uring SQ size for ringloop capacity - otherwise get_sqe could return NULL when space_left() was > 0 under load ... Raise default io_uring size to 1024 for the same effective capacity as previously	2023-11-20 03:04:06 +03:00
Vitaliy Filippov	6b33ae973d	%d -> %lu	2023-11-20 03:02:26 +03:00
Vitaliy Filippov	cf36445359	Reserve journal space for stabilize requests dynamically to prevent stalls	2023-11-20 03:01:57 +03:00
Vitaliy Filippov	3fd873d263	Add -fno-omit-frame-pointer by default	2023-11-20 02:59:54 +03:00
Vitaliy Filippov	a00e8ae9ed	Fix mismatch journal pos format in vitastor-disk	2023-11-19 15:19:54 +03:00
Vitaliy Filippov	75674545dc	Limit the number of printed object versions in slow op dump (otherwise it may overflow the fixed buffer)	2023-11-13 01:10:28 +03:00
Vitaliy Filippov	225eb2fe3d	Support RDMA without ODP by stupidly copying memory. Disable ODP by default ... ODP is slower than regular RDMA even with memory copy overhead Example numbers: - 3950000 random read iops without ODP vs 240000 iops with ODP - 1447000 random write iops without ODP vs 101000 iops with ODP Reference: https://tkygtr6.github.io/pub/ISPASS21_slides.pdf	2023-11-12 15:03:47 +03:00
Vitaliy Filippov	7e82573ed0	Fix RDMA connection leak which was preventing stable functioning of RDMA :)	2023-11-11 23:40:47 +03:00
Vitaliy Filippov	12a6bed2d5	Return the new accidentally rolled back json11 commit ("allow trailing comma")	2023-11-07 15:49:23 +03:00
Vitaliy Filippov	5524dbdab7	Release 1.2.0 ... New features: - Implement CSI volume expansion - Implement CSI volume snapshots - CSI driver now requires Kubernetes >= 1.20 Bug fixes: - Important bug fix for EC: fix EC n+k, k>=2 read recovery in ISA-L version returning incorrect data when reading at least the second chunk out of multiple missing chunks without reading the first one. All users of EC n+k, k>=2 should upgrade as soon as possible, and upgrade should be conducted with downtime: first stop all clients (VMs/containers), then all OSDs, then upgrade and restart everything. - Fix unstable statistics aggregation in monitor (affecting vitastor-cli status and df) - Make udev not wait for OSDs to start during boot - Do not report negative numbers of offline PGs in vitastor-cli status when changing PG count - Report both old and new PG counts in vitastor-cli df when changing it - Fix OSDs sometimes not starting with "The code only supports journal versions 1 and 2, but it is 2 on disk" error after upgrading from pre-1.0 versions and letting OSDs run for some time - Fix monitors sometimes returning old PG count back after OSD configuration changes - Make monitor PG changes more stable and timeout errors less probable	2023-11-05 01:48:57 +03:00
Vitaliy Filippov	cd3dec06ac	Remove spaces from old->new PG count in df	2023-11-05 01:45:45 +03:00
Vitaliy Filippov	371d79e059	Document vitastor-csi features	2023-11-05 01:05:26 +03:00
Vitaliy Filippov	0e888e6c60	Prevent spamming etcd with last_clean_pgs update requests	2023-11-05 00:12:00 +03:00