662 Commits (zerocopy-tcp-send)

Author	SHA1	Message	Date
Vitaliy Filippov	64bbf121b6	Experiment: zero-copy TCP send	2023-11-04 01:34:18 +03:00
Vitaliy Filippov	30dff8893f	Fix ISA-L version EC recovery with first missing data chunk not being read ... (Yes, all EC n + k with k >= 2 users should upgrade as soon as possible)	2023-11-04 01:34:18 +03:00
Vitaliy Filippov	becf14a705	Add a test for EC with multiple missing data chunks, but without recovery of first of them	2023-11-04 01:34:18 +03:00
Vitaliy Filippov	7c054c6f10	Add "id" to df --json output	2023-11-01 12:46:16 +03:00
Vitaliy Filippov	b4740acf62	Fix operations paused for 0.5-1 second when it happens that io_uring submit is not triggered	2023-10-28 13:18:21 +03:00
Vitaliy Filippov	8222e3c77d	Release 1.1.0 ... New features: - Implement [client writeback cache](docs/config/client.en.md#client_enable_writeback) - Add the third I/O mode: [O_DIRECT|O_SYNC](docs/config/osd.en.md#data_io) (good for Optane) - Reduce load on etcd by splitting OSD lease and statistics reporting intervals: [etcd_stats_interval](docs/config/osd.en.md#etcd_stats_interval) (default 30 sec) - Make MON automatically filter OSDs by layout (block_size/immediate_commit/bitmap_granularity) to prevent "refusing to start PGs of this pool" errors on misconfiguration - Support running fio benchmarks on systems without io_uring - Make QEMU driver compatible with QEMU 8.1 - Document usage of [vhost-user-blk](docs/usage/qemu.en.md#vhost-user-blk) Bug fixes: - Fix resizing disks in QEMU driver (for example, in Proxmox) - Fix "unexpected result" in Proxmox driver by making CLI flush output on exit - Remove unneeded block_size mismatch warnings on pools without matching PGs - Fix possible segfault in vitastor-cli ls -l (usually with deleted pools) - Fix QEMU driver compatibility with systems without io_uring - Fix monitor eating 100% CPU when etcd is down (caused by infinite retries) - Fix potential incorrect write processing with snapshots (not caught in tests but could probably lead to client hangs) - Fix buffer insertion in cluster_client (not caught in tests but could probably lead to incorrect writes in rare cases) - Fix rare OSD crash during sync operation processing - Fix a reenterability issue in cluster_client not reproducible in QEMU/fio, but reproducible with the currently developed K/V database implementation - Fix deletion of the first modified object - OSDs could crash if you modified the same object a lot of times, then deleted it, and then modified it again - Fix the fio_sec_osd test tool	2023-10-28 00:33:06 +03:00
Vitaliy Filippov	be7e76f849	Split etcd_stats_interval out of etcd_report_interval	2023-10-27 01:26:26 +03:00
Vitaliy Filippov	294a754c9e	Allow write-back by default in NBD & NFS	2023-10-27 01:26:26 +03:00
Vitaliy Filippov	8bfea6e7de	Support vitastor_c_create_epoll() in fio driver	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	bac9e34836	Allow to create vitastor_c with plain epoll without uring :-)	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	8aa4d492c1	Allow to use epoll_manager without ringloop	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	9336ee5476	Correctly free manual "small vector" in cluster_client %-)	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	ad30b11519	Add the missing ringloop creation check to vitastor_c_create_uring_json()	2023-10-26 18:07:23 +03:00
Vitaliy Filippov	a061246997	Do not attempt to initialize QEMU driver via vitastor_c_create_qemu_uring() ... It doesn't add any compatibility because vitastor_c_uring_register_eventfd() is added in the same VITASTOR_C_API_VERSION 2.	2023-10-26 17:46:19 +03:00
Vitaliy Filippov	5066e35a49	Fix write-over-delete failing for the very first entry in dirty_db	2023-10-21 17:00:14 +03:00
Vitaliy Filippov	93dc31f3fc	Fix possible segfault in vitastor-cli ls -l	2023-10-18 11:11:41 +03:00
Vitaliy Filippov	f245b56176	Fix another possible reenterability issue in cluster_client ... Non-reproducible in QEMU/FIO, only caught during K/V DB debugging	2023-10-08 11:02:53 +03:00
Vitaliy Filippov	38db53f5ee	Implement client writeback cache ... - Disabled by default, enable with client_enable_writeback=true - Even then only enabled in FIO when -direct is disabled and in QEMU when block device cache is enabled in settings - Can also be enabled in other clients like vitastor-cli using parameter client_writeback_allowed=true, but not recommended	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	cd543a90bc	Prevent stack overflows in cli_merge with CAS and writeback cache	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	f600cc07b0	Autosync in blockstore every autosync_writes, too	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	6a8e530e6b	Add FIXME to timerfd_manager	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	5cadb170b9	Fix possible OSD crash during sync due to missing min_flushed_journal_sector reset	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	e72d4ed1d4	Remove unused bs_sync fields	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	ff479a102d	Make MON filter OSDs by block layout to prevent "refusing to start PGs of this pool" errors on misconfiguration	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	27d0d5b06a	Reads do not have to wait for buffer flushes anymore	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	33950c1ec8	Fix fio_sec_osd attr_len	2023-09-16 17:49:10 +03:00
Vitaliy Filippov	cc0fdc6253	Remove erroneous block_size mismatch warnings on pools without matching PGs	2023-09-08 23:19:04 +03:00
Vitaliy Filippov	79ecd59b10	Flush STDOUT and STDERR before exiting from cli to fix Proxmox "Unexpected result"	2023-09-07 17:30:26 +03:00
Vitaliy Filippov	b7d398be5b	Fix sscanf validation usage (field count instead of null_byte == 0)	2023-09-07 02:34:35 +03:00
Vitaliy Filippov	85e9f67d9d	Add supported_truncate_flags	2023-09-06 17:37:52 +03:00
Vitaliy Filippov	79c6d6f323	Make QEMU driver compatible with QEMU 8.1	2023-08-24 02:23:55 +03:00
Vitaliy Filippov	ae760dbc1d	Fix co_truncate size division by BDRV_SECTOR_SIZE	2023-08-24 01:55:35 +03:00
Vitaliy Filippov	65487da4b1	Do not include msgr_rdma.h into messenger.h	2023-08-24 01:55:35 +03:00
Vitaliy Filippov	7862282938	Extract validation to check_rw(), remove duplicate code with OP_SYNC	2023-08-13 23:49:52 +03:00
Vitaliy Filippov	30ce2bd951	Fix buffer insert in cluster_client	2023-08-12 11:08:50 +03:00
Vitaliy Filippov	b1a0afd10a	Aggregate buffer flushes	2023-08-11 11:26:13 +03:00
Vitaliy Filippov	85b6134910	Return dirty buffers on read in client ... Required at least to return buffers when they need to be replayed, but until they are actually replayed	2023-08-09 00:57:08 +03:00
Vitaliy Filippov	b1b07a393d	Fix incorrect marking op parts as done with snapshots (could probably lead to client hangs)	2023-08-09 00:57:08 +03:00
Vitaliy Filippov	7333022adf	Add a third I/O mode: O_DIRECT|O_SYNC, change parameters to data_io/meta_io/journal_io	2023-08-09 00:57:08 +03:00
Vitaliy Filippov	6acf562e01	Release 1.0.0 ... New features: - Data and metadata checksums! - Metadata checksums are always used with new disk format - Data checksums can be turned on with --data_csum_type crc32c for new OSDs - Checksum block size can be configured - inmemory_metadata now also affects keeping checksums in memory - Linux page cache I/O caching support which can be enabled separately for data, metadata (including checksums) and journal (O_SYNC instead of O_DIRECT) - Details [here](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/config/layout-osd.en.md#data_csum_type) - Backwards compatibility is preserved, you can use new OSDs with old disks Release also includes bug fixes from [0.9.6](https://git.yourcmc.ru/vitalif/vitastor/releases/tag/v0.9.6). 0.9.6 is moved to "-oldstable" repositories and will be available for some additional time.	2023-07-29 18:57:19 +03:00
Vitaliy Filippov	564df2eb5d	Support using buffered I/O with O_SYNC instead of direct I/O	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	1a4ceb420d	Track used blocks, not object versions	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	21b5124a4b	Document data_csum_type and csum_block_size parameters	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	4181add1f4	Remove creepy "metadata copying" during overwrite ... Instead of it, just do not verify checksums of currently mutated objects. When clean data modification during flush runs in parallel to a read request, that request may read a mix of old and new data. It may even read a mix of multiple flushed versions if it lasts too long... And attempts to verify it using temporary copies of metadata make the algorithm too complex and creepy.	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	a8464c19af	Support keeping checksums on disk (not in memory) ... Definitely beneficial for SSD+HDD setups	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	3c8e4c6b72	Use clean_dyn_size for space check	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	8ef4cf89dc	Log more details about checksum mismatch in big_writes	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	7bfb1639ea	Use find_holes() in flusher for unification	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	628e481c32	Fill journal header to know checksum type & size when dumping journal with --all	2023-07-29 12:17:18 +03:00
Vitaliy Filippov	af6f2046fc	Fix journal read checksum verification with inmemory_journal=false	2023-07-29 12:17:18 +03:00