688 Commits (1a704e06ab5f0c560942236ab2aeb2366607decf)

Author	SHA1	Message	Date
Vitaliy Filippov	1a704e06ab	Allow multiple interfaces with the same IP address, for "simple routed" full mesh network	2023-12-17 13:25:56 +03:00
Vitaliy Filippov	575475de71	Do not ignore loopback addresses for OSD network (to make ECMP setups with frr possible)	2023-12-17 11:55:13 +03:00
Vitaliy Filippov	aca2bef15f	Add vitastor-disk update-sb command	2023-12-14 01:11:42 +03:00
Vitaliy Filippov	691ebd991a	Move 2 last log printfs to stderr from stdout in etcd_state_client	2023-12-08 00:01:52 +03:00
Vitaliy Filippov	6d5df908a3	Fix possible out of bounds when checking invalid journal entries	2023-12-08 00:01:07 +03:00
Vitaliy Filippov	a1c7cc3d8d	Release 1.3.1 ... Hotfix to 1.3.0 - new "journal space reservation" had a bug which caused OSDs to crash with EC and without immediate_commit.	2023-12-04 18:35:09 +03:00
Vitaliy Filippov	a5e3dfbc5a	Oops, 1.3.0 needs a hotfix	2023-12-04 13:45:54 +03:00
Vitaliy Filippov	7972502eaf	Release 1.3.0 ... New features: - RDMA without ODP - much faster and all cards are now supported, not just Mellanox - VDUSE in CSI - faster, more stable and can even recover after CSI pod restart! - Reserve journal space for stabilize requests dynamically to prevent stalls under load with EC - Raise default NBD timeout from 30 to 300 seconds and allow to take it from /etc/vitastor/vitastor.conf - Remove explicit etcdUrl/etcdPrefix K8S storage class parameter support to prevent etcd migration issues for volumes created with these parameters - Support QEMU 8.1 and pve-qemu 8.1 Bug fixes: - Fix RDMA connection (and thus memory) leak - Fix rare crashes under load due to incorrect io_uring queue size tracking - Fix monitor statistics aggregation in case of empty /osd/stats keys - Fix crash on unknown long argument to vitastor-disk - Allow trailing comma in JSONs again - Fix crash on attempts to dump a long listing of objects "to stabilize" or "to rollback" in a slow op	2023-12-04 02:36:43 +03:00
Vitaliy Filippov	845454742d	Fix warning with QEMU 8.1	2023-12-04 01:59:07 +03:00
Vitaliy Filippov	628aa59574	Raise default NBD timeout from 30 to 300 seconds and allow to take it from /etc/vitastor/vitastor.conf	2023-12-02 14:11:14 +03:00
Vitaliy Filippov	19e2d9d6fa	Fix crash on unknown long argument to vitastor-disk	2023-12-01 00:55:51 +03:00
Vitaliy Filippov	b5c020ce0b	Use io_uring SQ size for ringloop capacity - otherwise get_sqe could return NULL when space_left() was > 0 under load ... Raise default io_uring size to 1024 for the same effective capacity as previously	2023-11-20 03:04:06 +03:00
Vitaliy Filippov	6b33ae973d	%d -> %lu	2023-11-20 03:02:26 +03:00
Vitaliy Filippov	cf36445359	Reserve journal space for stabilize requests dynamically to prevent stalls	2023-11-20 03:01:57 +03:00
Vitaliy Filippov	3fd873d263	Add -fno-omit-frame-pointer by default	2023-11-20 02:59:54 +03:00
Vitaliy Filippov	a00e8ae9ed	Fix mismatch journal pos format in vitastor-disk	2023-11-19 15:19:54 +03:00
Vitaliy Filippov	75674545dc	Limit the number of printed object versions in slow op dump (otherwise it may overflow the fixed buffer)	2023-11-13 01:10:28 +03:00
Vitaliy Filippov	225eb2fe3d	Support RDMA without ODP by stupidly copying memory. Disable ODP by default ... ODP is slower than regular RDMA even with memory copy overhead Example numbers: - 3950000 random read iops without ODP vs 240000 iops with ODP - 1447000 random write iops without ODP vs 101000 iops with ODP Reference: https://tkygtr6.github.io/pub/ISPASS21_slides.pdf	2023-11-12 15:03:47 +03:00
Vitaliy Filippov	7e82573ed0	Fix RDMA connection leak which was preventing stable functioning of RDMA :)	2023-11-11 23:40:47 +03:00
Vitaliy Filippov	5524dbdab7	Release 1.2.0 ... New features: - Implement CSI volume expansion - Implement CSI volume snapshots - CSI driver now requires Kubernetes >= 1.20 Bug fixes: - Important bug fix for EC: fix EC n+k, k>=2 read recovery in ISA-L version returning incorrect data when reading at least the second chunk out of multiple missing chunks without reading the first one. All users of EC n+k, k>=2 should upgrade as soon as possible, and upgrade should be conducted with downtime: first stop all clients (VMs/containers), then all OSDs, then upgrade and restart everything. - Fix unstable statistics aggregation in monitor (affecting vitastor-cli status and df) - Make udev not wait for OSDs to start during boot - Do not report negative numbers of offline PGs in vitastor-cli status when changing PG count - Report both old and new PG counts in vitastor-cli df when changing it - Fix OSDs sometimes not starting with "The code only supports journal versions 1 and 2, but it is 2 on disk" error after upgrading from pre-1.0 versions and letting OSDs run for some time - Fix monitors sometimes returning old PG count back after OSD configuration changes - Make monitor PG changes more stable and timeout errors less probable	2023-11-05 01:48:57 +03:00
Vitaliy Filippov	cd3dec06ac	Remove spaces from old->new PG count in df	2023-11-05 01:45:45 +03:00
Vitaliy Filippov	e15b6e7805	Fix "cannot be narrowed" in clang	2023-11-04 18:14:44 +03:00
Vitaliy Filippov	31017d8412	Allow to start with V2 journal with header size from V1, as incorrectly updated by previous versions	2023-11-04 18:13:42 +03:00
Vitaliy Filippov	4819854064	Fix OSDs incorrectly updating journal superblock after upgrade to 1.x from pre-1.x and refusing to start after it	2023-11-04 15:02:24 +03:00
Vitaliy Filippov	1f509cca77	Fix unused capture warnings and void* arithmetic (clang)	2023-11-04 14:55:12 +03:00
Vitaliy Filippov	aa8e8e8271	Add version info to --help output	2023-11-04 13:32:12 +03:00
Vitaliy Filippov	4d79e531c5	Do not print "-X offline" in status when changing pool PG count, print it in df instead	2023-11-04 13:12:13 +03:00
Vitaliy Filippov	30dff8893f	Fix ISA-L version EC recovery with first missing data chunk not being read ... (Yes, all EC n + k with k >= 2 users should upgrade as soon as possible)	2023-11-04 01:34:18 +03:00
Vitaliy Filippov	becf14a705	Add a test for EC with multiple missing data chunks, but without recovery of first of them	2023-11-04 01:34:18 +03:00
Vitaliy Filippov	7c054c6f10	Add "id" to df --json output	2023-11-01 12:46:16 +03:00
Vitaliy Filippov	b4740acf62	Fix operations paused for 0.5-1 second when it happens that io_uring submit is not triggered	2023-10-28 13:18:21 +03:00
Vitaliy Filippov	8222e3c77d	Release 1.1.0 ... New features: - Implement [client writeback cache](docs/config/client.en.md#client_enable_writeback) - Add the third I/O mode: [O_DIRECT|O_SYNC](docs/config/osd.en.md#data_io) (good for Optane) - Reduce load on etcd by splitting OSD lease and statistics reporting intervals: [etcd_stats_interval](docs/config/osd.en.md#etcd_stats_interval) (default 30 sec) - Make MON automatically filter OSDs by layout (block_size/immediate_commit/bitmap_granularity) to prevent "refusing to start PGs of this pool" errors on misconfiguration - Support running fio benchmarks on systems without io_uring - Make QEMU driver compatible with QEMU 8.1 - Document usage of [vhost-user-blk](docs/usage/qemu.en.md#vhost-user-blk) Bug fixes: - Fix resizing disks in QEMU driver (for example, in Proxmox) - Fix "unexpected result" in Proxmox driver by making CLI flush output on exit - Remove unneeded block_size mismatch warnings on pools without matching PGs - Fix possible segfault in vitastor-cli ls -l (usually with deleted pools) - Fix QEMU driver compatibility with systems without io_uring - Fix monitor eating 100% CPU when etcd is down (caused by infinite retries) - Fix potential incorrect write processing with snapshots (not caught in tests but could probably lead to client hangs) - Fix buffer insertion in cluster_client (not caught in tests but could probably lead to incorrect writes in rare cases) - Fix rare OSD crash during sync operation processing - Fix a reenterability issue in cluster_client not reproducible in QEMU/fio, but reproducible with the currently developed K/V database implementation - Fix deletion of the first modified object - OSDs could crash if you modified the same object a lot of times, then deleted it, and then modified it again - Fix the fio_sec_osd test tool	2023-10-28 00:33:06 +03:00
Vitaliy Filippov	be7e76f849	Split etcd_stats_interval out of etcd_report_interval	2023-10-27 01:26:26 +03:00
Vitaliy Filippov	294a754c9e	Allow write-back by default in NBD & NFS	2023-10-27 01:26:26 +03:00
Vitaliy Filippov	8bfea6e7de	Support vitastor_c_create_epoll() in fio driver	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	bac9e34836	Allow to create vitastor_c with plain epoll without uring :-)	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	8aa4d492c1	Allow to use epoll_manager without ringloop	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	9336ee5476	Correctly free manual "small vector" in cluster_client %-)	2023-10-26 22:57:36 +03:00
Vitaliy Filippov	ad30b11519	Add the missing ringloop creation check to vitastor_c_create_uring_json()	2023-10-26 18:07:23 +03:00
Vitaliy Filippov	a061246997	Do not attempt to initialize QEMU driver via vitastor_c_create_qemu_uring() ... It doesn't add any compatibility because vitastor_c_uring_register_eventfd() is added in the same VITASTOR_C_API_VERSION 2.	2023-10-26 17:46:19 +03:00
Vitaliy Filippov	5066e35a49	Fix write-over-delete failing for the very first entry in dirty_db	2023-10-21 17:00:14 +03:00
Vitaliy Filippov	93dc31f3fc	Fix possible segfault in vitastor-cli ls -l	2023-10-18 11:11:41 +03:00
Vitaliy Filippov	f245b56176	Fix another possible reenterability issue in cluster_client ... Non-reproducible in QEMU/FIO, only caught during K/V DB debugging	2023-10-08 11:02:53 +03:00
Vitaliy Filippov	38db53f5ee	Implement client writeback cache ... - Disabled by default, enable with client_enable_writeback=true - Even then only enabled in FIO when -direct is disabled and in QEMU when block device cache is enabled in settings - Can also be enabled in other clients like vitastor-cli using parameter client_writeback_allowed=true, but not recommended	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	cd543a90bc	Prevent stack overflows in cli_merge with CAS and writeback cache	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	f600cc07b0	Autosync in blockstore every autosync_writes, too	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	6a8e530e6b	Add FIXME to timerfd_manager	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	5cadb170b9	Fix possible OSD crash during sync due to missing min_flushed_journal_sector reset	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	e72d4ed1d4	Remove unused bs_sync fields	2023-09-16 17:52:17 +03:00
Vitaliy Filippov	ff479a102d	Make MON filter OSDs by block layout to prevent "refusing to start PGs of this pool" errors on misconfiguration	2023-09-16 17:52:17 +03:00