Compare commits

base: vitalif:v0.6.7

Branches Tags

vitalif:master

vitalif:test1

vitalif:test

vitalif:test2

vitalif:kv-readahead

vitalif:qemu-fix-crash-idea

vitalif:eventloop-auto-init-trigger

vitalif:cow-meta

vitalif:hugo-docs

vitalif:node-mutex-and-postpone

vitalif:check-writeback

vitalif:msgr-iothreads-v2

vitalif:antietcd

vitalif:rel-1.4

vitalif:test-fix-ec-unknown-state-51

vitalif:kv-update

vitalif:kv-debug

vitalif:hotfix-1.2.0

vitalif:rdma-simple-nodp

vitalif:zerocopy-tcp-send

vitalif:csi-staging

vitalif:hotfix-1.1.0

vitalif:hotfix-1.0.0

vitalif:fsync-feedback

vitalif:cached-reads

vitalif:qemu-send-loop

vitalif:openssl

vitalif:test-double-alloc

vitalif:hier-failure-domains

vitalif:mon-self-restart

vitalif:epoch-deletions

vitalif:rdma-flow-control

vitalif:csi-use-vitastor-cli

vitalif:rdma-v2

vitalif:etcd-hide-base64

vitalif:lrc-matrix

vitalif:rm-left-on-dead

vitalif:nfs-proxy-old

vitalif:non-odp-rdma

vitalif:test-zctr

vitalif:separate-data-connections

vitalif:sec_osd_msgr

vitalif:test-assert

vitalif:nbd-vmsplice

vitalif:allow-etcd-address-option

vitalif:rdma-zerocopy

vitalif:rel-0.5

vitalif:openonload

vitalif:sync-io-test

vitalif:trace-sync

vitalif:trace-sqes

vitalif:test-submit-and-wait

vitalif:test-sq-poll

vitalif:blocking-uring-test

vitalif:v2.2.0

vitalif:v2.1.0

vitalif:v2.0.0

vitalif:v1.11.0

vitalif:v1.10.1

vitalif:v1.10.0

vitalif:v1.9.3

vitalif:v1.9.2

vitalif:v1.9.1

vitalif:v1.9.0

vitalif:v1.8.0

vitalif:v1.7.1

vitalif:v1.7.0

vitalif:v1.6.1

vitalif:v1.6.0

vitalif:v1.5.0

vitalif:v1.4.8

vitalif:v1.4.7

vitalif:v1.4.6

vitalif:v1.4.5

vitalif:v1.4.4

vitalif:v1.4.3

vitalif:v1.4.2

vitalif:v1.4.1

vitalif:v1.4.0

vitalif:v1.3.1

vitalif:v1.3.0

vitalif:v1.2.0

vitalif:v1.1.0

vitalif:v1.0.0

vitalif:v0.9.6

vitalif:v0.9.5

vitalif:v0.9.4

vitalif:v0.9.3

vitalif:v0.9.2

vitalif:v0.9.1

vitalif:v0.9.0

vitalif:v0.8.9

vitalif:v0.8.8

vitalif:v0.8.7

vitalif:v0.8.6

vitalif:v0.8.5

vitalif:v0.8.4

vitalif:v0.8.3

vitalif:v0.8.2

vitalif:v0.8.1

vitalif:v0.8.0

vitalif:v0.7.1

vitalif:v0.7.0

vitalif:v0.6.17

vitalif:v0.6.16

vitalif:v0.6.15

vitalif:v0.6.14

vitalif:v0.6.13

vitalif:v0.6.12

vitalif:v0.6.11

vitalif:v0.6.10

vitalif:v0.6.9

vitalif:v0.6.8

vitalif:v0.6.7

vitalif:v0.6.6

vitalif:v0.6.5

vitalif:v0.6.4

vitalif:v0.6.3

vitalif:v0.6.2

vitalif:v0.6.1

vitalif:v0.6.0

vitalif:v0.5.13

vitalif:v0.5.12

vitalif:v0.5.11

vitalif:v0.5.10

vitalif:v0.5.9

vitalif:v0.5.7

vitalif:v0.5.6

vitalif:v0.5.5

vitalif:v0.5.4

vitalif:v0.5.3

vitalif:v0.5.2

vitalif:v0.5.1

vitalif:v0.5.0

vitalif:v0.4.0

vitalif:v0.5.8

...

compare: vitalif:master

Branches Tags

vitalif:master

vitalif:test1

vitalif:test

vitalif:test2

vitalif:kv-readahead

vitalif:qemu-fix-crash-idea

vitalif:eventloop-auto-init-trigger

vitalif:cow-meta

vitalif:hugo-docs

vitalif:node-mutex-and-postpone

vitalif:check-writeback

vitalif:msgr-iothreads-v2

vitalif:antietcd

vitalif:rel-1.4

vitalif:test-fix-ec-unknown-state-51

vitalif:kv-update

vitalif:kv-debug

vitalif:hotfix-1.2.0

vitalif:rdma-simple-nodp

vitalif:zerocopy-tcp-send

vitalif:csi-staging

vitalif:hotfix-1.1.0

vitalif:hotfix-1.0.0

vitalif:fsync-feedback

vitalif:cached-reads

vitalif:qemu-send-loop

vitalif:openssl

vitalif:test-double-alloc

vitalif:hier-failure-domains

vitalif:mon-self-restart

vitalif:epoch-deletions

vitalif:rdma-flow-control

vitalif:csi-use-vitastor-cli

vitalif:rdma-v2

vitalif:etcd-hide-base64

vitalif:lrc-matrix

vitalif:rm-left-on-dead

vitalif:nfs-proxy-old

vitalif:non-odp-rdma

vitalif:test-zctr

vitalif:separate-data-connections

vitalif:sec_osd_msgr

vitalif:test-assert

vitalif:nbd-vmsplice

vitalif:allow-etcd-address-option

vitalif:rdma-zerocopy

vitalif:rel-0.5

vitalif:openonload

vitalif:sync-io-test

vitalif:trace-sync

vitalif:trace-sqes

vitalif:test-submit-and-wait

vitalif:test-sq-poll

vitalif:blocking-uring-test

vitalif:v2.2.0

vitalif:v2.1.0

vitalif:v2.0.0

vitalif:v1.11.0

vitalif:v1.10.1

vitalif:v1.10.0

vitalif:v1.9.3

vitalif:v1.9.2

vitalif:v1.9.1

vitalif:v1.9.0

vitalif:v1.8.0

vitalif:v1.7.1

vitalif:v1.7.0

vitalif:v1.6.1

vitalif:v1.6.0

vitalif:v1.5.0

vitalif:v1.4.8

vitalif:v1.4.7

vitalif:v1.4.6

vitalif:v1.4.5

vitalif:v1.4.4

vitalif:v1.4.3

vitalif:v1.4.2

vitalif:v1.4.1

vitalif:v1.4.0

vitalif:v1.3.1

vitalif:v1.3.0

vitalif:v1.2.0

vitalif:v1.1.0

vitalif:v1.0.0

vitalif:v0.9.6

vitalif:v0.9.5

vitalif:v0.9.4

vitalif:v0.9.3

vitalif:v0.9.2

vitalif:v0.9.1

vitalif:v0.9.0

vitalif:v0.8.9

vitalif:v0.8.8

vitalif:v0.8.7

vitalif:v0.8.6

vitalif:v0.8.5

vitalif:v0.8.4

vitalif:v0.8.3

vitalif:v0.8.2

vitalif:v0.8.1

vitalif:v0.8.0

vitalif:v0.7.1

vitalif:v0.7.0

vitalif:v0.6.17

vitalif:v0.6.16

vitalif:v0.6.15

vitalif:v0.6.14

vitalif:v0.6.13

vitalif:v0.6.12

vitalif:v0.6.11

vitalif:v0.6.10

vitalif:v0.6.9

vitalif:v0.6.8

vitalif:v0.6.7

vitalif:v0.6.6

vitalif:v0.6.5

vitalif:v0.6.4

vitalif:v0.6.3

vitalif:v0.6.2

vitalif:v0.6.1

vitalif:v0.6.0

vitalif:v0.5.13

vitalif:v0.5.12

vitalif:v0.5.11

vitalif:v0.5.10

vitalif:v0.5.9

vitalif:v0.5.7

vitalif:v0.5.6

vitalif:v0.5.5

vitalif:v0.5.4

vitalif:v0.5.3

vitalif:v0.5.2

vitalif:v0.5.1

vitalif:v0.5.0

vitalif:v0.4.0

vitalif:v0.5.8

1345 Commits

v0.6.7 ... master

Author	SHA1	Message	Date
Vitaliy Filippov	db915184c6	Allow to purge running OSDs again, as in 2.1.0 and earlier	2025-05-11 13:59:28 +03:00
Vitaliy Filippov	5ae6fea49c	Add a note about local reads	2025-05-11 01:23:48 +03:00
Vitaliy Filippov	95ec750b8c	Release 2.2.0 ... New features: - [Localized read support](https://vitastor.io/docs/config/pool.html#local_reads) for multi-datacenter setups. - io_uring-based [zero-copy send support](https://vitastor.io/docs/config/network.html#min_zerocopy_send_size) - read the instruction carefully for optimal performance! - Improve and speedup data distribution, especially in cases of very large hosts (100 OSD+). Previously, PG optimization speed depended on the number of OSDs, now it only depends on the number of failure domains. Distribution over specific OSDs is now also more even and becomes strictly more even when you increase the number of PGs. - Add a [very interesting instruction](https://vitastor.io/en/docs/usage/nfs.html#linux-nfs-write-size) to change NFS_MAX_FILE_IO_SIZE - Check operation sequencing and stop connections when it breaks - should help catch some very rare RDMA packet loss problems. - `vitastor-cli rm-osd` now refuses to remove OSDs which are still up and suggests to use `vitastor-disk purge`. - Allow removal of direntries referring non-existent in VitastorFS. - Change default vitastor-etcd data dir to /var/lib/etcd/vitastor. Bug fixes: - Fix compatibility with ISA-L 2.31+. ⚠️Very important: please upgrade Vitastor before upgrading ISA-L to 2.31+. - Fix in-memory state cleanup for incomplete PGs. - Fix monitor crash with non-existent node_placement nodes. - Slightly speedup `vitastor-kv dump` command by adding output buffering. - Fix theoretically possible slowdowns in OSD sub-operation failure handling code. - Fix very rare stack overflows in vitastor-kv. - Fix a possible crash in VitastorFS during handling of file creation race condition. - Fix modify-pool -s PG_SIZE which didn't work without --pg_minsize. - Fix marking peer OSDs as alive on receiving data from them via RDMA - in theory, the bug could result in instability with RDMA under high load with slow disks. - Fix a rare OSD crash due to double handle_primary_subop() call. - Fix latency aggregation in global stats (/vitastor/stats in etcd) - do not sum it. - Hide "Ran out of journal space" log messages by default. - Wait for RDMA-CM EVENT_ESTABLISHED after rdma_accept(), handle rdma_accept() before acking the event. - Fix VitastorFS total & free numbers multiplied by extra 2. - Fix systemd unit name in make-etcd. - Do not allow reweight > 1 in vitastor-cli modify-osd. - Fix docker build.	2025-05-11 00:26:08 +03:00
Vitaliy Filippov	90b1de307b	Support local reads in client	2025-05-10 16:42:02 +03:00
Vitaliy Filippov	7e6a95c678	Support primary-reads from clean replicated PGs on secondary OSDs	2025-05-10 16:42:02 +03:00
Vitaliy Filippov	b2416afb28	Lock PGs on secondary OSDs to allow local reads and guarantee splitbrain prevention	2025-05-10 15:18:00 +03:00
Vitaliy Filippov	66dc116f60	Cleanup PG_INCOMPLETE peering states	2025-05-10 02:51:26 +03:00
Vitaliy Filippov	0cb8629ab6	Remove finish_stop_pg shortcut	2025-05-08 16:14:35 +03:00
Vitaliy Filippov	b7322a405a	Move gethostname_str to utils	2025-05-05 02:16:06 +03:00
Vitaliy Filippov	5692630005	Move check_sequencing indication into config response features subkey	2025-05-05 02:07:50 +03:00
Vitaliy Filippov	00ced7cea7	Fix monitor crash with non-existing node_placement nodes	2025-05-05 02:05:04 +03:00
Vitaliy Filippov	ebdb75e287	Fix typo in docker docs	2025-05-04 18:09:33 +03:00
Vitaliy Filippov	f397fe9c6a	Add compatibility with ISA-L 2.31+	2025-05-04 18:09:33 +03:00
Vitaliy Filippov	28560b4ae5	Write K/V listings in buffered manner	2025-05-03 15:06:59 +03:00
Vitaliy Filippov	2d07449e74	Postpone cb() to set_immediate() to prevent stack overflows in kv_db	2025-05-03 15:06:59 +03:00
Vitaliy Filippov	80c4e8c20f	Add missing wakeup in ringloop->set_immediate to prevent slowdowns in code using set_immediate	2025-05-03 14:40:48 +03:00
Vitaliy Filippov	2ab0ae3bc9	Check operation sequencing and stop clients when it breaks	2025-05-02 17:01:50 +03:00
Vitaliy Filippov	05e59c1b4f	Fix MSG_WAITALL assertion added in the zero-copy patch	2025-05-02 17:01:43 +03:00
Vitaliy Filippov	e6e1c5b962	Check if OSDs are still up in rm-osd	2025-05-02 13:05:59 +03:00
Vitaliy Filippov	9556eeae45	Implement io_uring zero-copy send support	2025-05-01 18:47:10 +03:00
Vitaliy Filippov	96b5a72630	Allow removal of bad direntries in VitastorFS (direntries referring non-existent inodes)	2025-05-01 01:14:23 +03:00
Vitaliy Filippov	ef80f121f6	Fix "duplicate inode during create" deletion in VitastorFS	2025-04-30 20:37:49 +03:00
Vitaliy Filippov	bbdd1f3aa7	Fix modify-pool -s PG_SIZE without --pg_minsize	2025-04-28 02:20:54 +03:00
Vitaliy Filippov	5dd37f519a	Fix node folding in case of empty rules (pool with size 1), add a test	2025-04-28 02:16:49 +03:00
Vitaliy Filippov	a2278be84d	Improve data distribution: solve LP task on failure domains instead of individual OSDs ... This greatly speeds up PG placement and makes it more uniform both because the LP task becomes simpler and because the distribution of individual OSDs is optimised manually	2025-04-27 01:44:46 +03:00
Vitaliy Filippov	1393a2671c	Change default vitastor-etcd data dir to /var/lib/etcd/vitastor	2025-04-27 01:44:46 +03:00
Vitaliy Filippov	9fa8ae5384	Reset OSD ping state on receiving data from it via RDMA	2025-04-26 14:17:09 +03:00
Vitaliy Filippov	169a35a067	Followup to latency aggregation fix	2025-04-26 01:32:46 +03:00
Vitaliy Filippov	2b2a10581d	Prevent double handle_primary_subop in rare cases	2025-04-26 01:16:53 +03:00
Vitaliy Filippov	10fd51862a	Fix latency aggregation in global stats (/vitastor/stats in etcd)	2025-04-25 00:08:10 +03:00
Vitaliy Filippov	15d0204f96	Hide "Ran out of journal space" log messages by default	2025-04-20 01:21:03 +03:00
Vitaliy Filippov	21d6e88a1b	Add instructions to change NFS_MAX_FILE_IO_SIZE	2025-04-18 13:39:32 +03:00
Vitaliy Filippov	df2847df2d	Wait for RDMA-CM EVENT_ESTABLISHED after rdma_accept(), handle rdma_accept() before acking the event	2025-04-15 15:19:36 +03:00
Vitaliy Filippov	327c98a4b6	Fix index_tree	2025-04-13 16:13:44 +03:00
Vitaliy Filippov	3cc0abfd81	Fix NFS total & free multiplied by extra 2	2025-04-12 19:09:26 +03:00
Vitaliy Filippov	80e5f8ba76	Add missing WITH_RDMACM defines	2025-04-12 19:06:27 +03:00
Vitaliy Filippov	4b660f1ce8	Fix systemd unit name in make-etcd	2025-04-11 02:05:08 +03:00
Vitaliy Filippov	dfde0e60f0	Do not allow reweight > 1	2025-04-05 12:21:14 +03:00
Vitaliy Filippov	013f688ffe	Run check_peer_config on RDMA-CM connections too	2025-04-02 01:32:28 +03:00
Vitaliy Filippov	cf9738ddbe	Fix docker 2.1.0 build :)	2025-04-01 22:46:22 +03:00
Vitaliy Filippov	891b2811c7	Release 2.1.0 ... New features: - Support separate OSD cluster network - [osd_cluster_network](https://vitastor.io/docs/config/network.html#osd_cluster_network) and, in general, multiple OSD networks, including RDMA - Add an alternative RDMA implementation via RDMA-CM - [use_rdmacm](https://vitastor.io/docs/config/network.html#use_rdmacm), required for iWARP and, maybe, for some IB setups (but not for RoCE) - Change default PG behaviour to wait for all "up" OSDs to be connected before starting it. The old behaviour may be returned by enabling a new [allow_net_split](https://vitastor.io/docs/config/osd.html#allow_net_split) option. - Add a patch for QEMU 9.2 Bug fixes: - Fix incorrect "has_xxx" PG state names in ls-pgs - Fix possible QEMU crashes after detaching of Vitastor disks (and update all QEMU builds in Vitastor repos) - Fix clients sometimes spamming OSDs with infinite reconnections when some PGs are offline - Fall back to TCP on RDMA connection failures - Add missing logging of RDMA ibv_modify_qp() errors - Add a minimum interval for etcd_state_client to reload state	2025-04-01 20:16:27 +03:00
Vitaliy Filippov	01590df6da	Update QEMU version in vitastor-csi Dockerfile	2025-04-01 20:16:27 +03:00
Vitaliy Filippov	3e5f0be52c	Use separate port numbers for RDMA-CM	2025-04-01 16:16:03 +03:00
Vitaliy Filippov	58af897e73	s/listen on/listen to/ :)	2025-04-01 12:07:15 +03:00
Vitaliy Filippov	dbf9ecd171	Move osd_network to config/network docs	2025-03-31 21:12:09 +03:00
Vitaliy Filippov	8508e78288	Add an alternative RDMA implementation via RDMA-CM ... Required for non-RoCE cards: iWARP and, possibly, Infiniband	2025-03-31 21:01:25 +03:00
Vitaliy Filippov	f32dea02bf	Support multiple RDMA networks	2025-03-31 21:01:25 +03:00
Vitaliy Filippov	a103065d12	Support multiple OSD networks and separate OSD cluster network	2025-03-31 21:01:15 +03:00
Vitaliy Filippov	5d2e28d4a9	Remove unused used_max_cqe from nfs_proxy_rdma	2025-03-30 02:13:24 +03:00
Vitaliy Filippov	18e14eed11	Fix --pg_count formula in docs/usage/cli	2025-03-29 17:54:53 +03:00
Vitaliy Filippov	ccc32b9e68	Use TCP on RDMA connection failure	2025-03-23 12:04:23 +03:00
Vitaliy Filippov	ebaf3fee79	Add an assertion to prevent sending message to TCP channel when switched to RDMA	2025-03-23 12:04:09 +03:00
Vitaliy Filippov	196d28e987	Fix typo	2025-03-23 12:00:20 +03:00
Vitaliy Filippov	8f243b2328	Fix qemu buster build and bullseye version	2025-03-23 02:46:52 +03:00
Vitaliy Filippov	7a835fcd8f	Add allow_net_split parameter	2025-03-23 02:12:32 +03:00
Vitaliy Filippov	8b0389b4e8	Log RDMA ibv_modify_qp() errors	2025-03-22 15:58:13 +03:00
Vitaliy Filippov	f544c350ba	%l* -> %j*	2025-03-22 15:32:07 +03:00
Vitaliy Filippov	4eafb55b5c	Add a patch for QEMU 9.2, fix debian bookworm QEMU build	2025-03-22 15:30:52 +03:00
Vitaliy Filippov	5030396f71	Clear QEMU eventfd handler on vitastor block driver destruction	2025-03-21 20:47:17 +03:00
Vitaliy Filippov	be22c363ca	Do not skip client_retry_interval on reconnecting OSDs to prevent OSD spam	2025-03-20 00:12:38 +03:00
Vitaliy Filippov	0f80c87b43	Add a minimum interval for etcd_state_client to reload state ... (To prevent excessive load on etcd during outages)	2025-03-19 02:36:09 +03:00
Vitaliy Filippov	e0953fd502	Wait for all "up" OSDs to be connected before starting PG	2025-03-19 02:36:09 +03:00
Vitaliy Filippov	6e0ae47938	Add Proxmox QEMU 9.2 patch	2025-03-19 02:36:02 +03:00
Vitaliy Filippov	b8f19e85ad	Fix pg state formatting in ls-pgs	2025-03-17 01:37:58 +03:00
Vitaliy Filippov	b7636e595f	Update version in docker docs	2025-03-16 16:53:57 +03:00
Vitaliy Filippov	48c026bfa0	Release 2.0.0 ... No breaking features, it's 2.0.0 just because it includes S3 and because there are already too many 1.x releases :). New features: - S3 is finally available: https://vitastor.io/docs/installation/s3.html - node.js addon is now packaged as a Debian package - Support listing PGs by OSDs in `vitastor-cli ls-pgs` - Implement offline TRIM support: [vitastor-disk trim](https://vitastor.io/docs/usage/disk.html#trim), [discard_on_start](https://vitastor.io/docs/config/osd.html#discard_on_start) - Change used_for_fs pool option to used_for_app Bug fixes: - Fix several bugs in the node.js addon (a memory leak, an incorrectly triggered event loop) - Fix a client crash (vitastor-cli rm) during deletion when writeback is enabled - Fix PG object count statistics on deletion of non-existing objects - Fix vitastor-nbd crash when mapping by ID instead of inode name - Fix a client memory leak with enabled immediate_commit and write-back cache - Add seccomp=unconfined for vitastor docker OSDs to not break io_uring - Add udev and systemd to vitastor docker image - Fix upgrading from pre-0.7.1 (very old) systemd units O_o - Fix total object count calculation in rm_data	2025-03-16 14:34:31 +03:00
Vitaliy Filippov	a73b2a26b6	Fix blockstore initialization after moving clean_dyn_size calc to calc_lengths	2025-03-16 13:44:02 +03:00
Vitaliy Filippov	f3192b610d	Fix vitastor-disk in Docker installations	2025-03-16 13:44:01 +03:00
Vitaliy Filippov	a950889976	Add missing docs for discard_on_start	2025-03-16 12:29:22 +03:00
Vitaliy Filippov	ef5194d93c	Add S3 installation docs	2025-03-16 01:17:09 +03:00
Vitaliy Filippov	f904576ab1	Fix total calculation in rm_data	2025-03-15 17:01:10 +03:00
Vitaliy Filippov	4f9b1f2f62	Support listing PGs by OSDs	2025-03-15 16:42:57 +03:00
Vitaliy Filippov	1d94afbd51	Implement offline TRIM support	2025-03-14 01:37:16 +03:00
Vitaliy Filippov	3634f005f1	Fix upgrading from pre-0.7.1 systemd units O_o	2025-03-14 01:37:16 +03:00
Vitaliy Filippov	263a3b5ad6	Rename allocator to allocator_t	2025-03-13 00:53:34 +03:00
Vitaliy Filippov	b760951aa7	Add seccomp=unconfined for vitastor docker OSDs to not break io_uring	2025-03-11 00:42:10 +03:00
Vitaliy Filippov	c8321b8ed1	Add udev and systemd to vitastor docker image	2025-03-11 00:40:39 +03:00
Vitaliy Filippov	21066a095b	Fix a memory leak with enabled immediate_commit and write-back cache ... Remove dirty buffers after writing when immediate_commit is on instead of saving them for repeating later	2025-03-11 00:40:18 +03:00
Vitaliy Filippov	a96900b696	Explicitly destroy Nan::Persistents, otherwise it leaks memory	2025-03-09 16:45:10 +03:00
Vitaliy Filippov	8a6e461322	Fix license (VNPL 1.1, not 2.0)	2025-03-08 17:17:23 +03:00
Vitaliy Filippov	0b6a0463a4	Save a reference to the buffer during write	2025-03-08 16:00:26 +03:00
Vitaliy Filippov	35d4047f46	Fix vitastor-nbd crash when mapping by ID instead of inode name	2025-03-08 15:52:57 +03:00
Vitaliy Filippov	819f1125ae	Support used_for_app instead of used_for_fs	2025-03-07 01:03:43 +03:00
Vitaliy Filippov	108df7329f	Fix PG object count statistics on deletion of non-existing objects	2025-03-04 00:40:56 +03:00
Vitaliy Filippov	d32edf6cdf	Fix deletion writeback	2025-03-04 00:40:35 +03:00
Vitaliy Filippov	dca436d7e6	Trigger event loop automatically in libvitastor_c	2025-03-03 00:57:09 +03:00
Vitaliy Filippov	8129a0b4e3	Loop once after registering eventfd to prevent skipping previous events	2025-03-03 00:57:00 +03:00
Vitaliy Filippov	704c87d512	Trigger initial epoll when adding an FD	2025-03-03 00:56:17 +03:00
Vitaliy Filippov	10216a5fb5	Build node.js addon as a Debian package	2025-03-02 18:04:56 +03:00
Vitaliy Filippov	3932eb7ff6	Trigger event loop once after each vitastor_c_* call	2025-03-02 01:23:41 +03:00
Vitaliy Filippov	69cbe7bbb2	Release 1.11.0 ... New features: - Support containerized Vitastor installations: http://vitastor.io/docs/installation/docker.html - Add new functions to the node.js binding: delete(), get_immediate_commit(), on_ready(), get_min_io_size(), get_max_atomic_write_size() - S3 (Zenko Cloudserver with Vitastor support) is coming shortly and will be released separately Bug fixes: - Use IP-derived etcd node names in make-etcd - Set short name of the OSD process to display in `top` - Fix snap-create without pool_id failing when there are multiple pools - Several bugs are fixed in the write-back cache, it should now be stable: - Fix incorrect snapshot reads from dirty write-back cache - Do not try to repeat pending writebacks on OSD reconnections - Fix client hangs with multiple SYNCs in the writeback queue - Fix client hangs do to incorrect calculation of the writeback queue size - Several improvements for NBD mapping/unmapping: - Add a workaround for race condition in the Linux kernel NBD driver leading to vitastor-nbd sometimes breaking a previously mapped device instead of setting up a new one - Check if the device is actually mapped in vitastor-nbd unmap - Fix device name/number validation in vitastor-nbd - Fix OSD crashes after starting with corrupted metadata - from now it will skip corrupted metadata entries and heal itself - Fix scrubbing of misplaced objects and object state recalculation after vitastor-cli fix - previously, an OSD restart could be required to fix object states - Make primary OSD distribution more stable by using murmur3 hash instead of the old pseudo-rng - Fix monitor sometimes racing with itself - do not touch /pool/stats from stats aggregation if PG recheck is active - Sort vitastor-cli ls output by name by default - Update antietcd to 1.1.2	2025-03-01 13:39:42 +03:00
Vitaliy Filippov	4950a1636c	Allow "infinite" startup for clients if explicitly requested	2025-03-01 13:39:42 +03:00
Vitaliy Filippov	2eb20dff28	Do not crash on io_uring initialization failure in node-vitastor	2025-03-01 13:29:48 +03:00
Vitaliy Filippov	59f0b0427c	Support containerized Vitastor installations	2025-02-27 20:06:15 +03:00
Vitaliy Filippov	124162ad38	Use IP-derived etcd node names in make-etcd	2025-02-26 11:54:37 +03:00
Vitaliy Filippov	391c92af1a	Set OSD process name	2025-02-26 11:54:37 +03:00
Vitaliy Filippov	c3d8fdd855	Fix snap-create without pool_id ID generation with multiple pools	2025-02-26 11:54:28 +03:00
Vitaliy Filippov	9ccf3af97b	Add qemu-block-extra and qemu-utils	2025-02-23 15:08:16 +03:00
Vitaliy Filippov	568a209f0d	Update docker image to debian bookworm	2025-02-23 13:27:32 +03:00
Vitaliy Filippov	b151013201	Fix snapshot reads from a dirty write-back cache	2025-02-23 02:31:19 +03:00
Vitaliy Filippov	4a763725fe	Add free() to bindiff.c	2025-02-22 16:52:19 +03:00
Vitaliy Filippov	b8d83cd7f4	No, it's not a good idea to destroy client in the child nbd process ... Should probably have been an obvious side effect :-) Child process gets open file descriptors to parent's epoll/timerfd, and it's totally OK to just close() all of them, but it's absolutely NOT OK to run destructors - they modify the kernel state of epoll/timerfd before destroying. So, basically, when we destroy the client in the child process, we break it in the parent too. This also means that cluster_client_t doesn't support fork(). :-)	2025-02-22 15:10:27 +03:00
Vitaliy Filippov	2e9ee2fe20	Do not try to repeat pending writebacks	2025-02-22 14:16:44 +03:00
Vitaliy Filippov	508ae852e4	Fix trap in test_rebalance_verify	2025-02-22 02:18:41 +03:00
Vitaliy Filippov	97ee400505	Add a workaround for race condition in the Linux kernel NBD driver ... Do all NBD configuration in the child process, after the last fork. Why? It's needed because there is a race condition in the Linux kernel nbd driver in nbd_add_socket() - it saves `current` task pointer as `nbd->task_setup` and then rechecks if the new `current` is the same. Problem is that if that process is already dead, `current` may be freed and then replaced by another process with the same pointer value. So the check passes and NBD allows a different process to set up a device which is already set up. Proper fix would have to be done in the kernel code, but the workaround is obviously to perform NBD setup from the process which will then actually call NBD_DO_IT. That process stays alive during the whole time of NBD device execution and the (nbd->task_setup != current) check always works correctly, and we don't accidentally break previous NBD devices while setting up a new device. Forking to check every device is of course rather slow, so we also do an additional check by calling list_mapped() before searching for a free NBD device.	2025-02-21 13:17:37 +03:00
Vitaliy Filippov	5ee4894fab	Check if mapped in vitastor-nbd unmap	2025-02-21 01:28:06 +03:00
Vitaliy Filippov	125dcafb11	Prevent OSD crashes when metadata is corrupted	2025-02-20 02:19:32 +03:00
Vitaliy Filippov	9f44cf71df	Fix device name/number validation in vitastor-nbd	2025-02-20 01:33:11 +03:00
Vitaliy Filippov	df3c63ca7f	Sort vitastor-cli ls by name by default	2025-02-20 01:32:49 +03:00
Vitaliy Filippov	be66edd09f	Prevent infinite loops on syncs in writeback_overflow	2025-02-19 01:44:12 +03:00
Vitaliy Filippov	ccbc0c5928	Add assert !writeback_bytes	2025-02-19 01:15:46 +03:00
Vitaliy Filippov	78ca4538bf	Fix qemu docker build for ubuntu	2025-02-18 23:44:16 +03:00
Vitaliy Filippov	86b5760ec1	Fix writeback incorrectly calculating queue size which was leading to client hangs	2025-02-18 23:42:55 +03:00
Vitaliy Filippov	27f3803d2f	Add vitastor_c_delete() and delete() to the node.js binding	2025-02-15 18:27:17 +03:00
Vitaliy Filippov	2ead06e126	Add ubuntu jammy to docs	2025-02-12 15:32:35 +03:00
Vitaliy Filippov	a5d5559f8e	Add get_immediate_commit() to the node.js binding	2025-02-06 01:35:48 +03:00
Vitaliy Filippov	e8e7ba8fde	Add FIXME for CAS in non-immediate_commit mode	2025-02-06 01:35:48 +03:00
Vitaliy Filippov	6fd831a299	Add on_ready(), get_min_io_size(), get_max_atomic_write_size() to the node.js binding	2025-02-06 01:35:48 +03:00
Vitaliy Filippov	069808dfce	Fix --config_path option in docs	2025-01-24 17:21:11 +03:00
Vitaliy Filippov	bcefa42bc0	Scrub all chunks, not just 1 chunk per position	2025-01-23 02:02:55 +03:00
Vitaliy Filippov	4636e02d43	Remove scheme, pg_size, pg_data_size from op_data	2025-01-23 01:20:31 +03:00
Vitaliy Filippov	e4c7d1c147	s/3/4/	2025-01-23 01:20:31 +03:00
Vitaliy Filippov	a4677f3e69	Mention P5530	2025-01-23 01:20:31 +03:00
Vitaliy Filippov	7cbf207d65	Use murmur3 to select primary OSD instead of old pseudo-rng	2025-01-18 12:28:54 +03:00
Vitaliy Filippov	7c9711af20	Do not touch /pool/stats from stats aggregation if PG recheck is active	2025-01-16 20:41:16 +03:00
Vitaliy Filippov	33ef701464	Update antietcd to 1.1.2	2025-01-04 02:13:36 +03:00
Vitaliy Filippov	61ededa230	Release 1.10.1 ... New features: - Add "deleted" image flag which is set when vitastor-cli rm starts to delete an image, but can't delete it fully due to inactive PGs or stopped OSDs - Support JSON output in vitastor-disk prepare and purge - Show backfillfull pools in vitastor-cli status - Make object listings consistent (used in vitastor-cli rm/rm-data/merge/etc). This means that there is now a guarantee that if a data block is present when you invoke rm, rm will attempt to delete it, even if rm is invoked when the PG switches state. Previously in such cases rm could skip and leave some objects behind as garbage, and merge probably could incorrectly move data between snapshots. - Make deletions (rm/rm-data) consistent. This means that rm/rm-data will either complete successfully and delete all requested image data or complete with an error if some objects could not be deleted or if there is a possibility that some data is left on stopped OSDs. Previously, when some PGs or OSDs were inactive at the moment of deletion, rm-data was behaving incorrectly: it wasn't retrying deletions failed due to dropped OSD connections, it could hang waiting for PGs to activate, and it could return with a successful error code while some garbage was still possibly left on some OSDs. Deletions are not fully atomic cluster-wide yet, which means that you still have to repeat the deletion request after you return stopped OSDs back, but now you always know for sure if you have to repeat it. Bug fixes: - Fix vitastor-cli rm --exact / --matching command not working - Finally fix "Unexpected status" in the Proxmox plugin - Fix vitastor-cli create-snap incorrectly linking multiple snapshots in a different pool - Fix incomplete image parent_id loop check in OSD - Fix reads from snapshots in a different pool not working if there are more than 2 snapshots - Fix append of VITASTOR_CONF to cmdline in the opennebula prebackup script - Fix OSDs crashing again when the cluster is full with EC (was meant to work since 1.6.0 but didn't) - Improve logging of subop failures	2025-01-03 16:22:09 +03:00
Vitaliy Filippov	d9d90d3183	Fix build for debian buster	2025-01-03 16:21:56 +03:00
Vitaliy Filippov	9dbcdbcec9	Return left_on_dead OSD list in DELETE replies and use it in rm-data	2025-01-03 15:57:09 +03:00
Vitaliy Filippov	a147f7e7dc	Copy & repeat deletions too	2025-01-03 00:21:52 +03:00
Vitaliy Filippov	0e6bf66734	Add bindiff for tests	2025-01-02 19:59:04 +03:00
Vitaliy Filippov	ab822d3050	Support consistent listings in client (rm-data, merge and etc)	2025-01-02 18:07:12 +03:00
Vitaliy Filippov	d5366a0767	Support listings from primary OSDs (for consistent deletions)	2025-01-02 11:07:24 +03:00
Vitaliy Filippov	40b8a8b0da	Add wait_up_timeout support to cluster_client and use it in vitastor-cli rm-data & merge	2025-01-01 17:57:58 +03:00
Vitaliy Filippov	5c5119aba4	Pass min_offset/max_offset to list_inode()	2025-01-01 15:40:12 +03:00
Vitaliy Filippov	4edda88903	Wait for OSDs to either connect or stop infinitely during listing, not for peer_connect_timeout	2025-01-01 15:29:42 +03:00
Vitaliy Filippov	80dda3ca94	Remove separate list_inode_next()	2025-01-01 14:19:18 +03:00
Vitaliy Filippov	c8decb32e8	Rename to client_wait_up_timeout	2025-01-01 11:26:57 +03:00
Vitaliy Filippov	4995592e61	Retry listings on broken OSD connections	2025-01-01 11:14:36 +03:00
Vitaliy Filippov	d9f9b0bca5	Start listings consistently with the current PG state, add wait_up_timeout ... This still doesn't make listings 100% consistent yet; for 100% consistent listings we have to receive listings only from the primary OSD, not from all peer OSDs, but this issue will be fixed separately.	2025-01-01 10:58:22 +03:00
Vitaliy Filippov	d0396267d0	Clear retry_timeout when the client is destroyed	2025-01-01 10:58:22 +03:00
Vitaliy Filippov	b46d5db115	Support JSON output in vitastor-disk prepare and purge	2024-12-29 15:19:44 +03:00
Vitaliy Filippov	ecd92655fe	Fix rm --exact / --matching not removing one uppermost image in each chain	2024-12-28 21:53:49 +03:00
Vitaliy Filippov	383712148b	Fix rm --exact / --matching not being invoked at all O_o	2024-12-28 21:47:00 +03:00
Vitaliy Filippov	42d40153ff	Do not intercept STDERR in Proxmox plugin (finally fixes "unexpected status"!)	2024-12-28 21:18:49 +03:00
Vitaliy Filippov	561b36a4c1	Use revision from txn response header, not from put subresponse	2024-12-28 21:01:15 +03:00
Vitaliy Filippov	685af019f5	Allow :: and 0.0.0.0 as local IPs in antietcd_adapter	2024-12-28 20:52:27 +03:00
Vitaliy Filippov	a31592d131	Print sizes in "Auto-selecting" as "4K", not "4 K"	2024-12-28 19:15:23 +03:00
Vitaliy Filippov	28b0a2597d	Add a test for multiple snapshots in a second pool	2024-12-28 18:57:30 +03:00
Vitaliy Filippov	de6b345473	Fix create-snap taking parent_pool from incorrect key parent_pool_id	2024-12-28 18:53:29 +03:00
Vitaliy Filippov	8bf52d6e96	Fix inode parent_id loop check	2024-12-28 18:40:17 +03:00
Vitaliy Filippov	5623dca02c	Fix vitastor client passing incorrect mod_revision for snapshotted images ... This was leading to reads only working for the image itself and for its latest snapshot	2024-12-28 16:01:35 +03:00
Vitaliy Filippov	abdc207297	Fix append of VITASTOR_CONF to cmdline in the opennebula prebackup script	2024-12-28 13:33:24 +03:00
Vitaliy Filippov	044e621b62	Add test_rm_degraded to CI	2024-12-27 18:31:58 +03:00
Vitaliy Filippov	ba9aabf187	Return listing errors from list_inode_start(), abort merging and fail deletion on unsuccessfull listings	2024-12-27 18:31:21 +03:00
Vitaliy Filippov	5c890e4a12	Fix rm-data hanging when some OSDs are inactive, add a test for it ... There's also another case which also needs to be fixed - we shouldn't retry deletions for indefinite time if an OSD is stopped during deletion	2024-12-27 16:29:33 +03:00
Vitaliy Filippov	0b0c2afbce	Implement "deleted" flag	2024-12-27 01:18:55 +03:00
Vitaliy Filippov	651c055bd9	Show backfillfull pools in vitastor-cli status	2024-12-26 12:17:47 +03:00
Vitaliy Filippov	42eebfc1bd	Fix OSDs still crashing when the cluster is full with EC ... ENOSPC handling was introduced in 1.6.0 but it was not complete; now it is P.S: See also client_retry_enospc (true by default)	2024-12-26 01:56:33 +03:00
Vitaliy Filippov	cef98052f5	Improve logging of subop failures	2024-12-26 01:54:40 +03:00
Vitaliy Filippov	7fbb04fdfa	Release 1.10.0 ... New features: - Implement basic VitastorFS support in [CSI](https://vitastor.io/docs/installation/kubernetes.html) - Implement [NFS RDMA](https://vitastor.io/docs/usage/nfs.html#rdma) support - Pause pool rebalance when monitor detects that it can lead to any OSD becoming full ([osd_backfillfull_ratio](https://vitastor.io/docs/config/monitor.html#osd_backfillfull_ratio)) - Auto-select correct [RDMA device and GID](https://vitastor.io/docs/config/network.html#rdma_device) based on osd_network and RoCEv2 priority - Report slow ops in OSD stats in etcd and show them in vitastor-cli status Bug fixes: - Fix possibly incorrect linked list deserialization in NFS - Fix possible crash in vitastor-nfs --block READDIR operation - Map netlink after forking to show correct PID in vitastor-nbd ls - Simplify and fix create-pool OSD count checks for the case of hosts split into sub-nodes - Make monitor print "Waiting to become master" just once, not every 5s - Take out_size from oimg if not specified in vitastor-cli dd - Do not report OSDs with empty statistics as "full" in status - Trigger double autosync when switching PG state to prevent leaving garbage in non-immediate_commit clusters - Fix a lack of connection timeout for etcd websockets in OSD leading to slower etcd failover (~70s instead of ~10s) - Fix a rare OSD crash during client disconnect - Fix PGs sometimes sticking until OSD restart in the "has_unclean" state with EC pools - Fix metadata partition zeroing in vitastor-disk prepare - Add patches for qemu 9.1 and pve-qemu 9.0 and 9.1 - Fix libvirt 8 patch	2024-12-19 15:49:19 +03:00
Vitaliy Filippov	63b85b6bfb	Fix clang warnings/errors	2024-12-19 15:30:31 +03:00
Vitaliy Filippov	2f5959e3fa	Add pve-qemu 9.1 patch	2024-12-19 14:05:12 +03:00
Vitaliy Filippov	a4a286ed95	Document NFS-RDMA	2024-12-19 14:05:12 +03:00
Vitaliy Filippov	b8009bad5e	Add librdmacm-dev to build dockerfile	2024-12-19 14:05:12 +03:00
Vitaliy Filippov	9be3d27dc9	Document VitastorFS-based CSI	2024-12-19 13:06:47 +03:00
Vitaliy Filippov	a19d2066c2	Document osd_backfillfull_ratio	2024-12-19 02:15:02 +03:00
Vitaliy Filippov	2a8780b4b5	Add a note about slow ops	2024-12-19 02:02:37 +03:00
Vitaliy Filippov	109f51a015	Implement basic VitastorFS support in CSI	2024-12-17 02:26:23 +03:00
Vitaliy Filippov	8a86c123c3	Allow to auto-select and print the port	2024-12-14 16:55:13 +03:00
Vitaliy Filippov	b856524e0c	Workaround for Linux bug: return post_op_attr for NFS-RDMA READ3 ... Linux NFS RDMA transport has a stupid bug - when the reply doesn't contain post_op_attr, the data gets offsetted by 84 bytes (size of attributes) and first 84 bytes are filled with probably random data.	2024-12-11 21:09:36 +03:00
Vitaliy Filippov	ae3ca7451f	Use per-connection RDMA device contexts	2024-12-11 21:09:36 +03:00
Vitaliy Filippov	1dbbb0c3f8	Implement NFS RDMA support	2024-12-11 21:09:36 +03:00
Vitaliy Filippov	64db31ec10	Fix slow op warning format	2024-12-11 21:09:36 +03:00
Vitaliy Filippov	76470686b3	Fix possibly incorrect linked list deserialization in NFS	2024-12-08 02:54:13 +03:00
Vitaliy Filippov	652ca631bb	Fix possible crash in nfs_block readdir	2024-12-01 18:04:49 +03:00
Vitaliy Filippov	2105f4b654	Add lost netlink daemonize	2024-11-27 17:13:30 +03:00
Vitaliy Filippov	0d01573da3	Fix typos	2024-11-26 14:31:47 +03:00
Vitaliy Filippov	d84b84f58d	Fix new backfillfull feature, add more logs	2024-11-23 01:08:13 +03:00
Vitaliy Filippov	8cfe705d7a	Map netlink after forking to show correct PID in vitastor-nbd ls	2024-11-23 00:46:44 +03:00
Vitaliy Filippov	66c9271cbd	Radically simplify create-pool pg_size check	2024-11-22 01:44:14 +03:00
Vitaliy Filippov	7b37ba921d	Pause pool rebalance when monitor detects that it can lead to any OSD becoming full	2024-11-22 01:01:07 +03:00
Vitaliy Filippov	262c581400	Fix create-pool for the case of hosts split into sub-nodes	2024-11-22 01:01:07 +03:00
Vitaliy Filippov	ad3b6b7267	Add a note about GID and RDMA device auto-selection	2024-11-21 23:54:05 +03:00
Vitaliy Filippov	1f6a061283	Move ibv_query_gid under #ifdef to only build it with libibverbs 32+	2024-11-21 23:47:57 +03:00
Vitaliy Filippov	fc4d97da10	Print "Waiting to become master" just once	2024-11-21 00:55:22 +03:00
Vitaliy Filippov	c7a4ce7341	Take out_size from dd oimg if not specified	2024-11-19 02:13:34 +03:00
Vitaliy Filippov	ddea31d86d	Auto-select first RDMA device only if RoCE is not found, add rocev2->rocev1->ib priority	2024-11-19 01:54:00 +03:00
Vitaliy Filippov	156d005412	Add serialize_overlap to test_heal	2024-11-17 01:26:35 +03:00
Vitaliy Filippov	7e076c7049	Do not report OSDs with empty statistics as full	2024-11-16 23:36:16 +03:00
Vitaliy Filippov	7de38250ad	Auto-select RDMA device based on osd_network	2024-11-16 18:38:57 +03:00
Vitaliy Filippov	9c59d30e83	Report slow ops in OSD stats in etcd and show them in vitastor-cli status	2024-11-16 15:11:16 +03:00
Vitaliy Filippov	5db02cdf6e	Add pve-qemu 9.0 patch	2024-11-16 11:20:47 +03:00
Vitaliy Filippov	8202ee9d74	Trigger double autosync when switching PG state to prevent leaving garbage in non-immediate_commit clusters	2024-11-15 01:26:36 +03:00
Vitaliy Filippov	5864bd067c	Add missing connection timeout for etcd websockets in OSD	2024-11-12 02:28:07 +03:00
Vitaliy Filippov	c312557ace	Do not execute remaining operations if the client is stopped during read	2024-11-10 16:44:13 +03:00
Vitaliy Filippov	5ce20116d8	Postpone trigger_nearest to prevent timer callbacks called from setTimer/clearTimer	2024-11-10 15:51:16 +03:00
Vitaliy Filippov	be66791e59	Add another note about 1.8 upgrade	2024-11-09 00:57:58 +03:00
Vitaliy Filippov	141cec2383	Add missing refcounting for flush_batch errors	2024-11-09 00:46:38 +03:00
Vitaliy Filippov	1ce4b1b417	Fix stop condition in osd_flush ... Could probably lead to PGs hung in peering states on OSD restart in EC pools, fixable by primary OSD restart	2024-11-08 00:30:40 +03:00
Vitaliy Filippov	ebf24bac9a	Fix partition zeroing during prepare ... Previously it zeroed area beginning with 0 instead of actual metadata offset which was leading to non-zeroed metadata when the disk is very small	2024-11-08 00:14:37 +03:00
Vitaliy Filippov	edd9051f81	Fix arch.en toc	2024-11-08 00:14:18 +03:00
Vitaliy Filippov	662ca86dc0	Fix libvirt 8 patch	2024-11-07 12:21:32 +03:00
Vitaliy Filippov	a1ca573168	Support QEMU 9.1	2024-11-07 12:21:13 +03:00
Vitaliy Filippov	f69f801ffb	Release 1.9.3 ... - Support custom hybrid OSD creation (`vitastor-disk prepare --hybrid --fast-devices /dev/xxx,/dev/yyy`) - Auto-change partition paths to /dev/disk/by-partuuid/ in `vitastor-disk prepare` - Allow to select cached I/O in vitastor-disk commands - Fix multiple bugs in vitastor-disk resize & add tests for them - Fix vitastor-disk write-meta/write-journal in superblock-based mode writing it to an incorrect device - Fix vitastor-disk prepare sometimes again not seeing new partitions - Cleanup PG history and stats of deleted pools - Fix "is already mounted" checks in CSI	2024-11-07 01:28:31 +03:00
Vitaliy Filippov	af92cbdfcc	Dynamic device size in test	2024-11-06 14:16:58 +03:00
Vitaliy Filippov	a775db10cc	Also allow cached I/O in dsk.open_*() in disk_tool	2024-11-06 13:52:25 +03:00
Vitaliy Filippov	eafce26049	Add resize and resize-auto tests	2024-11-06 13:30:51 +03:00
Vitaliy Filippov	625c74294f	Support direct I/O	2024-11-06 13:30:12 +03:00
Vitaliy Filippov	ef8c21ad6f	Change %lu to %ju	2024-11-06 02:58:51 +03:00
Vitaliy Filippov	2bb8e8999e	Do not check length in "data alignment mismatch"	2024-11-06 02:58:26 +03:00
Vitaliy Filippov	c2e7c28672	Fix calc_lengths data size recalc during auto-resize	2024-11-06 02:27:17 +03:00
Vitaliy Filippov	bd22beefb5	Auto-extend new_data_len if new_data_offset is changed too	2024-11-06 02:13:30 +03:00
Vitaliy Filippov	e7038ab99c	Auto-change partition paths to /dev/disk/by-partuuid/	2024-11-06 01:04:05 +03:00
Vitaliy Filippov	b6f75ebcfd	Add missing I/O path description in english	2024-11-06 00:43:17 +03:00
Vitaliy Filippov	9def199981	Auto-reduce new_data_len in resize	2024-11-05 02:57:11 +03:00
Vitaliy Filippov	c72e8e649e	Support test mode for vitastor-disk	2024-11-05 02:43:55 +03:00
Vitaliy Filippov	8bdb3e8786	Write meta/journal to correct device when used in superblock mode	2024-11-05 02:43:55 +03:00
Vitaliy Filippov	a87e236c70	Fix resize --data-size, particularly when expanding the device	2024-11-04 18:55:03 +03:00
Vitaliy Filippov	16f67cf6f1	Fix missing metadata checksums after resize	2024-11-04 18:36:35 +03:00
Vitaliy Filippov	56de4a520d	Support custom hybrid OSD creation (--hybrid --fast-devices /dev/xxx,/dev/yyy)	2024-11-04 17:52:29 +03:00
Vitaliy Filippov	adca162278	Note that osd_per_disk is also incompatible	2024-11-04 15:20:01 +03:00
Vitaliy Filippov	490b314d72	Rework & fix new partition waiting code	2024-11-04 15:16:30 +03:00
Vitaliy Filippov	9f52074e1e	Delete PG history and stats of deleted pools	2024-11-01 02:38:31 +03:00
Vitaliy Filippov	2b3e877546	Add notes about vitastor-disk in disable_data_fsync	2024-11-01 02:38:18 +03:00
Vitaliy Filippov	01d55e5420	Merge pull request #64 from 0x00ace/fio_version_fix ... use fio 3.35-1 for AlmaLinux 9	2024-10-31 11:55:40 +03:00
Vitaliy Filippov	f5aa5cfdfe	Fix "is already mounted" checks in CSI	2024-10-26 14:06:21 +03:00
Vitaliy Filippov	2826bb9e7e	Add more logging to CSI	2024-10-24 02:07:55 +03:00
Vitaliy Filippov	30d1ad0f66	Add Intel D5-P4320	2024-10-22 23:22:48 +03:00
Vitaliy Filippov	79719e44ac	Release 1.9.2 ... New features: - Support resizing normal vitastor-disk partitions and moving journal/metadata: [vitastor-disk resize](https://vitastor.io/docs/usage/disk.html#resize) - Support simple forms of vitastor-disk {dump,write}-{meta,journal} for OSD partitions Bug fixes: - Fix block RWX volumes broken after introducing stage/unstage support - Do not allow to create non-block RWX volumes in CSI - Fix vitastor-disk prepare not seeing the newly created partition in rare cases - Fix non-array tags not showing up in ls-osd/osd-tree - Make OpenNebula oned.conf patching during installation smarter - Fix iseek option in vitastor-cli dd not working - Validate conv=, iflag=, oflag= options in vitastor-cli dd - Fix vitastor-disk write-meta not writing header checksum to the disk - Fix JSON format in vitastor-disk dump-meta - Fix read_chain_bitmap not working for snapshot in another pool - Fix a possible OSD crash during parallel read & write to an image with snapshots - Several followups to the READ_CHAIN_BITMAP fix: avoid data reads, fix possible overflow in is_zero(), fix bitmap size	2024-10-20 01:49:13 +03:00
Vitaliy Filippov	f5626655df	Add new disk command docs	2024-10-20 01:47:46 +03:00
Vitaliy Filippov	7e2dde2702	Fix block RWX volumes broken after introducing stage/unstage support	2024-10-19 11:56:56 +03:00
Vitaliy Filippov	3b0ab317cf	Validate non-block RWX in CSI	2024-10-18 01:55:38 +03:00
Vitaliy Filippov	18eb99c494	Implement resizing partitions created with vitastor-disk	2024-10-18 01:55:19 +03:00
Vitaliy Filippov	4e8a1a8895	Run partprobe in add_partition() if /dev/disk/by-partuuid symlink is not present	2024-10-12 18:07:53 +03:00
Vitaliy Filippov	d27a8bdabc	Make get_parent_device return full path	2024-10-12 13:44:52 +03:00
Vitaliy Filippov	ebd616e42f	Extract clear_osd_superblock()	2024-10-12 13:44:52 +03:00
Vitaliy Filippov	b18d296e01	Extract check_existing_partition(), get_device_size()	2024-10-12 13:44:52 +03:00
Vitaliy Filippov	a03508320e	Move json_is_true/json_is_false to json_util.cpp	2024-10-12 00:40:39 +03:00
Vitaliy Filippov	c9ccc790ec	Fix non-array tags not showing up in ls-osd/osd-tree	2024-10-11 18:33:35 +03:00
Vitaliy Filippov	db2d9c5b3d	Fix tables in NFS doc	2024-10-08 00:20:10 +03:00
Vitaliy Filippov	09f15f44c9	Fix Toshiba MG and VDUSE Debian kernel note in docs	2024-10-08 00:17:14 +03:00
Vitaliy Filippov	c5a58c2e81	Support reading parameters automatically from the superblock in vitastor-disk {dump,write}-{meta,journal}	2024-10-07 02:21:58 +03:00
Vitaliy Filippov	30e7c2ad1e	Add custom OpenNebula oned.conf patcher (it uses a SHITTY configuration file format)	2024-10-06 13:46:05 +03:00
Vitaliy Filippov	2e76ceabbe	Fix iseek option in vitastor-cli dd	2024-10-05 18:25:38 +03:00
Vitaliy Filippov	3df088c207	Validate conv=, iflag=, oflag= options in vitastor-cli dd	2024-10-05 18:02:36 +03:00
Vitaliy Filippov	d882a19eab	Fix vitastor-disk write-meta not writing header checksum to the disk...	2024-10-05 17:32:55 +03:00
Vitaliy Filippov	702be3da7a	Fix JSON format in vitastor-disk dump-meta	2024-10-05 16:08:34 +03:00
Vitaliy Filippov	99533e1c2f	Fix .yml links	2024-10-02 00:38:07 +03:00
Vitaliy Filippov	a6cceb43bf	Fix read_chain_bitmap not working for snapshot in another pool	2024-10-02 00:24:48 +03:00
Vitaliy Filippov	745d89459a	Fix link, add title	2024-09-29 22:05:56 +03:00
Vitaliy Filippov	48f023292d	Fix extra data reads on read_chain	2024-09-21 17:05:42 +03:00
Vitaliy Filippov	b58bf3ada5	Fix possible OSD crash during parallel read & write to an image with snapshots ... OSDs could crash with the following "assertion failed" message (crash didn't affect data and was caused by OSD thinking upper blocks are full while they weren't). Reproduction without introducing artificial delays is hard because you have to force OSD to read an object with enqueued but not handled write which fills previously non-full bitmap. O_o. ``` vitastor-osd: ./src/osd/osd_primary_chain.cpp:613: void osd_t::send_chained_read_results(pg_t&, osd_op_t*): Assertion `stripes[role].read_buf' failed. ```	2024-09-21 13:44:36 +03:00
Vitaliy Filippov	f18a749324	READ_CHAIN fix was incomplete :-)	2024-09-21 13:40:31 +03:00
Vitaliy Filippov	6e9307c522	Fix possible overflow in is_zero()	2024-09-21 13:40:10 +03:00
Vitaliy Filippov	99adbb9483	Release 1.9.1 ... Hotfixes for OpenNebula and upgrade hotfix for 1.7 - Fix deploy.vitastor, save.vitastor, restore.vitastor scripts not working for nodes other than master oned - Fix deploy.vitastor not working for VMs without Vitastor disks - Disable clearing old PG configuration when upgrading from 1.7 or older versions (it was breaking old clients)	2024-09-14 19:17:30 +03:00
Vitaliy Filippov	b489a611a9	Add 1.8 upgrade note	2024-09-14 19:17:30 +03:00
Vitaliy Filippov	c6c0b8957a	Stop updating old PG configuration when the user manually deletes it	2024-09-14 19:15:40 +03:00
Vitaliy Filippov	5d40d2a459	Fix oned.conf patch	2024-09-14 19:08:44 +03:00
Vitaliy Filippov	f449c28c3b	Always write decoded base64 deployment file (otherwise it breaks VMs without Vitastor disks)	2024-09-14 15:25:02 +03:00
Vitaliy Filippov	a6274f58cc	Same fix for save/restore: they also need to ssh to target node	2024-09-14 02:46:48 +03:00
Vitaliy Filippov	ac29ffea6a	Add ssh to target node to deploy.vitastor - without it it always tried to deploy VMs on oned host	2024-09-14 02:15:24 +03:00
Vitaliy Filippov	bc06acc153	Disable clearing old PG configuration - we can not be sure that old clients do not need it	2024-09-13 19:00:12 +03:00
Vitaliy Filippov	fe8e611e23	Release 1.9.0 ... - OpenNebula support! [Installation instructions](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/installation/opennebula.en.md) - Added [vitastor-cli rm --exact|--matching](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/usage/cli.en.md#rm) command - Added [vitastor-cli dd](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/usage/cli.en.md#dd) command - copy files between Vitastor images, files and pipes - Add a startup timeout to vitastor-cli to not wait for etcd infinitely - Fix non-working OSD_OP_READ_CHAIN_BITMAP O_o - Autodetect block_size/bitmap_granularity/immediate_commit when creating pools - Do not allow to create multiple pools with the same name from vitastor-cli - Fix skip_cache_check option not applied due to type issue (see github issue #70)	2024-09-06 01:46:16 +03:00
Vitaliy Filippov	7636f9c726	Turn off brp-python-bytecompile in RPM specs	2024-09-06 01:44:44 +03:00
Vitaliy Filippov	d5f7005ddd	Add dd and rm --exact|--matching documentation	2024-09-05 02:22:05 +03:00
Vitaliy Filippov	70d6fcd32a	Add OpenNebula to README	2024-09-05 02:00:14 +03:00
Vitaliy Filippov	17caaa59af	vitastor-opennebula is probably more correct than opennebula-vitastor	2024-09-05 01:44:16 +03:00
Vitaliy Filippov	2dac6ee38b	Fix OpenNebula reinstall	2024-09-04 11:05:56 +03:00
Vitaliy Filippov	8be67a2d5b	Fix OpenNebula save/restore	2024-09-04 11:05:56 +03:00
Vitaliy Filippov	9c2132882c	Fix unaligned last block read/write in cli_dd	2024-09-04 11:05:56 +03:00
Vitaliy Filippov	9f25bb059b	Use just IMAGE_PREFIX, not IMAGE_PREFIX+"one"	2024-09-04 01:23:00 +03:00
Vitaliy Filippov	ee3094c5e5	Add OpenNebula plugin docs	2024-09-04 01:22:39 +03:00
Vitaliy Filippov	ba9f263b75	Add wildcard removal command	2024-08-31 14:13:09 +03:00
Vitaliy Filippov	30eaa1a8e6	Add vitastor-cli ls --exact	2024-08-31 02:36:25 +03:00
Vitaliy Filippov	6a8daedbe2	rm --wildcard	2024-08-31 02:36:25 +03:00
Vitaliy Filippov	2b96ac0b44	Implement OpenNebula driver	2024-08-30 23:46:37 +03:00
Vitaliy Filippov	986cd11705	Implement CLI "dd" command - copy data between Vitastor images, files and pipes	2024-08-30 02:31:06 +03:00
Vitaliy Filippov	b804051eaf	Remove debug print in nbd-proxy	2024-08-30 02:31:06 +03:00
Vitaliy Filippov	3cc326500e	Fix non-working OSD_OP_READ_CHAIN_BITMAP O_o	2024-08-30 01:25:05 +03:00
Vitaliy Filippov	f848c450a4	Clients should not wait infinitely for etcd to start if it's unavailable	2024-08-28 02:03:35 +03:00
Vitaliy Filippov	4121c66281	Autodetect block_size/bitmap_granularity/immediate_commit when creating pools	2024-08-28 02:03:35 +03:00
Vitaliy Filippov	b3716fbe23	Validate pool name when creating a pool	2024-08-28 02:03:35 +03:00
Vitaliy Filippov	97f49d7d94	Fix #70 from github - skip_cache_check type issue	2024-08-14 01:35:43 +03:00
Vitaliy Filippov	131de4b790	Disable trace in header	2024-08-13 11:21:35 +03:00
Vitaliy Filippov	ce359c5a69	Release 1.8.0 ... Bugfix release, would be 1.7.2, but etcd layout changes mandate it to be 1.8.0. :-) - Change etcd layout: /config/pgs is now /pg/config, /pg/stats/* is now /pgstats/*. This is required to fix a rare PG history tracking issue caused by non-atomic delivery of etcd events sometimes resulting in `incomplete` objects in EC pools after mass OSD restarts. Upgrading can be performed freely, downgrade requires additional action: [1.8.0 to 1.7.1](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/usage/admin.en.md#1-8-0-to-1-7-1) - Fix a rare client hang on PG primary OSD switch - Fix vitastor-nfs started using mount command sometimes not stopping automatically after unmount - Fix vitastor-nfs mounts started using mount command sometimes hanging after daemonizing - Fix merge/flatten into a pool with different object size (image migration between pools case) - Do not print extra "PG disappeared after reload" verbose log messages for non-existing PGs - Fix clustered Antietcd support and persistence filter - Do not try to purge the same OSD multiple times if its multiple devices are passed to purge - Various node.js binding fixes	2024-08-11 14:28:31 +03:00
Vitaliy Filippov	521e867b10	Run check_exit also on deferred stop. Now vitastor-nfs should finally always stop on umount	2024-08-11 00:05:20 +03:00
Vitaliy Filippov	333c54ebbf	Cleanup clients correctly during stop(). Was also affecting #67, but could also reproduce during normal operation	2024-08-11 00:00:13 +03:00
Vitaliy Filippov	58d3da95c8	Fix github issue #67 by closing active NFS sockets before daemonize()	2024-08-10 20:13:37 +03:00
Vitaliy Filippov	4e90e752eb	Fix merge/flatten into a pool with different object size	2024-08-10 19:23:26 +03:00
Vitaliy Filippov	09342d7189	node.js binding fixes	2024-08-05 00:10:37 +03:00
Vitaliy Filippov	eb3e8b8c19	Do not print "PG disappeared after reload" verbose log messages when *it* was not present	2024-08-04 01:42:05 +03:00
Vitaliy Filippov	e2ca3ad99e	Add a note about storage ID in proxmox storage config doc	2024-07-31 01:19:44 +03:00
Vitaliy Filippov	dd4b0aed2b	Support scattered write in node.js binding	2024-07-31 01:17:06 +03:00
Vitaliy Filippov	42851a061c	Always continue operations to not miss resuming after the lack of PG primary ... Should fix spurious client hangs during PG primary switchover	2024-07-31 01:17:03 +03:00
Vitaliy Filippov	8e0f242d30	Add downgrade docs	2024-07-31 01:15:37 +03:00
Vitaliy Filippov	0daa8ea39b	Support seamless upgrade to new PG config and stats etcd key names	2024-07-31 01:15:37 +03:00
Vitaliy Filippov	b263d311ef	Use separate watch revisions for different watchers	2024-07-31 01:15:37 +03:00
Vitaliy Filippov	8720185780	Run tests in CI in memory (in tmpfs)	2024-07-31 01:15:37 +03:00
Vitaliy Filippov	20584414d8	Report OSD version in /osd/state/ and /osd/stats/ (for the future)	2024-07-31 01:15:37 +03:00
Vitaliy Filippov	306a3db7f3	Rename VERSION define to VITASTOR_VERSION	2024-07-31 01:15:37 +03:00
Vitaliy Filippov	5b0aebada4	Rename /config/pgs to /pg/config and /pg/stats/* to /pgstats/*	2024-07-31 01:15:37 +03:00
Vitaliy Filippov	d6f0b480c8	Fix broken link	2024-07-22 14:01:53 +03:00
Vitaliy Filippov	f1f8531fd4	Make tests compatible with antietcd, add 2 antietcd tests to CI	2024-07-20 02:16:38 +03:00
Vitaliy Filippov	8d79d59964	Update antietcd to 1.1.0	2024-07-20 02:15:48 +03:00
Vitaliy Filippov	551a209a50	Fix persistence filter initialization	2024-07-20 02:15:48 +03:00
Vitaliy Filippov	06cafd7702	Do not merge config an extra unneeded time	2024-07-20 02:15:48 +03:00
Vitaliy Filippov	3018352443	Fix clustered Antietcd support	2024-07-19 18:58:58 +03:00
Vitaliy Filippov	f8edfb4a71	No need to check for PG intersection if a history set is smaller than EC data part count	2024-07-18 19:29:05 +03:00
Vitaliy Filippov	8239ea2356	Do not try to purge the same OSD multiple times if its multiple devices are passed to purge	2024-07-16 16:48:16 +03:00
Vitaliy Filippov	e898335b8d	Release 1.7.1 ... Some stupid hotfixes for 1.7.0 :) - Fix NFS mount - Fix modify-osd - Fix use_antietcd not taken from /etc	2024-07-16 00:07:03 +03:00
Vitaliy Filippov	e7869611fa	Another stupid fix for NFS (no idea how it worked for me)	2024-07-16 00:05:51 +03:00
Vitaliy Filippov	e1c2500b60	Use modify-osd in the disk removal instruction	2024-07-16 00:01:42 +03:00
Vitaliy Filippov	42cf3a11df	Oops, fix reweight :)	2024-07-16 00:01:11 +03:00
Vitaliy Filippov	4d9293f0e9	Fix QEMU 8.2 and 9.0 patches (add @location comments)	2024-07-15 16:30:14 +03:00
Vitaliy Filippov	7a13f85ae2	Fix mon config merge	2024-07-15 16:25:22 +03:00
Vitaliy Filippov	fc219b8602	Add pg-list to docs	2024-07-15 13:29:22 +03:00
Vitaliy Filippov	989d73f874	Release 1.7.0 ... Omnidirectional release New features: - Support handling TCP I/O in simple separate io_uring-based [I/O threads](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/config/client.en.md#client_iothread_count) - may increase linear performance to 7-8 GB/s - Experimental internal etcd replacement - [antietcd](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/config/monitor.en.md#use_antietcd) - Monitor now has a [built-in Prometheus exporter](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/config/monitor.en.md#enable_prometheus) - Added a reference [Grafana dashboard](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/mon/scripts/Vitastor-Grafana-6+.json) - Implement vitastor-cli [osd-tree](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/usage/cli.en.md#osd-tree) and [ls-osd](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/usage/cli.en.md#ls-osd) commands - Implement vitastor-cli [modify-osd](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/usage/cli.en.md#modify-osd) command - Implement vitastor-cli [pg-list](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/usage/cli.en.md#pg-list) command - Implement [VitastorFS defragmentation](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/usage/nfs.en.md#defrag) - Implement basic node.js binding (not published on npm yet) Changes: - Make immediate_commit=all the default everywhere to match default vitastor-disk behaviour - Make pool-create error message more obvious and add details to it - Set default etcd_ws_keepalive_interval to 5 seconds (speedup client etcd failover) - Support OpenStack 2023.2 in Nova and Cinder drivers/patches - Add patches for libvirt 10.x - Add patches for QEMU 8.2 and 9.0 - Implement internal restart / run_forever in monitor - Some source tree refactoring - sources are now moved into subdirectories, monitor is now split into multiple files - Add vitastor_c_inode_get_immediate_commit in vitastor_c client library - Make vitastor_kv.h header public Bug fixes: - Fix total statistics usec/count/bytes not being reported when delta (bps/iops/lat) is zero - Prevent infinite loop in NFS on files with incorrect metadata pointing to an empty volume - Fix READDIR offsets (cookies) in VitastorFS sometimes leading to client infinite loops when reading a directory - Fix a rare infinite loop during OSD journal flushing (OSD hanging and eating 100 % CPU) - Fix several bugs which could lead to lost writes in setups without immediate_commit: - Client library treated writes as completed before actually completing them, thus missing them in a subsequent fsync - Client library didn't repeat writes on the new PG primary when it changed - OSDs didn't drop peer connections with dirty writes when stopping PG - Fix Block Pseudo-FS initialization leading to ENOENTs some time after start - Fix vitastor-cli merge-based commands (merge/flatten/rm snapshot) slowing down and finally failing when using CAS optimistic locks - Fix pool create/modify --block_size validation - Fix TTL comparison for determining failed lease/keepalive requests in OSD - Add support for size suffixes in pool-create --block_size and --immediate_commit values	2024-07-15 11:48:35 +03:00
Vitaliy Filippov	f0630722ce	Make pool-create error message more obvious, add details	2024-07-15 11:47:49 +03:00
Vitaliy Filippov	93b0947720	Support size suffixes in pool-create --block_size / --bitmap_granularity	2024-07-15 11:47:05 +03:00
Vitaliy Filippov	9c628646fa	Remove bullseye-backports from build, remove buster-backports from docs	2024-07-15 11:47:05 +03:00
Vitaliy Filippov	cf476a3b95	Add mkdir /var/lib/vitastor	2024-07-15 11:47:05 +03:00
Vitaliy Filippov	23f9273ba3	Take use_antietcd setting from /etc/vitastor/vitastor.conf too	2024-07-15 02:02:56 +03:00
Vitaliy Filippov	74b88bf8ba	Use own repo instead of buster-backports as it is EOL	2024-07-14 20:25:44 +03:00
Vitaliy Filippov	1254d5a0de	Fix delta stats when counters may be hypothetically reset	2024-07-14 13:11:00 +03:00
Vitaliy Filippov	f87bece253	Fix build with antietcd & tinyraft, remove some version hardcode	2024-07-14 13:04:25 +03:00
Vitaliy Filippov	ba85d0ef16	Add vitastor_kv.h to RPM specs	2024-07-14 11:20:37 +03:00
Vitaliy Filippov	17a909ea3a	Stop metrics/future API HTTP server when closing Monitor instance	2024-07-14 11:16:41 +03:00
Vitaliy Filippov	a4dfc220ab	Implement basic node.js binding (not published on npm yet)	2024-07-14 10:58:38 +03:00
Vitaliy Filippov	26426dd95e	Return it back, but fix stats in another way	2024-07-13 19:14:34 +03:00
Vitaliy Filippov	9f38b7e5c1	Fix osd_ping_time_remaining reset from 990c3ba7eb, leading to osd disconnections	2024-07-13 16:09:56 +03:00
Vitaliy Filippov	20057defbe	Revert 8ad63465cd	2024-07-13 15:34:34 +03:00
Vitaliy Filippov	b4e9140755	Add defrag docs, fix trace message	2024-07-13 00:45:53 +03:00
Vitaliy Filippov	413959e75a	Prevent infinite loop in NFS - return EIO when an inode points to an incorrect volume position	2024-07-12 20:53:54 +03:00
Vitaliy Filippov	8973982570	Delete keys from internal state instead of setting them to null on DELETE event in mon	2024-07-12 16:42:21 +03:00
Vitaliy Filippov	990c3ba7eb	Implement FS defragmentation	2024-07-12 16:11:35 +03:00
Vitaliy Filippov	1771d2ef36	Fix READDIR cookie/offset bug	2024-07-12 16:11:35 +03:00
Vitaliy Filippov	d88ab76636	Fix active mon stat	2024-07-11 01:34:59 +03:00
Vitaliy Filippov	c010a0aa54	Fix OSD "local write" latency sum	2024-07-11 01:30:03 +03:00
Vitaliy Filippov	0d42712d29	Fix refresh in dashboard variable	2024-07-11 01:13:02 +03:00
Vitaliy Filippov	66b438106a	Add vitastor-cli pg-list command	2024-07-10 02:27:41 +03:00
Vitaliy Filippov	3aef6682fb	Add vitastor-cli modify-osd command	2024-07-09 16:52:19 +03:00
Vitaliy Filippov	8535bccf4c	Add a note about antietcd dump/load	2024-07-09 15:58:03 +03:00
Vitaliy Filippov	0487b3b239	Add clusterid to Grafana dashboard	2024-07-09 15:58:03 +03:00
Vitaliy Filippov	a54ef97f5d	Add Grafana dashboard link	2024-07-09 15:37:25 +03:00
Vitaliy Filippov	10434a9b2b	Add notes about antietcd to documentation	2024-07-09 15:01:41 +03:00
Vitaliy Filippov	c6be194508	Implement experimental antietcd-based version of monitor	2024-07-09 13:54:58 +03:00
Vitaliy Filippov	df668286fb	Add Grafana dashboard	2024-07-09 02:39:36 +03:00
Vitaliy Filippov	667c5999c9	Report all PG states	2024-07-08 19:52:56 +03:00
Vitaliy Filippov	8ad63465cd	Do not wipe previous metrics at moments when difference is 0	2024-07-08 02:20:12 +03:00
Vitaliy Filippov	976290e6a9	Implement built-in Prometheus exporter in monitor	2024-07-08 02:20:12 +03:00
Vitaliy Filippov	79f1d1969b	Make immediate_commit=all the default	2024-07-07 11:45:18 +03:00
Vitaliy Filippov	918e1f83b0	Add JSON output for ls-osd	2024-07-07 02:24:36 +03:00
Vitaliy Filippov	abbba6ade4	Support handling TCP I/O in simple separate io_uring-based I/O threads ... Required mainly for clients, allows to scale parallel client I/O with TCP from 100-150k iops to ~400k iops and from 2-3 GB/s to at least 7-8 GB/s with 4 I/O threads, at the same time increasing Q=1 latency by 2x thread switching delay, which is ~10 us when CPU powersaving is disabled and may be as high as 200 us when it's enabled.	2024-07-04 13:29:20 +03:00
Vitaliy Filippov	21d1171ba4	Fix parsing after "slightly decopypasting" :)	2024-06-29 00:09:30 +03:00
ace	8f83086889	Nova and cinder driver patches for OpenStack 2023.2	2024-06-28 00:04:57 +03:00
Vitaliy Filippov	ceb18f25db	Add libvirt 10.0 patch (same as 9.10 and 10.4 actually)	2024-06-28 00:03:46 +03:00
Vitaliy Filippov	ed51a89f70	Add QEMU 8.2 and 9.0 patches	2024-06-27 12:33:16 +03:00
Vitaliy Filippov	f59456f22d	Add libvirt 10.4 patch (same as 9.10 actually)	2024-06-27 01:35:29 +03:00
Vitaliy Filippov	ca63cd507d	Fix possible infinite loop in flusher (surprisingly reproduced in test_write.sh with iothreads)	2024-06-27 00:38:01 +03:00
Vitaliy Filippov	ea0d72289c	Treat copied buffers as written only after completing the write in client ... SYNC operation fsyncs only completed operations, so treating writes as "eligible for fsync" before actually completing them is incorrect It affected SCHEME=ec test_heal.sh (with immediate_commit=none) test - it was flapping with lost writes - some non-fsynced writes were legitimately lost by the OSD, but weren't repeated by the client	2024-06-20 02:11:53 +03:00
Vitaliy Filippov	e400a851f4	Repeat dirty buffer flushes on any PG primary change because the new primary may not know about unfinished operations of the old primary	2024-06-19 00:28:26 +03:00
Vitaliy Filippov	0fec7a9fea	Drop dirty peer connections also when stopping PG to guarantee that clients do not miss fsync	2024-06-19 00:28:26 +03:00
Vitaliy Filippov	b9de2a92a9	Print OSD performance stats	2024-06-17 13:02:58 +03:00
Vitaliy Filippov	5360a70853	Make OSD also report derived stats	2024-06-17 13:02:52 +03:00
Vitaliy Filippov	4c2328eb13	Implement ls-osd command	2024-06-17 02:22:14 +03:00
Vitaliy Filippov	313daef12d	Slightly decopypaste etcd key parsing	2024-06-17 01:38:42 +03:00
Vitaliy Filippov	ad9c12e1b9	Fix Pseudo-FS initialization leading to ENOENTs some time after start	2024-06-16 23:43:09 +03:00
Vitaliy Filippov	4473eb5512	Fix slow & failing CAS layer merge	2024-06-14 02:15:49 +03:00
Vitaliy Filippov	6501abc060	Set default etcd_ws_keepalive_interval to 5	2024-06-08 00:38:48 +03:00
Vitaliy Filippov	1228403e74	Implement internal restart / run_forever in monitor	2024-06-08 00:35:18 +03:00
Vitaliy Filippov	4eabebd245	Put all configuration to Mon.config	2024-06-07 00:20:38 +03:00
Vitaliy Filippov	cf60b6818c	Extract PG generation into pg_gen.js	2024-06-05 11:22:06 +03:00
Vitaliy Filippov	1a4a7cdc37	Extract OSD Tree generation functions to osd_tree.js	2024-06-05 11:19:35 +03:00
Vitaliy Filippov	1b48085e21	Extract remote etcd interaction to etcd_adapter.js	2024-06-05 11:19:35 +03:00
Vitaliy Filippov	a71847244e	Rename PGUtil.js to pg_utils.js	2024-06-05 10:51:20 +03:00
Vitaliy Filippov	848c2d2722	Move LPOptimizer, DSL and tests to lp_optimizer/	2024-06-05 10:51:20 +03:00
Vitaliy Filippov	86832dc43f	Add eslint import/no-unresolved	2024-06-05 10:51:20 +03:00
Vitaliy Filippov	1f6da79463	Extract stats calculation into a separate file	2024-06-05 10:51:20 +03:00
Vitaliy Filippov	9bf57c3760	Mention generic Toshiba MG instead of specific MGxx, fix russian vitastorfs link	2024-06-05 02:08:09 +03:00
Vitaliy Filippov	a0305b5b4a	Extract pool configuration validation into a separate file	2024-06-05 02:08:08 +03:00
Vitaliy Filippov	1546f8e447	Extract etcd data "schema" into a separate file	2024-06-05 02:07:53 +03:00
Vitaliy Filippov	8ce962b312	Move scripts	2024-06-05 02:07:53 +03:00
Vitaliy Filippov	50e56b3b92	Add vitastor_c_inode_get_immediate_commit	2024-05-19 01:57:18 +03:00
ace	b85dab8583	use fio 3.35-1 for AlmaLinux 9	2024-05-18 21:17:16 +03:00
Vitaliy Filippov	a12d328793	Rename cli/ to cmd/, fix cmake install	2024-05-15 23:04:50 +03:00
Vitaliy Filippov	c79b38bd26	Move all sources to subdirs	2024-05-15 11:06:01 +03:00
Vitaliy Filippov	44692d148a	Make vitastor_kv.h header public	2024-05-15 01:49:38 +03:00
Vitaliy Filippov	ba52359611	Fix last master commit	2024-05-15 01:49:31 +03:00
Vitaliy Filippov	23a9aa93b5	Fix pool create/modify --block_size validation	2024-05-04 16:33:22 +03:00
Vitaliy Filippov	2412d9e239	Fix TTL comparison for lease/keepalive	2024-04-30 01:53:05 +03:00
Vitaliy Filippov	9301c857b1	Release 1.6.1 ... A bunch of monitor fixes - Add noout flag for OSDs (/vitastor/config/osd/xx) - Fix "effective" size of degraded PGs (and thus "used space") calculation in monitor - Fix monitor not clearing PGs of deleted pools - Fix incorrect PG generation with hosts with 0 OSDs - Fix monitor crashing during primary OSD recheck when pool has no PGs - Fix monitor crashing when node_placement included non-existing OSDs - Fix possible data movement after removing OSDs reweighted to 0 - Remove extra empty keys from pool configurations created by vitastor-cli create-pool - Fix 32-bit build	2024-04-22 02:01:29 +03:00
Vitaliy Filippov	3094358ec2	Fix autovivification leading to extra empty keys in pool-create	2024-04-20 02:04:09 +03:00
Vitaliy Filippov	87f666d2a2	Filter out OSDs reweighted to 0	2024-04-20 02:03:53 +03:00
Vitaliy Filippov	bd7fe4ef8f	Filter out non-existing OSDs added in node_placement	2024-04-20 02:03:36 +03:00
Vitaliy Filippov	1b3f9a1416	Do not set non-existing OSD weight to 0, we'll remove them instead	2024-04-20 02:03:11 +03:00
Vitaliy Filippov	a7b7354f38	Do not recheck primary distribution when pool has no PGs	2024-04-20 02:02:47 +03:00
Vitaliy Filippov	765befa22f	Remove empty nodes from tree because PG DSL expects that all leaf nodes are OSDs	2024-04-20 02:02:28 +03:00
Vitaliy Filippov	87b3ab94fe	Do not disable require-atomic-updates and no-unused-vars	2024-04-20 02:02:13 +03:00
Vitaliy Filippov	2c0801f6e4	Configure ESLint and add it to CI	2024-04-16 02:39:31 +03:00
Vitaliy Filippov	fd83fef1d9	Fix pool deletion	2024-04-16 02:20:26 +03:00
Vitaliy Filippov	8d1067971b	Fix pg_effsize (and thus "used space") calculation in monitor	2024-04-16 02:20:18 +03:00
Vitaliy Filippov	ae5af04fde	Add noout flag for OSDs	2024-04-16 02:19:55 +03:00
Vitaliy Filippov	266d038b11	Fix 32-bit build warnings and one error again :-)	2024-04-11 22:49:33 +03:00
Vitaliy Filippov	ff4414d37e	Release 1.6.0 ... New features: - Implement "hierarchical failure domains" and other complex distribution rules, for example EC 4+2 over 3 DC, with 2 chunks per each DC ([documentation](docs/config/pool.en.md#level_placement)) - Make OSDs handle ENOSPC - now cluster stays online even if some OSDs fill up to 100 %, only writes requiring free space hang - Implement Stage/Unstage & volume locking for CSI to prevent parallel mounting and/or modifications of the same volume - Warn about full and almost full OSDs in vitastor-cli status - Add an experimental NBD netlink map mode as an option ([documentation](docs/usage/nbd.en.md)) - Add --pg parameter to vitastor-cli describe, print objects with 0x in human-readable format too - Add [administration docs](docs/usage/admin.en.md) Bug fixes: - Fix client operation retry timeout - previously the timeout wasn't applied and writes were retries almost instantly - Fix monitors crashing on invalid pool configurations - Fix journaling - make each journal write wait for all previous journal writes - Fix monitor thinking that OSD weight is 0 after deleting /osd/config/ key online - Fix a write stall caused by flusher possibly not trimming journal on rollback - Set 32k csum_block_size for HDD by default in vitastor-disk	2024-04-09 16:57:59 +03:00
Vitaliy Filippov	0fa7ecc03f	Add also a test for OSD tags	2024-04-09 16:57:59 +03:00
Vitaliy Filippov	c29bfe12eb	Oops - fix filter_by_root_node, add a test for it	2024-04-09 15:48:44 +03:00
Vitaliy Filippov	57bf84ddb2	Fix filtering in mon	2024-04-09 14:51:05 +03:00
Vitaliy Filippov	dff4879c8c	Check if NBD_ATTR_BACKEND_IDENTIFIER is defined	2024-04-09 13:16:58 +03:00
Vitaliy Filippov	af9a853db6	Move NBD netlink map&unmap to separate commands, add "netlink-revive" command	2024-04-08 16:34:41 +03:00
Vitaliy Filippov	b7a3275af3	Make netlink optional	2024-04-08 01:51:28 +03:00
Vitaliy Filippov	64c5c4ca26	Fix code style	2024-04-08 01:35:03 +03:00
idelson	442a9d838d	nbd-proxy: add configuration via netlink to support kinds of timeouts. ... PR #58 - https://github.com/vitalif/vitastor/pull/58/commits By MIND Software LLC By submitting this pull request, I accept Vitastor CLA	2024-04-08 00:50:08 +03:00
Vitaliy Filippov	6366972fe8	Warn about full and almost full OSDs in status