211 Commits (1228403e74d026d59755069d376b4d1c81dddaba)

Author	SHA1	Message	Date
Vitaliy Filippov	1228403e74	Implement internal restart / run_forever in monitor	2024-06-08 00:35:18 +03:00
Vitaliy Filippov	4eabebd245	Put all configuration to Mon.config	2024-06-07 00:20:38 +03:00
Vitaliy Filippov	cf60b6818c	Extract PG generation into pg_gen.js	2024-06-05 11:22:06 +03:00
Vitaliy Filippov	1a4a7cdc37	Extract OSD Tree generation functions to osd_tree.js	2024-06-05 11:19:35 +03:00
Vitaliy Filippov	1b48085e21	Extract remote etcd interaction to etcd_adapter.js	2024-06-05 11:19:35 +03:00
Vitaliy Filippov	a71847244e	Rename PGUtil.js to pg_utils.js	2024-06-05 10:51:20 +03:00
Vitaliy Filippov	848c2d2722	Move LPOptimizer, DSL and tests to lp_optimizer/	2024-06-05 10:51:20 +03:00
Vitaliy Filippov	86832dc43f	Add eslint import/no-unresolved	2024-06-05 10:51:20 +03:00
Vitaliy Filippov	1f6da79463	Extract stats calculation into a separate file	2024-06-05 10:51:20 +03:00
Vitaliy Filippov	a0305b5b4a	Extract pool configuration validation into a separate file	2024-06-05 02:08:08 +03:00
Vitaliy Filippov	1546f8e447	Extract etcd data "schema" into a separate file	2024-06-05 02:07:53 +03:00
Vitaliy Filippov	8ce962b312	Move scripts	2024-06-05 02:07:53 +03:00
Vitaliy Filippov	23a9aa93b5	Fix pool create/modify --block_size validation	2024-05-04 16:33:22 +03:00
Vitaliy Filippov	9301c857b1	Release 1.6.1 ... A bunch of monitor fixes - Add noout flag for OSDs (/vitastor/config/osd/xx) - Fix "effective" size of degraded PGs (and thus "used space") calculation in monitor - Fix monitor not clearing PGs of deleted pools - Fix incorrect PG generation with hosts with 0 OSDs - Fix monitor crashing during primary OSD recheck when pool has no PGs - Fix monitor crashing when node_placement included non-existing OSDs - Fix possible data movement after removing OSDs reweighted to 0 - Remove extra empty keys from pool configurations created by vitastor-cli create-pool - Fix 32-bit build	2024-04-22 02:01:29 +03:00
Vitaliy Filippov	87f666d2a2	Filter out OSDs reweighted to 0	2024-04-20 02:03:53 +03:00
Vitaliy Filippov	bd7fe4ef8f	Filter out non-existing OSDs added in node_placement	2024-04-20 02:03:36 +03:00
Vitaliy Filippov	1b3f9a1416	Do not set non-existing OSD weight to 0, we'll remove them instead	2024-04-20 02:03:11 +03:00
Vitaliy Filippov	a7b7354f38	Do not recheck primary distribution when pool has no PGs	2024-04-20 02:02:47 +03:00
Vitaliy Filippov	765befa22f	Remove empty nodes from tree because PG DSL expects that all leaf nodes are OSDs	2024-04-20 02:02:28 +03:00
Vitaliy Filippov	87b3ab94fe	Do not disable require-atomic-updates and no-unused-vars	2024-04-20 02:02:13 +03:00
Vitaliy Filippov	2c0801f6e4	Configure ESLint and add it to CI	2024-04-16 02:39:31 +03:00
Vitaliy Filippov	fd83fef1d9	Fix pool deletion	2024-04-16 02:20:26 +03:00
Vitaliy Filippov	8d1067971b	Fix pg_effsize (and thus "used space") calculation in monitor	2024-04-16 02:20:18 +03:00
Vitaliy Filippov	ae5af04fde	Add noout flag for OSDs	2024-04-16 02:19:55 +03:00
Vitaliy Filippov	ff4414d37e	Release 1.6.0 ... New features: - Implement "hierarchical failure domains" and other complex distribution rules, for example EC 4+2 over 3 DC, with 2 chunks per each DC ([documentation](docs/config/pool.en.md#level_placement)) - Make OSDs handle ENOSPC - now cluster stays online even if some OSDs fill up to 100 %, only writes requiring free space hang - Implement Stage/Unstage & volume locking for CSI to prevent parallel mounting and/or modifications of the same volume - Warn about full and almost full OSDs in vitastor-cli status - Add an experimental NBD netlink map mode as an option ([documentation](docs/usage/nbd.en.md)) - Add --pg parameter to vitastor-cli describe, print objects with 0x in human-readable format too - Add [administration docs](docs/usage/admin.en.md) Bug fixes: - Fix client operation retry timeout - previously the timeout wasn't applied and writes were retries almost instantly - Fix monitors crashing on invalid pool configurations - Fix journaling - make each journal write wait for all previous journal writes - Fix monitor thinking that OSD weight is 0 after deleting /osd/config/ key online - Fix a write stall caused by flusher possibly not trimming journal on rollback - Set 32k csum_block_size for HDD by default in vitastor-disk	2024-04-09 16:57:59 +03:00
Vitaliy Filippov	0fa7ecc03f	Add also a test for OSD tags	2024-04-09 16:57:59 +03:00
Vitaliy Filippov	c29bfe12eb	Oops - fix filter_by_root_node, add a test for it	2024-04-09 15:48:44 +03:00
Vitaliy Filippov	57bf84ddb2	Fix filtering in mon	2024-04-09 14:51:05 +03:00
Vitaliy Filippov	6366972fe8	Warn about full and almost full OSDs in status	2024-04-07 19:39:51 +03:00
Vitaliy Filippov	2b863fb715	Add ENOSPC handling tests	2024-04-07 19:39:33 +03:00
Vitaliy Filippov	3629dbc54d	Plug the new PG combinator into monitor	2024-04-07 02:44:17 +03:00
Vitaliy Filippov	29284bef40	Implement new DSL/rule-based PG generation algorithm	2024-04-07 00:36:20 +03:00
Vitaliy Filippov	6a924d6066	Extract PG combinator into a separate module	2024-04-07 00:36:20 +03:00
Vitaliy Filippov	9fe779a691	Do not die on invalid pool configurations	2024-04-07 00:36:20 +03:00
Vitaliy Filippov	f36d7eb76c	Fix monitor thinking that OSD weight is 0 after deleting /osd/config/ key	2024-04-05 23:14:46 +03:00
Vitaliy Filippov	9ad6822353	Release 1.5.0 ... After half a year of hard work, VitastorFS is finally here ! :-) New features: - VitastorFS, a full-featured clustered (read-write-many) file system. Documentation: [VitastorFS](docs/usage/nfs.en.md) - Embedded key-value database implementation based on Parallel Optimistic B-Tree algorithm and used for the metadata of VitastorFS - Pool management commands in vitastor-cli (create-pool, list-pools, rm-pool, modify-pool). Thanks MIND Software (https://mindsw.io) for their contribution! [Documentation](docs/usage/cli.en.md#create-pool) Bug fixes: - Fix a very rare "infinite loop" in the client library - Fix a rare OSD hang on during start when zeroing out bad metadata entries left from the previous run	2024-03-16 15:35:10 +03:00
Vitaliy Filippov	8840c84572	Fix "bad key in etcd" in mon for FS pools	2024-03-16 13:24:36 +03:00
Vitaliy Filippov	3aee37eadd	Allow to disable per-inode stats for VitastorFS pools	2024-03-16 13:24:36 +03:00
Vitaliy Filippov	38b8963330	Release 1.4.8 ... - Do not use \r if output is not a terminal (should fix unexpected job output in proxmox) - Fix rm/rm-data error return code, add --down-ok option to bypass the error - Add EIO retry timeout and allow to disable these retries, rename up_wait_retry_interval to client_retry_interval - Add ubuntu jammy build - Wait for blockstore initialisation before starting OSD (prevent timeouts when init takes time) - Fix a rare use-after-free in automatic sync after delete in blockstore	2024-02-29 09:58:34 +03:00
Vitaliy Filippov	6bf1f539a6	Add EIO retry timeout and allow to disable these retries, rename up_wait_retry_interval to client_retry_interval	2024-02-28 13:10:02 +03:00
Vitaliy Filippov	5e934264cf	Release 1.4.7 ... - Fix another old "BUG: Attempt to overwrite used offset" in a very simple case: bs=4k rw=write iodepth=16 from OSD start; add this case to tests - Fix a rare crash with "unexpected state during flush: 0x51" possible with EC since 1.4.2 during rebalance and OSD outages - Fix a rare write stall with EC & immediate_commit=none caused by sync operations reserving unneeded space in the journal - Fix 32-bit build warnings, most in printf/scanf format strings	2024-02-22 12:45:52 +03:00
Vitaliy Filippov	f87964861d	Release 1.4.6 ... Unwavering stabilization of 1.4.x, continued :-) - Include the accidentally lost part of 1.4.5 journal trimming fix - Fix a possible OSD crash with "BUG: Attempt to overwrite used offset" which was probably present for long time, but became apparent after fixing flapping tests in CI - Fix remaining flapping tests in CI. It was the first time when tests actually passed without retries :-)	2024-02-20 17:01:26 +03:00
Vitaliy Filippov	f882c7dd87	Release 1.4.5 ... - Fix a write stall caused by incorrect journal trimming introduced in 1.4.4 :) - Fix PGs sometimes hanging in "starting" state on mass OSD restarts - Fix a rare crash with "map::at" during OSD pings - Use new defaults for non-capacitor (desktop) SSDs - improves T1Q256 random write from ~6k iops to ~45k iops - Make journal_trim_interval configurable	2024-02-16 10:13:33 +03:00
Vitaliy Filippov	c777a0041a	Release 1.4.4 ... A couple of fixes for EC pools - Fix a segfault possible on partial EC overwrite in 1234 -> 5030 rebalance scenario - Fix two problems leading to EC pools stalling on rebalance & parallel sudden stops of OSDs, for example during a sudden poweroff of a host: - Recovery auto-tuning (1.4.0 feature) could apply too large delays and stall the EC journal - fixed by limiting delays with a new recovery_tune_sleep_cutoff_us parameter (10 seconds by default) and applying recovery pauses before write operations, not after them, to not occupy space in the journal for long time - Dynamic journal space reservation (1.3.0 feature) wasn't accounting new writes when checking the limit so OSDs could still fill the journal fully and stall - fixed by including new writes into the limit - Print etcd dbSize instead of dbSizeInUse in status	2024-02-11 16:23:08 +03:00
Vitaliy Filippov	27e9f244ec	Release 1.4.3 ... Hotfix for hotfix O:-) - "Write stall fix" was incomplete and EC write stalls could continue even on 1.4.2. Now they're finally fixed O:-) - Make monitor ignore statistics of stopped OSDs. Previously if you stopped all OSDs the last total I/O numbers would remain the same indefinitely	2024-02-09 00:29:31 +03:00
Vitaliy Filippov	8e25a28a08	Ignore down OSDs in monitor statistics aggregation	2024-02-09 00:22:36 +03:00
Vitaliy Filippov	016115c0d4	Release 1.4.2 ... - Log to systemd by default - Fix excessive autosyncs after every operation with disabled immediate_commit (introduced in 1.1.0) - Fix a possible write stall with EC due to the lack of OSD wakeup after stabilizing previous writes - Change sync operation semantics as a final fix to possible write stalls with EC and disabled immediate_commit - Sync after deleting data in CLI rm / rm-data if immediate_commit is disabled - Fix OSDs ignoring syncs & autosyncs for delete operations - Fix OSD space reporting sometimes adding garbage zeros for deleted inodes (causing extra pool/stats etcd keys for deleted pools) - Speed up monitor failover - change default etcd_mon_ttl from 30 to 5 seconds - Speed up operation retries - change default up_wait_retry_interval to 50 ms - Add patch for libvirt 9.10	2024-02-04 02:23:49 +03:00
Vitaliy Filippov	e026de95d5	Log to systemd by default	2024-02-04 01:21:31 +03:00
Vitaliy Filippov	d2b43cb118	Change default etcd_mon_ttl	2024-01-29 23:45:19 +03:00
Vitaliy Filippov	1c322b33ed	Change default up_wait_retry_interval to 50 ms	2024-01-26 01:51:08 +03:00