Commit Graph

12 Commits (6acf562e0142cb2b0c8397b45790d17bddd2f7c4)

Author SHA1 Message Date
Vitaliy Filippov 6acf562e01 Release 1.0.0
New features:

- Data and metadata checksums!
  - Metadata checksums are always used with new disk format
  - Data checksums can be turned on with --data_csum_type crc32c for new OSDs
  - Checksum block size can be configured
  - inmemory_metadata now also affects keeping checksums in memory
- Linux page cache I/O caching support which can be enabled separately for
  data, metadata (including checksums) and journal (O_SYNC instead of O_DIRECT)
- Details [here](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/config/layout-osd.en.md#data_csum_type)
- Backwards compatibility is preserved, you can use new OSDs with old disks

Release also includes bug fixes from [0.9.6](https://git.yourcmc.ru/vitalif/vitastor/releases/tag/v0.9.6).

0.9.6 is moved to "-oldstable" repositories and will be available for some additional time.
2023-07-29 18:57:19 +03:00
Vitaliy Filippov e651c93a90 Release 0.9.6
Test / test_interrupted_rebalance (push) Successful in 1m59s Details
Test / test_interrupted_rebalance_imm (push) Successful in 3m41s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m29s Details
Test / test_failure_domain (push) Successful in 50s Details
Test / test_snapshot (push) Successful in 45s Details
Test / test_snapshot_ec (push) Successful in 23s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_chain (push) Successful in 2m23s Details
Test / test_snapshot_chain_ec (push) Successful in 3m1s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_rebalance_verify (push) Successful in 3m34s Details
Test / test_rebalance_verify_imm (push) Successful in 3m31s Details
Test / test_rebalance_verify_ec (push) Successful in 5m14s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m14s Details
Test / test_write (push) Successful in 44s Details
Test / test_write_xor (push) Successful in 54s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m38s Details
Test / test_heal_ec (push) Successful in 3m56s Details
Test / test_scrub (push) Successful in 36s Details
Test / test_scrub_zero_osd_2 (push) Successful in 35s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 46s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 25s Details
Test / test_scrub_ec (push) Successful in 24s Details
- Fix vitastor-disk partition zeroing (sometimes it was writing garbage instead of zeroes)
- Fix incorrect EC space statistics in `vitastor-cli status`
- Several bug fixes for NFS:
  - Add . and .. in NFS directory listings
  - Return FILE_SYNC from NFS writes if immediate_commit is enabled
  - Return the same "verifier" in NFS COMMIT as in NFS WRITE
  - Make parallel NFS extending writes work correctly, without conflicts
  - Handle parallel NFS extending writes without imposing extra load on etcd
- Support UTF-8 in vitastor-cli table output
- Also allow "0" and "no" as false for inmemory_metadata and inmemory_journal
- Use HDD defaults for HDD-only in automatic `vitastor-disk prepare` mode
2023-07-29 10:54:00 +03:00
Vitaliy Filippov 10a5fd6abb Release 0.9.5
Test / test_etcd_fail (push) Successful in 50s Details
Test / test_interrupted_rebalance (push) Successful in 2m27s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m39s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m49s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m29s Details
Test / test_failure_domain (push) Successful in 19s Details
Test / test_snapshot (push) Successful in 31s Details
Test / test_snapshot_ec (push) Successful in 26s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot_chain (push) Successful in 1m49s Details
Test / test_snapshot_chain_ec (push) Successful in 2m51s Details
Test / test_snapshot_down (push) Successful in 26s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 18s Details
Test / test_rebalance_verify (push) Successful in 3m8s Details
Test / test_rebalance_verify_imm (push) Successful in 3m13s Details
Test / test_rebalance_verify_ec (push) Successful in 3m36s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m59s Details
Test / test_write (push) Successful in 48s Details
Test / test_write_xor (push) Successful in 37s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s Details
Test / test_heal_ec (push) Successful in 4m6s Details
Test / test_scrub (push) Successful in 35s Details
Test / test_scrub_zero_osd_2 (push) Successful in 34s Details
Test / test_scrub_xor (push) Successful in 42s Details
Test / test_scrub_pg_size_3 (push) Successful in 52s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 33s Details
Test / test_scrub_ec (push) Successful in 28s Details
A hotfix to 0.9.4 containing only one bugfix: 100% CPU usage in the new QEMU
driver caused by the lack of eventfd reset on io_uring event handling :)
2023-07-21 00:04:41 +03:00
Vitaliy Filippov 1c10430ae1 Release 0.9.4
Test / test_interrupted_rebalance (push) Successful in 1m54s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m4s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m40s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m25s Details
Test / test_failure_domain (push) Successful in 15s Details
Test / test_snapshot (push) Successful in 25s Details
Test / test_snapshot_ec (push) Successful in 20s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 1m56s Details
Test / test_snapshot_chain_ec (push) Successful in 2m33s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m3s Details
Test / test_rebalance_verify_imm (push) Successful in 3m2s Details
Test / test_rebalance_verify_ec (push) Successful in 3m13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 8m35s Details
Test / test_write (push) Successful in 33s Details
Test / test_write_xor (push) Successful in 40s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m25s Details
Test / test_heal_ec (push) Successful in 3m9s Details
Test / test_scrub (push) Successful in 1m0s Details
Test / test_scrub_zero_osd_2 (push) Successful in 46s Details
Test / test_scrub_xor (push) Successful in 1m1s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m55s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m25s Details
Test / test_scrub_ec (push) Successful in 52s Details
- Improve QEMU driver performance by integrating io_uring in it (up to 1.5x total iops improvement)
- Fix QEMU driver deadlocks which started to reproduce in qemu-img after iothread fixes
- Fix `vitastor-cli status` reporting more etcds than actually exists (fix etcd address duplication in config on reload)
- Fix `vitastor-cli ls` crashing on inodes in non-existing pools
- Delete old garbage /pool/stats/ keys for non-existing (deleted) pools
- Reduce memory usage of etcds initialized by make-etcd script
- Fix OSDs almost always crashing on etcd restart due to "revisions were compacted" (support reloading state from etcd)
- Fix a crash and a stall possible mostly in HDD setups with small journal and big (512k, 900k) random writes
- Add notes about HDDs to documentation. You are officially allowed to use HDD-only Vitastor with HGST/Toshiba/EXOS :)
2023-07-19 02:50:30 +03:00
Vitaliy Filippov a612cdca47 Release 0.9.3
- Add patch for libvirt 9.0
- Add support for Proxmox VE 8.0
- Fix compatibility of the QEMU driver with iothread (QEMU rebuilds are coming)
- Fix vitastor-cli rm-data/rm/merge hanging when some OSDs are down.
  Allow deletions in unclean cluster at the cost of some data possibly
  "reappearing" when those OSDs start back. In that case you can just repeat
  the deletion request using rm-data.
- A bunch of bug fixes for snapshots:
  - Fix snapshot reads often not working at all with snapshot chain size > 2
  - Fix optimized snapshot data merge (children to parent)
  - Fix updating of image name index key during optimized merge
  - Fix auto-selection preventing the use of optimized merge with only 1 snapshot
  - Fix incorrect CAS retries during snapshot merge
  - Fix snapshot merge progress reporting
- Fix primary_read bitmap buffers use-after-free which could lead to
  incorrect allocation map reads
- Remove /usr/local/bin path from make-etcd
- Some documentation fixes
2023-07-01 00:25:58 +03:00
Vitaliy Filippov 926be372fd Release 0.9.2
Test / test_change_pg_count (push) Successful in 35s Details
Test / test_change_pg_count_ec (push) Successful in 34s Details
Test / test_change_pg_size (push) Successful in 9s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 1m1s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_interrupted_rebalance (push) Successful in 1m26s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m15s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m31s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m32s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 2m49s Details
Test / test_rebalance_verify_imm (push) Successful in 2m45s Details
Test / test_rebalance_verify_ec (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m3s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 35s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_write (push) Successful in 1m10s Details
Test / test_write_xor (push) Successful in 2m23s Details
Test / test_write_no_same (push) Successful in 19s Details
Test / test_heal_pg_size_2 (push) Successful in 3m58s Details
Test / test_heal_ec (push) Successful in 4m8s Details
Test / test_scrub (push) Successful in 1m3s Details
Test / test_scrub_zero_osd_2 (push) Successful in 36s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 51s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 32s Details
Test / test_scrub_ec (push) Successful in 43s Details
- Measure and report scrub I/O statistics in vitastor-cli status
- Make aggregated statistics in vitastor-cli status much smoother
  (first derive, then sum instead of first summing and then deriving)
- Fix an old rare bug leading to journal corruption
  (try to use scrub if you think you're affected...)
- Do not start EC PGs without at least <data chunks> OSDs in each old set
  (prevents spurious read errors with EC during reconnections/restarts)
- Fix failed assert(!scrub_list_op) on OSD restart with pending scrubs
- Fix future planned scrubs not starting because of incorrect time comparison
- Build packages for Debian 12 (Bookworm)
2023-06-18 19:44:33 +03:00
Vitaliy Filippov bdd48e4cf1 Release 0.9.1
Test / test_change_pg_count (push) Successful in 35s Details
Test / test_change_pg_count_ec (push) Successful in 32s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 6s Details
Test / test_etcd_fail (push) Successful in 49s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_interrupted_rebalance (push) Successful in 1m4s Details
Test / test_interrupted_rebalance_imm (push) Successful in 57s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m2s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 53s Details
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 1m45s Details
Test / test_rebalance_verify_imm (push) Successful in 1m41s Details
Test / test_rebalance_verify_ec (push) Successful in 1m53s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 1m48s Details
Test / test_rm (push) Successful in 10s Details
Test / test_snapshot (push) Successful in 14s Details
Test / test_snapshot_ec (push) Successful in 16s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_write (push) Successful in 49s Details
Test / test_write_xor (push) Successful in 1m1s Details
Test / test_write_no_same (push) Successful in 11s Details
Test / test_heal_pg_size_2 (push) Successful in 3m12s Details
Test / test_scrub (push) Successful in 28s Details
Test / test_scrub_zero_osd_2 (push) Successful in 22s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 25s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 26s Details
Test / test_scrub_ec (push) Successful in 24s Details
- Fix "Client XX command out of sync" messages sometimes happening on OSD reconnections
- Fix a bug where EC reads parallel with writes to the same object failed with -ERANGE error
- Slightly reduce the amount of metadata writes during journal flushing
- Correctly unmap NBD volumes when Proxmox forces map_volume use (with SWTPM and maybe some other cases)
2023-06-10 11:42:49 +03:00
Vitaliy Filippov 3b4cf29e65 Release 0.9.0
New features:
- Scrubbing! Check documentation: [auto_scrub](src/branch/master/docs/config/osd.en.md#auto_scrub)
- Document online-updatable configuration parameters

Bug fixes:
- Fix NaN during PG optimisation if there are nonexisting OSDs in node_placement
- Fix monitor crash on pool deletion
- Clear journal_device and meta_device before initialising the next OSD in automatic mode
- Sync unsynced deletes before overwriting them with a lower version
  (reproducted mostly/only after scrubbing)
2023-05-21 15:07:14 +03:00
Vitaliy Filippov eeaba11ebd Use fio 3.27-8 for alma9 2023-05-21 14:48:26 +03:00
Vitaliy Filippov 5a9e1ede52 Release 0.8.9
Test / buildenv (push) Successful in 9s Details
Test / build (push) Successful in 2m31s Details
Test / test_cas (push) Successful in 12s Details
Test / make_test (push) Successful in 33s Details
Test / test_change_pg_size (push) Successful in 19s Details
Test / test_change_pg_count (push) Successful in 55s Details
Test / test_create_nomaxid (push) Successful in 21s Details
Test / test_change_pg_count_ec (push) Successful in 58s Details
Test / test_failure_domain (push) Successful in 13s Details
Test / test_etcd_fail (push) Successful in 1m4s Details
Test / test_interrupted_rebalance (push) Successful in 1m13s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m7s Details
Test / test_add_osd (push) Successful in 2m59s Details
Test / test_move_reappear (push) Successful in 24s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m22s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m1s Details
Test / test_rebalance_verify (push) Successful in 2m12s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_rebalance_verify_imm (push) Successful in 2m4s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m9s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot (push) Successful in 23s Details
Test / test_rebalance_verify_ec (push) Successful in 2m31s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_write (push) Successful in 53s Details
Test / test_write_xor (push) Successful in 1m19s Details
Test / test_heal_pg_size_2 (push) Successful in 4m30s Details
Test / test_heal_ec (push) Successful in 4m32s Details
- The tests are now stable and run in a CI system based on Gitea CI
- The release includes final bug fixes for EC:
  - Implement missing EC recovery of allocation bitmap when built with ISA-L
  - Fix broken snapshot export with EC (allocation bitmap reads were giving incorrect results previously)
- Also fixed bugs manifesting under heavy load:
  - Fix monitor possibly applying incorrect PG history on retries
  - Fix monitor incorrectly changing PG count when last_clean_pgs contains less PGs than the new number
  - Allow writes to wait for free space again, but now correctly (previously dropped in 0.8.2)
  - Fix a rare segfault in client (handle client stop during incoming stream handling in 1 more place)
  - Make monitor correctly handle etcd connection errors - it could die instead of connecting to another etcd
  - Fix OSD rarely being unable to report PG states after a PG was taken over by another OSD
- Fixed return code for incomplete EC objects (now EIO) and made cluster client retry this error
- Made other small changes for tests: timeouts, nice/ionice for etcd, waiting conditions, NBD device checks and so on
2023-05-14 01:25:09 +03:00
Vitaliy Filippov ab615849d6 Release 0.8.8
- Fix vitastor-cli rm/rm-data broken in 0.8.6 (missing messenger initialization)
- Prepare OSD read handler for upcoming version with scrub - allow "secondary reads" to return errors
- Fix OSDs re-peering PGs infinitely with a big number of PGs (reproduced in test_add_osd)
- Fix another variant of flusher sync-waiting stall (reproduced in test_write)
- Fix other tests in tests/ (will add them to Gitea CI soon)
- Add patches for QEMU 6.2-8.0
- Fix QEMU driver compatibility with QEMU 8.0
- Build packages for RHEL 9 clones (based on AlmaLinux 9)
2023-04-28 11:22:00 +03:00
Vitaliy Filippov 2a2f4f6738 Add Almalinux 9 build 2023-04-28 00:40:50 +03:00