111 Releases 111 Tags

RSS Feed

• v3.0.10 4acfe149cb

  Compare

  Release 3.0.10 Stable

  vitalif released this 2026-04-27 13:53:48 +03:00 | 558 commits to master since this release

  Important bug fixes (new store):

  • Fix OSDs possibly refusing to start with "write metadata failed at offset xxx: Invalid argument"
    (fix buffer alignment during initial garbage collection)
  • Rollback change from 3.0.4 - on-disk garbage entries are not skipped on start again. This change
    doesn't have any impact normally, but OSDs originally running 3.0.0-3.0.2 and then upgraded
    to 3.0.9 may hit a bug where 3.0.9 refuses to start due to entries marked as garbage too
    early and flushed to disk in 3.0.0-3.0.2.

  Other changes:

  • Auto-select the only RDMA device/port if there is only one
  • Rollback one 3.0.9 change - there was no actual use-after-free :)
    (the problem was only relevant to an unstable development version)
  • Fix vitastor-nfs --trace option
  • Fix an unintended 1 second sleep in vitastor-cli rm-data
  • Fix inode statistics not being cleared for a deleted pool
  • Fix print to stdout in client

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.9 dd5941b9a4

  Compare

  Release 3.0.9 Stable

  vitalif released this 2026-04-04 18:38:40 +03:00 | 567 commits to master since this release

  • Fix broken garbage collection in the new store leading to very high memory usage
  • Make the new store do full garbage collection on OSD startup by default (can be turned off with a new gc_on_start parameter)
  • Make startup logging with the new store more informative, allow to log amount of garbage with log_level > 1
  • Fix clients reading from snapshots incorrectly on retries (for example when OSDs were restarted during read)
  • Fix a possible "send-after-free" in OSD which could probably also affect snapshot reads
  • Fix OSD crashing when handling a read_bitmap operation from a deleted object with the new store

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.8 b30635b932

  Compare

  Release 3.0.8 Stable

  vitalif released this 2026-03-30 02:09:02 +03:00 | 578 commits to master since this release

  • Surrogate increment-only peer IDs are now used instead of file descriptor numbers,
    and client cleanup sequence is reworked in another attempt to fix a possible PG hang
    in 'peering' states. The history of this bug is still unclear and it still hasn't been
    reproduced in tests, but several users have reported it since 3.0.4 (only with enabled RDMA).
  • Fix a minor bug where an OSD could skip reporting misplaced PG state when moving
    an object in non-immediate_commit mode.
  • Add Ubuntu 26.04 (Resolute Raccoon) support and patches for qemu 10.2 and libvirt 12.0

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.6 538620b400

  Compare

  Release 3.0.6 Stable

  vitalif released this 2026-03-22 22:49:51 +03:00 | 589 commits to master since this release

  • Fix one more bug in old store checksums O_o - checksum verification wasn't
    working correctly on reads quickly after small initial offsetted writes
    (when a BIG_WRITE was read from the journal).
  • Fix clients possibly not destroyed after being switched to RDMA. Not 100%
    verified, but in theory it could rarely lead to PGs hanging in peering/stopping
    states with RDMA enabled.
  • Use vitastor-cli in the Cinder driver for modifications instead of direct etcd
    interaction.
  • Fix mon_https_ca parsing.
  • Fix Docker install.sh, add instructions for Podman.

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.5 cb639a130d

  Compare

  Release 3.0.5 Stable

  vitalif released this 2026-03-07 13:54:44 +03:00 | 596 commits to master since this release

  A single urgent fix for the new store: metadata was written incorrectly when
  filled metadata area exceeded 4 GB due to a simple & stupid 32-bit integer overflow.
  This was leading to OSDs corrupting metadata and being unable to start on the next
  run.

  The bug affected only the new store (meta_format=3) and OSDs with metadata area size
  larger than 4 GB. You can check your OSD metadata area size by checking data_offset
  in vitastor-disk read-sb /dev/vitastor/osdXX-data for single-disk OSDs or
  blockdev --getsize64 /dev/vitastor/osdXX-meta for hybrid OSDs.

  If you're affected, it's highly recommended to:

  • Install updated packages without restarting OSDs
  • Restart OSDs one by one
  • If a restart of an OSD triggers "has_degraded" rebalance - wait for it to finish
    before restarting another OSD
  • If an OSD fails to start with "double-claimed" errors in the log - purge and recreate it,
    and again wait for "has_degraded" rebalance to finish before proceeding to the next OSD

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.4 09df74cfac

  Compare

  Release 3.0.4 Stable

  vitalif released this 2026-03-03 20:12:31 +03:00 | 601 commits to master since this release

  New store bug fixes:

  • Fix very slow OSD startup (hours) possible in unlucky cases
  • Fix incorrect garbage entry validation on start in some cases, preventing OSD startup
  • Fix small_write_throttling not functioning and either hanging or crashing OSDs
  • Fix "different fact_versions returned from subops" error possible with EC in some cases
  • Add a recovery option for the new store: skip_corrupted_meta_entries
  • Disable intent writes (even 4k) for hybrid OSDs by default for better write buffering

  Other changes:

  • Fix chmod/chown of a new empty VitastorFS root directory not working
  • Fix vitastor-dd treating all < 32k reads as always empty
  • Fix vitastor-disk always setting 32k csum_block_size for HDDs when checksums are enabled
  • Fix OSD missing PG state updates and PGs hanging in "starting" in rare cases
  • Fix a possible OSD crash with "map::at" message in rare cases when stopping a client
  • Fix a possible OSD crash on start when RDMA is available but fails to initialize
  • Fix removal and overwriting of totaled objects (objects with all corrupted copies)
  • Do not allow data_block_size < 8*bitmap_granularity
  • Remove RDMA ODP (On-Demand Paging) support
  • Update antietcd to 1.2.4
  • Update Docker build to Debian Trixie
  • Add packages for AlmaLinux 10

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.3 facaa2cc6a

  Compare

  Release 3.0.3 Stable

  vitalif released this 2026-02-08 01:39:57 +03:00 | 628 commits to master since this release

  • Fix csum_block_size > 0 unusable with atomic writes in the new store
    (almost all atomic write requests generated invalid checksums with csum_block_size > 0)
  • Fix monitor sometimes randomly failing to optimize PGs with the "problem is
    infeasible or unbounded" message due to not waiting to read full stdout of lp_solve
  • Remove reshard_abort optimisation to fix chunked resharding introduced in 3.0.2 possibly
    corrupting in-memory OSD state when handling multiple rapid PG count change requests
  • Fix removed inodes not disappearing from OSD statistics in the new store, leading to
    bloating of the statistics with old inodes
  • Increase test coverage for the new store and fix several minor bugs:
    • Enabling/disabling of used_for_app was recalculating inode space statistics incorrectly
    • Fix object metadata validation on OSD startup rejecting some correct sequences of entries,
      possibly leading to OSD being unable to start
    • Fix PG activation with EC failing in rare cases with EBUSY when requesting to commit an
      already committed write
    • Fix a theoretically possible metadata writeback issue on ENOSPC during commit
  • Fix monitor failing to optimize PGs in presence of a host with name convertible to
    js Number (like 04e278988710) :D
  • Fix vitastor-cli dd sometimes (rarely) truncating the image when writing to stdout
  • Fix a theoretically possible client connection object leak when io_uring is full
  • Fix a leak of RDMA-CM connection objects
  • Fix crashes with data_block_size < 32KB (useless setup, but anyway) (#113)

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.2 d75334ddf0

  Compare

  Release 3.0.2 Stable

  vitalif released this 2026-01-25 16:08:02 +03:00 | 655 commits to master since this release

  • Antietcd is now officially safe to use: the release includes a fixed version of antietcd
    which passes Jepsen transaction serializability tests.
  • Fix a huge checksum bug in the old store: incorrect checksums for small initial writes.
    The bug affected writes of exactly csum_block_size (4k by default) into new (unallocated)
    objects and generated invalid checksums in the store for the written 4k block. Moreover,
    generation of these invalid checksums was very slow because it was calculating CRC32
    for 4 GB of zeroes. If the block wasn't then overwritten as a part of a larger write
    request it became unreadable even though the stored data was correct. The bug affected
    all versions since 1.0.0, or since 2.3.0 because vitastor-disk didn't allow to enable
    checksums prior to 2.3.0 because of another bug. O:-)
  • Prevent OSD disconnections due to long blocking of event-loop caused by PG resharding
    (moving objects between old and new PGs in memory) when changing pool PG count or
    just on restart of an OSD with a large database (for example, with a filled 8 TB SSD).
    The issue should now be fixed because OSD now performs resharding in chunks with pauses
    between chunks.
  • Prevent pools stuck in paused state on an aborted PG count change.
  • Fix a possible OSD crash with "division by zero" when trying to handle an operation
    before pool PG count is applied to the in-memory store.

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.1 fe83825ead

  Compare

  Release 3.0.1 Stable

  vitalif released this 2025-12-22 02:17:43 +03:00 | 668 commits to master since this release

  Important fixes:

  • Disable RWF_ATOMIC by default because Linux incorrectly requires all atomic
    writes to be power-of-2-sized and length-aligned. Details: use_atomic_flag
  • Fix cross-pool snapshots not working at all - always reading old data after taking the snapshot
  • Fix level_placement (broken in 2.2.0)
  • Fix CAS write return values in the client library (broken in 2.4.4), also breaking unaligned writes in VitastorFS
  • Fix VitastorFS possibly losing some of intersecting parallel unaligned writes
  • Prevent possible reads of the old data during unfinished intent writes in the new store
  • Tests added for all of above problems to prevent future regressions

  Other fixes:

  • Allow to specify OSD tags and weights during prepare
  • Only clear the first block instead of whole OSD metadata and journal areas during vitastor-disk
    prepare - OSD anyway clears them on the first run
  • Fix vitastor-cli dd non-seekable detection and error status
  • Do not stop OSD on zero-copy tcp short send errors - these aren't a bug, but just caused
    by TCP client disconnections
  • Fix modify-osd not working after deleting the OSD configuration key from etcd
  • Fix regular antietcd websocket disconnections in OSD due to lack of WS_PONG responses to WS_PING
  • Reduce the number of allocated RDMA memory regions
  • Allow routed RoCE

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.0 a67c415e0f

  Compare

  Release 3.0.0 Stable

  vitalif released this 2025-12-06 02:12:00 +03:00 | 690 commits to master since this release

  A single new feature: the new log-structured metadata store implementation, described
  in the presentation from Moscow Highload'2025 (check it out here).

  It's now the default store for new OSDs. The support for the old store is also left in place,
  you can still choose it for new OSDs with vitastor-disk prepare --meta_format 2.

  OSDs from previous versions with the old store format will also continue to operate just like before.

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)