111 Releases 111 Tags

RSS Feed

• v3.2.2 df593badb5

  Compare

  Release 3.2.2 Stable

  vitalif released this 2026-09-27 14:29:43 +03:00 | 85 commits to master since this release

  • New store (thanks Changwei Ye)
    • Fix rejected compaction when an object is already overwritten again
    • Reduce superblock I/O amplification introduced after previous correctness fixes
  • Fix possible vitastor-cli rm crash with assert(rwo->todo == 1) introduced in 3.2.1
  • Fix replicated objects possibly incorrectly transitioning to the INCOMPLETE state
    after multiple failed partial writes
  • Fix possible "operation sequencing broken ... expected num 2, got 3, stopping client"
    possible when a cluster is overloaded by peering/OSD flaps

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.2.1 06a5d24240

  Compare

  Release 3.2.1 Stable

  vitalif released this 2026-09-13 11:11:12 +03:00 | 94 commits to master since this release

  • Blockstores
    • Add more asynchrony to the chaos test to uncover more issues...
    • ...and fix them (~6 in the old store, ~9 in the new store)
    • Fix size check for journal_offset when sharing meta device (thanks Changwei Ye)
  • Network layer
    • Correctly handle out-of-order zero-copy notifications (rare but possible in theory)
    • Fix double checksumming of header in proto_checksums=full mode
    • Add rxbounce option for Windows guests with TCP and enabled checksums
    • Close stopped peer sockets early to prevent hung zerocopy notifications and slow failover (~15 min instead of ~10 sec)
  • OSD
    • Do not die on local blockstore modification errors (irrelevant since added write error handling)
  • CSI
    • Fix VitastorFS NFS daemon lifecycle (thanks Dmitry Naumov)
    • Use a newer qemu-storage-daemon in CSI
  • vitastor-cli
    • Add dd opool option
    • Fix rm with "inverse" optimisation incorrectly removing layer data before rename
      possibly leading to incorrect reads from child images during removal of an older-than-latest snapshot
    • Fix a possible hang in merge and derived commands (rm, flatten)
    • Fix mistakes in the serve OpenAPI spec (thanks Stepan Rabotkin)
    • Fix possible segfaults in serve due to invalid request parsing (thanks Vincent Schweiger)
    • Fix serve no-HTTPS mode (thanks both :))
  • Build
    • Fix non-x86 builds failing due to unsupported xxhash x86dispatch
    • Forcibly enable assertions in all builds to remove build profile dependency
    • Fix QEMU driver build with QEMU < 10.2

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.2.0 f9a247df42

  Compare

  Release 3.2.0 Stable

  vitalif released this 2026-08-31 02:02:34 +03:00 | 157 commits to master since this release

  Read more at https://vitastor.io/blog/2026-08-31-v3.2.0.html

  • Add a DST-style chaos test for both blockstores and fix a large number of problems uncovered with it.
  • Implement online resize for ublk, NBD, QEMU and VDUSE, and CSI.
  • Add VitastorFS mount helpers.
  • Allow to configure NFS --readahead.
  • Allow to use the client without io_uring again (use_sync_send_recv) - with RDMA it's just as fast.
  • Add a /healthcheck handler to the monitor.
  • Handle SIGINT and SIGTERM in NBD, ublk and NFS servers.
  • Prevent infinite blocking of the blockstore operations - retry with EAGAIN.

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.1.0 90ed7a800b

  Compare

  Release 3.1.0 Stable

  vitalif released this 2026-07-24 22:10:11 +03:00 | 263 commits to master since this release

  New features:

  • Fully secure cluster setup support:
    • etcd transport encryption (TLS/mTLS)
    • OSD transport encryption (AES-GCM), header-only by default, thus very fast
    • End-to-end image data encryption (AES-XTS)
    • Optional external Vault AES-XTS key storage
    • Certificate-based authentication
    • Optional etcd- and OSD-based access control
    • xxhash3 transport checksums (data-only by default)
  • New on-disk checksum algorithm option: xxh3_32 - xxhash3 truncated to 32 bits
  • DNS name support for etcd and Vault URLs
  • Mixing CAS and non-CAS writes within a single client instance is now allowed and works correctly
  • Debian 10 (Buster) and CentOS 7 support is dropped

  Read detailed security feature documentation here: https://vitastor.io/docs/usage/security.html

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.16 5dec46c82e

  Compare

  Release 3.0.16 Stable

  vitalif released this 2026-07-19 11:09:25 +03:00 | 351 commits to master since this release

  New store:

  • Finally fix the source of 'double-claim' errors worked around in 3.0.12 - delete
    entry removal on restart possibly happening before the removal of the masked big_write
  • Fix padded checksum (csum_block_size > 4k) verification during some partial reads
  • Fix truncated unstable object list responses in limited listing requests (sent by scrub)
  • OSD should now stop with an error on data fsync error during batched big_writes, like
    on other disk errors
  • Add missing disable_meta_fsync check for enabling immediate_commit

  Client:

  • Fix CAS writes skipping their built-in fsync (for pools without immediate_commit)
  • Validate show_config, sec_list and sec_read_bmp reply sizes
  • Fix show_config reply parsed as a null terminated string without taking length into account

  Old store:

  • Skip too small (corrupted) journal entries

  OSD:

  • Forbid pools with block_size*pg_size larger than UINT32_MAX
  • Do not forcibly report OSD statistics after reporting PG states (should reduce etcd load)
  • Prevent repeated reporting of PG epoch bump on multiple writes possibly leading
    to spurious PG state reporting errors, extra PG repeers and increased etcd load
  • Fix handling of PGs with pg_size between 32 and 64 (more than 64 is forbidden)
  • Add safety checks for ENOENT (missing objects) during read/write operations
  • Abort primary writes with a retriable error code when the PG becomes inactive during write
    instead of trying to finish it cleanly
  • Fix sync operation correctness and add tests to verify it:
    • Drop dirty PG connections when repeering PG before waiting for actual peering, like in stop
    • Complete sync with an error if PGs dirtied by the client become inactive
    • Skip inactive PGs during sync (double-check). Without this fix and write error handling
      changes inflight operation counter could be incremented on offline PGs and lead to use-after-free
    • Drop client's connection instead of returning SYNC error
  • Several scrub fixes:
    • Check bitmaps when grouping identical EC chunks in scrub
    • Fix scrub possibly trying too many combinations in EC pools with large k+n
    • Fix reads from degraded objects marked as inconsistent after scrub not taking
      actual object state into account and possibly returning zeros
    • Fix incorrect array indexing in scrub (possibly affecting scrub results)

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.15 5ef9d78461

  Compare

  Release 3.0.15 Stable

  vitalif released this 2026-06-28 11:29:38 +03:00 | 382 commits to master since this release

  • QEMU virtual disk migration with enabled iothread is finally fixed correctly.
  • Fixed operation of Proxmox VMs with swTPM without enabling NBD for all disks.
  • Debian packages are now again built with stable Antietcd released instead of
    the unstable master branch.
  • Antietcd cluster mode previously broken in that master branch is fixed. The
    symptom was Antietcd being unable to elect the leader in a cluster.
  • Fixed monitor startup with embedded Antietcd when using IPv6.
  • Fixed space statistics calculation for FS and S3 pools in the new storage (PR #127).

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.14 278852b4d5

  Compare

  Release 3.0.14 Stable

  vitalif released this 2026-06-21 18:17:02 +03:00 | 391 commits to master since this release

  How many (bugs!) I've knifed, how many I've slit!

  General note: most bug fixes now include regression tests to verify that they don't repeat in the future.
  Most bugs fixed in this release were detected by using LLM analysis (Claude Opus/Fable, GPT 5.5).

  OSD:

  • Fix OSD hanging with an infinite loop when setting autosync_interval to 0 at runtime
  • (IMPORTANT) Fix EC PGs hanging in REPEERING when the last final commit/rollback in a
    batch completes with an error
  • Limit pg_size by 64 because peering doesn't handle larger values with EC — they just
    lead to 'incomplete' objects
  • Fix OSD crash with an "assertion failed" error on EIO retry in snapshot chain read
    (i.e. when some chunks belong to a corrupted replica with checksum mismatch)
  • (IMPORTANT) Disable chunked PG count resharding due to possible interference with
    compaction changes (will be re-enabled after fixes)
  • (IMPORTANT) Fix incorrect snapshot allocation bitmap recovery during EC chained read
  • Add on-wire request size validation to prevent possible OOM/DoS/heap corruption
    on receiving invalid data from the network
  • (IMPORTANT) Fix parity-less EC writes destroying snapshot allocation bitmaps
    (i.e. when all parity OSDs in a PG are missing)
  • (IMPORTANT) Fix EC N+K, K>=2 recovery destroying snapshot allocation bitmaps
    of live parity chunks
  • (IMPORTANT) Fix a possible OSD crash during EC misplaced object scrubbing
  • (IMPORTANT) Verify object bitmap consistency during scrub (only data was checked previously)
  • (IMPORTANT) Fix corrupted object chunks incorrectly marked as non-corrupted on the second scrub
  • (IMPORTANT) Fix cached EC decoding of multiple stripes with ISA-L (ISA-L is the default)

  New store:

  • Fix a theoretically possible OSD crash on startup when using the previously added
    workaround for the "double-claim" problem
  • Remove theoretically possible incorrect metadata block writes during batch EC COMMITs
    restarted due to a full metadata area
  • Fix incorrect compaction counter tracking after OSD restart (could probably lead to
    compaction not restarted correctly after a restart)
  • (IMPORTANT) Fix some of parallel big_writes possibly not waiting for data fsync, thus not providing durability
  • Fix possible OSD crash on sync retry when io_uring is full
  • Fix a possible crash during startup on corrupted on-disk data with too small entry sizes

  Old store:

  • Prevent loading extra garbage metadata entries from the last 4 MB of metadata area
  • Fix read operations possibly crashing if a metadata read (with inmemory_metadata=false)
    was restarted due to a full io_uring
  • Fix a possible memory leak of temporary buffers and bitmaps/checksums when a read
    was restarted due to a full io_uring (reproducible with either inmemory_metadata=false or block_size>256k)
  • Fix a possible OSD crash during padded checksum reads if buffer count exceeded 1024
    (IOV_MAX) (reproducible only with csum_block_size > 4k and block_size >= 4M)
  • (IMPORTANT) Fix partial padded read journal checksum verification with csum_block_size > 4k
  • Fix incorrect marking of corrupted objects as non-corrupted after flushing data
    from journal (with inmemory_journal=false)
  • (IMPORTANT) Fix deferred freeing of a different block when a block was used by a parallel read
  • Fix per-inode statistics not being disabled for FS and S3 pools correctly, leading to etcd
    overload with unneeded per-inode statistics, slower etcd operation, increased memory usage,
    and too many Prometheus statistics exported by the monitor

  Both stores:

  • Fix possibly left garbage in the metadata area if the first OSD startup was interrupted —
    metadata header is now written only after initializating metadata
  • Check for short reads during initialization (just in case, doesn't happen in real life)

  Clients:

  • Fix write-back queue item split in case when write-back is enabled at runtime
  • Implement bdrv_detach_aio_context & bdrv_attach_aio_context in the QEMU driver (should fix migration with iothread)
  • Do not crash on full io_uring in ublk server
  • Fix missing --readonly option handling in NBD server
  • Stop gracefully on NBD_CMD_DISC instead of just exit(0) in NBD server
  • Fix writeback detection in ublk server for --image mode
  • Limit the amount of incoming data for NFS clients to prevent choking on memory in async mount mode

  Tools (vitastor-disk/vitastor-cli):

  • Prevent vitastor-cli merge possibly exiting before completing the last sync/delete operations
  • Fix vitastor-disk incorrectly validating too large small_write entry length
  • Fix vitastor-cli merge ignoring input option validation errors
  • Fix vitastor-cli rm-data always skipping the final fsync
  • Fix vitastor-disk resize not moving the last used data block
  • Fix vitastor-disk write-meta incorrectly importing new store small_write entries
  • Fix vitastor-disk write-journal and write-meta importing old store data incorrectly
    when csum_block_size is > 4k
  • Support --io option for vitastor-disk dump-journal/write-journal
  • Fix vitastor-disk resize crash when converting from very old (0.5.x) metadata
  • Fix vitastor-disk trim incorrectly rounding block ranges with --discard_granularity
    option explicitly set to a value > 4k, possibly leading to discarding live data
  • Fix vitastor-disk write-meta importing new store metadata incorrectly with > 4 GB metadata area size
  • (IMPORTANT) Fix vitastor-cli modify --resize to a smaller size clearing all image data O_o

  Other:

  • Do not crash with an uncaught exception when an invalid /osd/state/ with a non-numeric
    suffix is present in etcd (in OSD and all client services)
  • Fix possible crash in vitastor-kv when handling a corrupted DB due to a uint32 overflow
  • Fix NFS-RDMA memory allocator crashing in some situations
  • Fix small shared file extend-write potentially reading unallocated memory (NFS)
  • Add bounds checks to prevent uint32 overflows in NFS/XDR
  • Re-enable accidentally disabled safety checks (asserts) in files with included cpp-btree
  • Fix too small memory allocation in NFS portmap

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.13 ec6a70bbd3

  Compare

  Release 3.0.13 Stable

  vitalif released this 2026-05-31 12:22:43 +03:00 | 471 commits to master since this release

  • New store fixes:
    • Fix repeated rollback logic
    • Fix crash on rolled back object compaction
    • Fix postpone_load possibly merging different object chains
    • Fix block_csums import in vitastor-disk write-meta
    • Fix header checksum after vitastor-disk write-meta
  • Old store fixes:
    • Fix batched fsync possibly skipped by some flush coroutines
  • Improve ENOSPC test, fix possible crash on ENOSPC
  • Add fsyncs to vitastor-disk prepare
  • Fix possible crash on pg_lock check failure in sec_read_bmp
  • Fix VitastorFS initialization when local_reads are enabled

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.12 63fe3c323a

  Compare

  Release 3.0.12 Stable

  vitalif released this 2026-05-18 02:24:22 +03:00 | 491 commits to master since this release

  Important fixes (except the new store):

  • Fixed a possible use-after-free in the OSD during error handling of initial
    commit/rollback of objects in EC pools.
  • Fixed a possible free of an invalid pointer in the OSD during read errors
    from snapshot/clone chains in EC pools.
  • Fixed possibly incorrect handling of commit/rollback operations in EC pools
    during pool PG count changes.
  • Fixed the inverted fsync enable parameter in the ublk driver (fsync was not
    enabled on pools without immediate_commit).
  • Added the raw-ls command for debugging purposes to find object versions in
    the cluster using listing operations.

  New store fixes:

  • Improved startup speed by using LSN-based sorting only for objects with a
    large number of intermediate versions.
  • Added skip_double_claim option as a temporary workaround to fix the rare OSD
    startup error with the "double claimed block" message, observed by several
    users. This option does not affect data integrity.
  • Fixed incorrect rechecking of small writes during startup, which in theory
    could lead to duplicate small write object entries on the OSD.
  • Fixed fsync operation for disks with a writeback cache (without capacitors):
    • Fixed incorrect semantics of consecutive fsyncs (next fsync was not blocked
      by the previous one).
    • Added fsync when copying small writes from the buffer to the data device
      (somehow forgotten during initial development).
    • Added fsync after the initial garbage collection during OSD startup.
    • Fixed incorrect cast of LSN from uint64 to uint32, breaking fsync when
      reaching LSN 2^32.
  • Added missing verification of the metadata header checksum during startup.
  • Fixed incorrect updating of object checksums in perfect_csum_update=true mode.
  • Fixed a possible OSD crash with "assertion failed" when processing a malformed
    EC STABILIZE operation.
  • Fixed the accounting of active compactor coroutines.
  • Removed broken and untested new->old store conversion support.

  Minor issues fixed:

  • Incorrect accounting of OSD local operation statistics in replicated pools.
  • Missing non-zero exitcodes on vitastor-disk resize command errors.
  • Missing reset of the list of inconsistent objects during PG restarts.
  • Theoretically possible hangs of various OSD operations when working with
    completely corrupted objects (without a single available copy), and possibly
    in some other very rare situations.
  • Incorrect fsyncs when deleting objects from pools without immediate_commit
    (on disks with a writeback cache), which previously could leave garbage when
    deleting misplaced objects.
  • Possible crash/memory corruption of the NFS server during a targeted attack
    on NFS-RDMA.
  • Possibly incorrect handling of ENOSPC/EIO write errors in replicated pools,
    leading to inability to retry the write later.
  • Possible crash instead of a clean error exit when starting an OSD with the
    old storage engine on a disk with corrupted journal data.
  • Shallow copying of PG configuration in the monitor, however, not related to
    actual bugs.
  • Incorrect checking of allocated blocks in the QEMU driver in an unused code
    branch (without the BDRV_WANT_ZERO flag).
  • Possible memory leak on read errors of corrupted objects.
  • Possible incorrect PG states when corrupted objects are detected.
  • Possible failure to mark all "bad" copies of an object during scrubs without
    checksums and with a large number of replicas (> 4).
  • Incorrect checksum calculation in the old storage engine when
    bitmap_granularity < 4096 (a practically unused configuration).
  • Theoretically possible OSD crash in rare cases during a scrub and simultaneous
    object recovery.
  • Theoretically possible OSD crash when handling PING operation errors.
  • Slightly suboptimal logic for reusing the RDMA send buffer.
  • Possible memory leak when canceling an already running scrub via no_scrub.
  • Possible memory corruption when a client (e.g., QEMU code) passes invalid
    buffers and the writeback cache is enabled.
  • Potentially incorrect search for corrupted parts of EC objects (inability
    to find a "good" combination) during a scrub with checksums disabled.
  • Possible additional memory usage on the OSD side when handling failed reads
    from snapshots (not a leak however - the memory was freed upon client
    disconnection).
  • Potential sudden write slowdown at certain pg epoch values due to incorrect
    epoch update logic in etcd.

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)
• v3.0.11 041185c673

  Compare

  Release 3.0.11 Stable

  vitalif released this 2026-05-09 18:49:36 +03:00 | 555 commits to master since this release

  A single hotfix:

  • Fix missing error handling in EC snapshot/cloned image reads potentially
    leading to corrupted reads during OSD/PG restarts

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)