Vitaliy Filippov
5dec46c82e
New store:
- Finally fix the source of 'double-claim' errors worked around in 3.0.12 - delete
entry removal on restart possibly happening before the removal of the masked big_write
- Fix padded checksum (csum_block_size > 4k) verification during some partial reads
- Fix truncated unstable object list responses in limited listing requests (sent by scrub)
- OSD should now stop with an error on data fsync error during batched big_writes, like
on other disk errors
- Add missing disable_meta_fsync check for enabling immediate_commit
Client:
- Fix CAS writes skipping their built-in fsync (for pools without immediate_commit)
- Validate show_config, sec_list and sec_read_bmp reply sizes
- Fix show_config reply parsed as a null terminated string without taking length into account
Old store:
- Skip too small (corrupted) journal entries
OSD:
- Forbid pools with block_size*pg_size larger than UINT32_MAX
- Do not forcibly report OSD statistics after reporting PG states (should reduce etcd load)
- Prevent repeated reporting of PG epoch bump on multiple writes possibly leading
to spurious PG state reporting errors, extra PG repeers and increased etcd load
- Fix handling of PGs with pg_size between 32 and 64 (more than 64 is forbidden)
- Add safety checks for ENOENT (missing objects) during read/write operations
- Abort primary writes with a retriable error code when the PG becomes inactive during write
instead of trying to finish it cleanly
- Fix sync operation correctness and add tests to verify it:
- Drop dirty PG connections when repeering PG before waiting for actual peering, like in stop
- Complete sync with an error if PGs dirtied by the client become inactive
- Skip inactive PGs during sync (double-check). Without this fix and write error handling
changes inflight operation counter could be incremented on offline PGs and lead to use-after-free
- Drop client's connection instead of returning SYNC error
- Several scrub fixes:
- Check bitmaps when grouping identical EC chunks in scrub
- Fix scrub possibly trying too many combinations in EC pools with large k+n
- Fix reads from degraded objects marked as inconsistent after scrub not taking
actual object state into account and possibly returning zeros
- Fix incorrect array indexing in scrub (possibly affecting scrub results)
Vitastor
The Idea
Make Clustered Block Storage Fast Again.
Vitastor is a distributed block, file and object SDS, direct replacement of Ceph RBD, CephFS and RGW, and also internal SDS's of public clouds. However, in contrast to them, Vitastor is fast and simple at the same time. The only thing is it's slightly young :-).
Vitastor is architecturally similar to Ceph which means strong consistency, primary-replication, symmetric clustering and automatic data distribution over any number of drives of any size with configurable redundancy (replication or erasure codes/XOR).
Vitastor targets primarily SSD and SSD+HDD clusters with at least 10 Gbit/s network, supports TCP and RDMA and may achieve 4 KB read and write latency as low as ~0.1 ms with proper hardware which is ~10 times faster than other popular SDS's like Ceph or internal systems of public clouds.
Vitastor supports QEMU, UBLK, NBD, NFS protocols, OpenStack, OpenNebula, Proxmox, Kubernetes drivers. More drivers may be created easily.
Read more details in the documentation. You can start from here: Quick Start.
Talks and presentations
- KuberConf'2025: video
- Highload'2025: video, youtube, presentation (in Russian, in English)
- Highload'2022: presentation (in Russian), video
- DevOpsConf'2021: presentation (in Russian, in English), video
Documentation
- Introduction
- Installation
- Configuration
- Usage
- vitastor-cli (command-line interface)
- vitastor-disk (disk management tool)
- fio for benchmarks
- UBLK for kernel mounts
- NBD - old interface for kernel mounts
- QEMU, qemu-img and VDUSE
- NFS clustered file system and pseudo-FS proxy
- Administration
- Performance
Author and License
Copyright (c) Vitaliy Filippov (vitalif [at] yourcmc.ru), 2019+
Join Vitastor Telegram Chat: https://t.me/vitastor
All server-side code (OSD, Monitor and so on) is licensed under the terms of Vitastor Network Public License 1.1 (VNPL 1.1), a copyleft license based on GNU GPLv3.0 with the additional "Network Interaction" clause which requires opensourcing all programs directly or indirectly interacting with Vitastor through a computer network and expressly designed to be used in conjunction with it ("Proxy Programs"). Proxy Programs may be made public not only under the terms of the same license, but also under the terms of any GPL-Compatible Free Software License, as listed by the Free Software Foundation. This is a stricter copyleft license than the Affero GPL.
Please note that VNPL doesn't require you to open the code of proprietary software running inside a VM if it's not specially designed to be used with Vitastor.
Basically, you can't use the software in a proprietary environment to provide its functionality to users without opensourcing all intermediary components standing between the user and Vitastor or purchasing a commercial license from the author 😀.
Client libraries (cluster_client and so on) are dual-licensed under the same VNPL 1.1 and also GNU GPL 2.0 or later to allow for compatibility with GPLed software like QEMU and fio.
You can find the full text of VNPL-1.1 in the file VNPL-1.1.txt. GPL 2.0 is also included in this repository as GPL-2.0.txt.