Commit Graph

855 Commits (master)

Author SHA1 Message Date
Vitaliy Filippov 294a754c9e Allow write-back by default in NBD & NFS 2023-10-27 01:26:26 +03:00
Vitaliy Filippov 8bfea6e7de Support vitastor_c_create_epoll() in fio driver
Test / test_snapshot_ec (push) Successful in 29s Details
Test / test_move_reappear (push) Successful in 21s Details
Test / test_rm (push) Successful in 21s Details
Test / test_snapshot_down (push) Successful in 27s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m29s Details
Test / test_splitbrain (push) Successful in 20s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_snapshot_chain (push) Successful in 2m35s Details
Test / test_rebalance_verify (push) Successful in 3m22s Details
Test / test_rebalance_verify_imm (push) Successful in 3m22s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_rebalance_verify_ec (push) Successful in 3m56s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m51s Details
Test / test_heal_pg_size_2 (push) Successful in 3m21s Details
Test / test_heal_ec (push) Successful in 5m45s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m43s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m10s Details
Test / test_heal_csum_32k (push) Successful in 6m29s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m22s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m51s Details
Test / test_scrub (push) Successful in 36s Details
Test / test_scrub_zero_osd_2 (push) Successful in 34s Details
Test / test_heal_csum_4k (push) Successful in 5m35s Details
Test / test_scrub_xor (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 1m31s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 42s Details
Test / test_scrub_ec (push) Successful in 20s Details
Test / test_snapshot_chain_ec (push) Successful in 1m23s Details
2023-10-26 22:57:36 +03:00
Vitaliy Filippov bac9e34836 Allow to create vitastor_c with plain epoll without uring :-) 2023-10-26 22:57:36 +03:00
Vitaliy Filippov 8aa4d492c1 Allow to use epoll_manager without ringloop 2023-10-26 22:57:36 +03:00
Vitaliy Filippov 9336ee5476 Correctly free manual "small vector" in cluster_client %-) 2023-10-26 22:57:36 +03:00
Vitaliy Filippov ad30b11519 Add the missing ringloop creation check to vitastor_c_create_uring_json()
Test / test_minsize_1 (push) Failing after 1s Details
Test / test_move_reappear (push) Failing after 1s Details
Test / test_rm (push) Failing after 0s Details
Test / test_snapshot_chain (push) Failing after 0s Details
Test / test_snapshot_chain_ec (push) Failing after 1s Details
Test / test_snapshot_down (push) Failing after 1s Details
Test / test_snapshot_down_ec (push) Failing after 0s Details
Test / test_splitbrain (push) Failing after 1s Details
Test / test_rebalance_verify (push) Failing after 1s Details
Test / test_rebalance_verify_imm (push) Failing after 1s Details
Test / test_rebalance_verify_ec (push) Failing after 1s Details
Test / test_rebalance_verify_ec_imm (push) Failing after 1s Details
Test / test_write (push) Failing after 1s Details
Test / test_write_xor (push) Failing after 1s Details
Test / test_write_no_same (push) Failing after 1s Details
Test / test_heal_pg_size_2 (push) Failing after 1s Details
Test / test_heal_ec (push) Failing after 1s Details
Test / test_heal_csum_32k_dmj (push) Failing after 1s Details
Test / test_heal_csum_32k_dj (push) Failing after 1s Details
Test / test_heal_csum_32k (push) Failing after 1s Details
Test / test_heal_csum_4k_dmj (push) Failing after 1s Details
Test / test_heal_csum_4k_dj (push) Failing after 1s Details
Test / test_heal_csum_4k (push) Failing after 1s Details
Test / test_scrub (push) Failing after 1s Details
Test / test_scrub_zero_osd_2 (push) Failing after 1s Details
Test / test_scrub_xor (push) Failing after 1s Details
Test / test_scrub_pg_size_3 (push) Failing after 1s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Failing after 0s Details
Test / test_scrub_ec (push) Failing after 1s Details
Test / build (push) Successful in 2m35s Details
2023-10-26 18:07:23 +03:00
Vitaliy Filippov a061246997 Do not attempt to initialize QEMU driver via vitastor_c_create_qemu_uring()
Test / test_snapshot_ec (push) Successful in 38s Details
Test / test_minsize_1 (push) Successful in 18s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot_down (push) Failing after 26s Details
Test / test_move_reappear (push) Failing after 52s Details
Test / test_snapshot_down_ec (push) Failing after 27s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_snapshot_chain (push) Successful in 2m28s Details
Test / test_snapshot_chain_ec (push) Successful in 3m6s Details
Test / test_rebalance_verify (push) Successful in 3m30s Details
Test / test_rebalance_verify_imm (push) Successful in 3m29s Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
It doesn't add any compatibility because vitastor_c_uring_register_eventfd()
is added in the same VITASTOR_C_API_VERSION 2.
2023-10-26 17:46:19 +03:00
Vitaliy Filippov 5066e35a49 Fix write-over-delete failing for the very first entry in dirty_db
Test / test_minsize_1 (push) Successful in 17s Details
Test / test_snapshot_ec (push) Successful in 38s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot_down (push) Failing after 27s Details
Test / test_move_reappear (push) Failing after 53s Details
Test / test_snapshot_down_ec (push) Failing after 27s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_snapshot_chain (push) Successful in 2m28s Details
Test / test_snapshot_chain_ec (push) Successful in 2m59s Details
Test / test_rebalance_verify (push) Successful in 3m22s Details
Test / test_rebalance_verify_imm (push) Successful in 3m21s Details
Test / test_write (push) Successful in 42s Details
Test / test_write_xor (push) Successful in 48s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_rebalance_verify_ec (push) Successful in 4m41s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m8s Details
Test / test_heal_pg_size_2 (push) Successful in 4m17s Details
Test / test_heal_ec (push) Successful in 4m33s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m23s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m18s Details
Test / test_heal_csum_32k (push) Successful in 6m44s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m53s Details
Test / test_scrub (push) Successful in 1m11s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m5s Details
Test / test_scrub_xor (push) Successful in 1m1s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m58s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m2s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m41s Details
Test / test_scrub_ec (push) Successful in 37s Details
Test / test_heal_csum_4k (push) Successful in 6m4s Details
2023-10-21 17:00:14 +03:00
Vitaliy Filippov 93dc31f3fc Fix possible segfault in vitastor-cli ls -l
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m21s Details
Test / test_snapshot_ec (push) Successful in 32s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot_down (push) Failing after 26s Details
Test / test_move_reappear (push) Failing after 52s Details
Test / test_splitbrain (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Failing after 26s Details
Test / test_snapshot_chain (push) Successful in 2m22s Details
Test / test_snapshot_chain_ec (push) Successful in 2m44s Details
Test / test_rebalance_verify (push) Successful in 3m20s Details
Test / test_rebalance_verify_imm (push) Successful in 3m20s Details
Test / test_write (push) Successful in 36s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_rebalance_verify_ec (push) Successful in 3m59s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m49s Details
Test / test_write_xor (push) Failing after 3m11s Details
Test / test_heal_pg_size_2 (push) Failing after 4m40s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m13s Details
Test / test_heal_ec (push) Successful in 5m32s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m1s Details
Test / test_heal_csum_32k (push) Successful in 6m54s Details
Test / test_scrub (push) Successful in 1m28s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m1s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m7s Details
Test / test_heal_csum_4k_dj (push) Successful in 7m25s Details
Test / test_scrub_xor (push) Successful in 59s Details
Test / test_heal_csum_4k (push) Successful in 6m39s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 59s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m6s Details
Test / test_scrub_ec (push) Successful in 35s Details
2023-10-18 11:11:41 +03:00
Vitaliy Filippov f245b56176 Fix another possible reenterability issue in cluster_client
Test / test_interrupted_rebalance_ec_imm (push) Successful in 2m9s Details
Test / test_snapshot_ec (push) Successful in 42s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_down (push) Failing after 27s Details
Test / test_move_reappear (push) Failing after 50s Details
Test / test_snapshot_down_ec (push) Failing after 26s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_chain (push) Successful in 2m26s Details
Test / test_snapshot_chain_ec (push) Successful in 2m58s Details
Test / test_rebalance_verify (push) Successful in 3m26s Details
Test / test_rebalance_verify_imm (push) Successful in 3m27s Details
Test / test_write (push) Successful in 42s Details
Test / test_write_xor (push) Successful in 51s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_rebalance_verify_ec (push) Successful in 4m56s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m11s Details
Test / test_heal_pg_size_2 (push) Successful in 4m18s Details
Test / test_heal_ec (push) Successful in 5m5s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m7s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m14s Details
Test / test_heal_csum_32k (push) Successful in 6m54s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m55s Details
Test / test_scrub (push) Successful in 1m23s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m8s Details
Test / test_scrub_xor (push) Successful in 1m0s Details
Test / test_heal_csum_4k_dj (push) Successful in 7m15s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m7s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m40s Details
Test / test_scrub_ec (push) Successful in 42s Details
Test / test_heal_csum_4k (push) Successful in 6m17s Details
Non-reproducible in QEMU/FIO, only caught during K/V DB debugging
2023-10-08 11:02:53 +03:00
Vitaliy Filippov 38db53f5ee Implement client writeback cache
- Disabled by default, enable with client_enable_writeback=true
- Even then only enabled in FIO when -direct is disabled and in QEMU when
  block device cache is enabled in settings
- Can also be enabled in other clients like vitastor-cli using parameter
  client_writeback_allowed=true, but not recommended
2023-09-16 17:52:17 +03:00
Vitaliy Filippov cd543a90bc Prevent stack overflows in cli_merge with CAS and writeback cache 2023-09-16 17:52:17 +03:00
Vitaliy Filippov f600cc07b0 Autosync in blockstore every autosync_writes, too 2023-09-16 17:52:17 +03:00
Vitaliy Filippov 6a8e530e6b Add FIXME to timerfd_manager 2023-09-16 17:52:17 +03:00
Vitaliy Filippov 5cadb170b9 Fix possible OSD crash during sync due to missing min_flushed_journal_sector reset 2023-09-16 17:52:17 +03:00
Vitaliy Filippov e72d4ed1d4 Remove unused bs_sync fields 2023-09-16 17:52:17 +03:00
Vitaliy Filippov ff479a102d Make MON filter OSDs by block layout to prevent "refusing to start PGs of this pool" errors on misconfiguration 2023-09-16 17:52:17 +03:00
Vitaliy Filippov 27d0d5b06a Reads do not have to wait for buffer flushes anymore 2023-09-16 17:52:17 +03:00
Vitaliy Filippov 33950c1ec8 Fix fio_sec_osd attr_len
Test / test_failure_domain (push) Has been cancelled Details
Test / test_snapshot (push) Has been cancelled Details
Test / test_snapshot_ec (push) Has been cancelled Details
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot_chain (push) Has been cancelled Details
Test / test_snapshot_chain_ec (push) Has been cancelled Details
Test / test_snapshot_down (push) Has been cancelled Details
Test / test_snapshot_down_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / build (push) Has been cancelled Details
2023-09-16 17:49:10 +03:00
Vitaliy Filippov cc0fdc6253 Remove erroneous block_size mismatch warnings on pools without matching PGs
Test / test_snapshot_ec (push) Successful in 36s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_move_reappear (push) Failing after 52s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m23s Details
Test / test_snapshot_chain_ec (push) Successful in 2m58s Details
Test / test_rebalance_verify (push) Successful in 3m32s Details
Test / test_rebalance_verify_imm (push) Successful in 3m29s Details
Test / test_write (push) Successful in 52s Details
Test / test_write_xor (push) Successful in 56s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_rebalance_verify_ec (push) Successful in 5m0s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m30s Details
Test / test_heal_ec (push) Successful in 4m6s Details
Test / test_heal_pg_size_2 (push) Failing after 4m19s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m2s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m12s Details
Test / test_heal_csum_32k (push) Successful in 6m24s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m19s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m8s Details
Test / test_scrub (push) Successful in 1m15s Details
Test / test_scrub_xor (push) Successful in 1m8s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m45s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m58s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m9s Details
Test / test_scrub_ec (push) Successful in 42s Details
Test / test_heal_csum_4k (push) Successful in 5m26s Details
2023-09-08 23:19:04 +03:00
Vitaliy Filippov 79ecd59b10 Flush STDOUT and STDERR before exiting from cli to fix Proxmox "Unexpected result"
Test / test_interrupted_rebalance_ec (push) Successful in 1m51s Details
Test / test_snapshot_ec (push) Successful in 33s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot_down (push) Successful in 29s Details
Test / test_snapshot_down_ec (push) Successful in 30s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_snapshot_chain (push) Successful in 2m18s Details
Test / test_snapshot_chain_ec (push) Successful in 2m55s Details
Test / test_rebalance_verify (push) Successful in 3m15s Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s Details
Test / test_write (push) Successful in 47s Details
Test / test_write_xor (push) Successful in 57s Details
Test / test_write_no_same (push) Successful in 17s Details
Test / test_rebalance_verify_ec (push) Successful in 5m3s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m24s Details
Test / test_heal_pg_size_2 (push) Failing after 3m44s Details
Test / test_heal_ec (push) Successful in 4m55s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m48s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m11s Details
Test / test_heal_csum_32k (push) Successful in 6m14s Details
Test / test_scrub (push) Successful in 1m32s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m33s Details
Test / test_scrub_xor (push) Successful in 59s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m7s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m25s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m21s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m46s Details
Test / test_scrub_ec (push) Successful in 53s Details
Test / test_heal_csum_4k (push) Successful in 5m20s Details
2023-09-07 17:30:26 +03:00
Vitaliy Filippov b7d398be5b Fix sscanf validation usage (field count instead of null_byte == 0)
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot_chain (push) Has been cancelled Details
Test / test_snapshot_chain_ec (push) Has been cancelled Details
Test / test_snapshot_down (push) Has been cancelled Details
Test / test_snapshot_down_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / build (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
Test / test_heal_csum_32k_dmj (push) Has been cancelled Details
Test / test_heal_csum_32k_dj (push) Has been cancelled Details
Test / test_heal_csum_32k (push) Has been cancelled Details
Test / test_heal_csum_4k_dmj (push) Has been cancelled Details
Test / test_heal_csum_4k_dj (push) Has been cancelled Details
Test / test_heal_csum_4k (push) Has been cancelled Details
Test / test_scrub (push) Has been cancelled Details
Test / test_scrub_zero_osd_2 (push) Has been cancelled Details
Test / test_scrub_xor (push) Has been cancelled Details
Test / test_scrub_pg_size_3 (push) Has been cancelled Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Has been cancelled Details
Test / test_scrub_ec (push) Has been cancelled Details
2023-09-07 02:34:35 +03:00
Vitaliy Filippov 85e9f67d9d Add supported_truncate_flags
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m19s Details
Test / test_rm (push) Successful in 16s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_snapshot_chain (push) Successful in 2m13s Details
Test / test_snapshot_chain_ec (push) Successful in 2m54s Details
Test / test_rebalance_verify (push) Successful in 3m25s Details
Test / test_rebalance_verify_imm (push) Successful in 3m22s Details
Test / test_write (push) Successful in 52s Details
Test / test_write_xor (push) Successful in 55s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_rebalance_verify_ec (push) Successful in 4m50s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 6m8s Details
Test / test_heal_ec (push) Successful in 5m1s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m32s Details
Test / test_heal_csum_32k (push) Successful in 6m14s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m23s Details
Test / test_scrub (push) Successful in 1m27s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m24s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m15s Details
Test / test_scrub_xor (push) Successful in 58s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m1s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 55s Details
Test / test_scrub_ec (push) Successful in 53s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m24s Details
Test / test_heal_csum_4k (push) Successful in 4m44s Details
Test / test_move_reappear (push) Successful in 20s Details
Test / test_heal_pg_size_2 (push) Successful in 4m1s Details
2023-09-06 17:37:52 +03:00
Vitaliy Filippov 79c6d6f323 Make QEMU driver compatible with QEMU 8.1
Test / test_snapshot_ec (push) Successful in 31s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 2m2s Details
Test / test_snapshot_chain_ec (push) Successful in 2m44s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m10s Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec (push) Successful in 3m39s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 7m1s Details
Test / test_write (push) Successful in 36s Details
Test / test_write_xor (push) Successful in 1m11s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m1s Details
Test / test_heal_ec (push) Successful in 6m7s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m19s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m7s Details
Test / test_heal_csum_32k (push) Successful in 6m21s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m22s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m40s Details
Test / test_heal_csum_4k (push) Successful in 5m34s Details
Test / test_scrub (push) Successful in 1m2s Details
Test / test_scrub_zero_osd_2 (push) Successful in 46s Details
Test / test_scrub_xor (push) Successful in 38s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m7s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 50s Details
Test / test_scrub_ec (push) Successful in 48s Details
2023-08-24 02:23:55 +03:00
Vitaliy Filippov ae760dbc1d Fix co_truncate size division by BDRV_SECTOR_SIZE 2023-08-24 01:55:35 +03:00
Vitaliy Filippov 65487da4b1 Do not include msgr_rdma.h into messenger.h 2023-08-24 01:55:35 +03:00
Vitaliy Filippov 7862282938 Extract validation to check_rw(), remove duplicate code with OP_SYNC
Test / test_snapshot_ec (push) Successful in 19s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 59s Details
Test / test_snapshot_chain_ec (push) Successful in 1m41s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_rebalance_verify (push) Successful in 3m23s Details
Test / test_rebalance_verify_imm (push) Successful in 3m19s Details
Test / test_rebalance_verify_ec (push) Successful in 5m33s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m28s Details
Test / test_write (push) Successful in 51s Details
Test / test_write_xor (push) Successful in 1m6s Details
Test / test_write_no_same (push) Successful in 18s Details
Test / test_heal_pg_size_2 (push) Successful in 4m26s Details
Test / test_heal_ec (push) Successful in 5m21s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m56s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m2s Details
Test / test_heal_csum_32k (push) Successful in 6m41s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m53s Details
Test / test_heal_csum_4k_dj (push) Successful in 3m58s Details
Test / test_heal_csum_4k (push) Successful in 4m55s Details
Test / test_scrub (push) Successful in 48s Details
Test / test_scrub_zero_osd_2 (push) Successful in 54s Details
Test / test_scrub_xor (push) Successful in 50s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m4s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 46s Details
Test / test_scrub_ec (push) Successful in 47s Details
2023-08-13 23:49:52 +03:00
Vitaliy Filippov 30ce2bd951 Fix buffer insert in cluster_client 2023-08-12 11:08:50 +03:00
Vitaliy Filippov b1a0afd10a Aggregate buffer flushes
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot_chain (push) Successful in 2m21s Details
Test / test_snapshot_chain_ec (push) Successful in 2m56s Details
Test / test_snapshot_down (push) Successful in 25s Details
Test / test_snapshot_down_ec (push) Successful in 23s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_rebalance_verify (push) Successful in 3m39s Details
Test / test_rebalance_verify_imm (push) Successful in 3m35s Details
Test / test_rebalance_verify_ec (push) Successful in 5m6s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 6m8s Details
Test / test_write (push) Successful in 50s Details
Test / test_write_xor (push) Successful in 55s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m14s Details
Test / test_heal_ec (push) Successful in 5m2s Details
Test / test_heal_csum_32k_dmj (push) Successful in 5m1s Details
Test / test_heal_csum_32k_dj (push) Successful in 6m12s Details
Test / test_heal_csum_32k (push) Successful in 6m35s Details
Test / test_heal_csum_4k_dmj (push) Successful in 7m18s Details
Test / test_heal_csum_4k_dj (push) Successful in 6m24s Details
Test / test_heal_csum_4k (push) Successful in 5m12s Details
Test / test_scrub (push) Successful in 1m39s Details
Test / test_scrub_zero_osd_2 (push) Successful in 49s Details
Test / test_scrub_xor (push) Successful in 45s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m19s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 57s Details
Test / test_scrub_ec (push) Successful in 59s Details
2023-08-11 11:26:13 +03:00
Vitaliy Filippov 85b6134910 Return dirty buffers on read in client
Test / test_snapshot_ec (push) Successful in 22s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 1m57s Details
Test / test_snapshot_chain_ec (push) Successful in 2m39s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m4s Details
Test / test_rebalance_verify_imm (push) Successful in 3m7s Details
Test / test_rebalance_verify_ec (push) Successful in 3m26s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 7m35s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 38s Details
Test / test_write_no_same (push) Successful in 21s Details
Test / test_heal_pg_size_2 (push) Successful in 4m58s Details
Test / test_heal_ec (push) Successful in 3m15s Details
Test / test_heal_csum_32k_dmj (push) Successful in 4m52s Details
Test / test_heal_csum_32k_dj (push) Successful in 5m3s Details
Test / test_heal_csum_32k (push) Successful in 5m26s Details
Test / test_heal_csum_4k_dmj (push) Successful in 6m12s Details
Test / test_heal_csum_4k_dj (push) Successful in 5m9s Details
Test / test_heal_csum_4k (push) Successful in 5m13s Details
Test / test_scrub (push) Successful in 1m24s Details
Test / test_scrub_zero_osd_2 (push) Successful in 1m15s Details
Test / test_scrub_xor (push) Successful in 1m5s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m45s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m11s Details
Test / test_scrub_ec (push) Successful in 56s Details
Required at least to return buffers when they need to be replayed, but until
they are actually replayed
2023-08-09 00:57:08 +03:00
Vitaliy Filippov b1b07a393d Fix incorrect marking op parts as done with snapshots (could probably lead to client hangs) 2023-08-09 00:57:08 +03:00
Vitaliy Filippov 7333022adf Add a third I/O mode: O_DIRECT|O_SYNC, change parameters to data_io/meta_io/journal_io 2023-08-09 00:57:08 +03:00
Vitaliy Filippov 6acf562e01 Release 1.0.0
New features:

- Data and metadata checksums!
  - Metadata checksums are always used with new disk format
  - Data checksums can be turned on with --data_csum_type crc32c for new OSDs
  - Checksum block size can be configured
  - inmemory_metadata now also affects keeping checksums in memory
- Linux page cache I/O caching support which can be enabled separately for
  data, metadata (including checksums) and journal (O_SYNC instead of O_DIRECT)
- Details [here](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/config/layout-osd.en.md#data_csum_type)
- Backwards compatibility is preserved, you can use new OSDs with old disks

Release also includes bug fixes from [0.9.6](https://git.yourcmc.ru/vitalif/vitastor/releases/tag/v0.9.6).

0.9.6 is moved to "-oldstable" repositories and will be available for some additional time.
2023-07-29 18:57:19 +03:00
Vitaliy Filippov 564df2eb5d Support using buffered I/O with O_SYNC instead of direct I/O 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 1a4ceb420d Track used blocks, not object versions 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 21b5124a4b Document data_csum_type and csum_block_size parameters 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 4181add1f4 Remove creepy "metadata copying" during overwrite
Instead of it, just do not verify checksums of currently mutated objects.
When clean data modification during flush runs in parallel to a read request,
that request may read a mix of old and new data. It may even read a mix of
multiple flushed versions if it lasts too long... And attempts to verify it
using temporary copies of metadata make the algorithm too complex and creepy.
2023-07-29 12:17:18 +03:00
Vitaliy Filippov a8464c19af Support keeping checksums on disk (not in memory)
Definitely beneficial for SSD+HDD setups
2023-07-29 12:17:18 +03:00
Vitaliy Filippov 3c8e4c6b72 Use clean_dyn_size for space check 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 8ef4cf89dc Log more details about checksum mismatch in big_writes 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 7bfb1639ea Use find_holes() in flusher for unification 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 628e481c32 Fill journal header to know checksum type & size when dumping journal with --all 2023-07-29 12:17:18 +03:00
Vitaliy Filippov af6f2046fc Fix journal read checksum verification with inmemory_journal=false 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 9357e5293e Call fill_partial_checksum_blocks() correctly in regard to COPY_BUF_CSUM_FILL 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 12851dc07d Wait for journal reads before checking them in clear_incomplete_csum_block_bits 2023-07-29 12:17:18 +03:00
Vitaliy Filippov d6ee1ca17c Use zero checksum size for zero-length writes 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 71674d00cf Fix journal data checksum mangling on corrupted block overwrite 2023-07-29 12:17:18 +03:00
Vitaliy Filippov ddb078d5a7 Check journal entry size when checking block checksums 2023-07-29 12:17:18 +03:00
Vitaliy Filippov d22d56f90a Fix journal data checksum verification on start 2023-07-29 12:17:18 +03:00
Vitaliy Filippov eb1331a079 Add more details to "journal entry data is corrupt" messages 2023-07-29 12:17:18 +03:00
Vitaliy Filippov c5274f655b ...and partially remove the perversion with bitmap inlining 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 45e07d6294 Sadly we have to refcount dyn_data... 2023-07-29 12:17:18 +03:00
Vitaliy Filippov a8ee391e05 Fix clean block checksum read 2023-07-29 12:17:18 +03:00
Vitaliy Filippov de48fa3fd2 Allow to forcibly set meta_format 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 874a766b62 Rename meta_version to meta_format 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 384bd8e28f Support old metadata format in vitastor-disk dump-meta 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 430994f48a Fix journal big_write simple reads after checksum changes 2023-07-29 12:17:18 +03:00
Vitaliy Filippov b909d81f41 Fix bitmap-granular checksums 2023-07-29 12:17:18 +03:00
Vitaliy Filippov e42975ffd1 Fix wait_journal_count not being zeroed 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 93778324e5 Rewrite and fix find_holes into a more obvious version 2023-07-29 12:17:18 +03:00
Vitaliy Filippov eeb6727170 Fix missing checksum read offset 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 92c6e16eba Fix checksum verification in big_write journal reads 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 213a9ccb4d Verify checksums during journal reads 2023-07-29 12:17:18 +03:00
Vitaliy Filippov a166147110 Add backwards compatibility with non-checksum metadata and journal formats 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 7d532880c3 Implement large csum_block_size support (more than 4k) + refactor blockstore_flush 2023-07-29 12:17:18 +03:00
Vitaliy Filippov 0b0405d115 Implement bitmap-granular (4k) metadata & data checksums 2023-07-29 12:17:18 +03:00
Vitaliy Filippov e651c93a90 Release 0.9.6
Test / test_interrupted_rebalance (push) Successful in 1m59s Details
Test / test_interrupted_rebalance_imm (push) Successful in 3m41s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m29s Details
Test / test_failure_domain (push) Successful in 50s Details
Test / test_snapshot (push) Successful in 45s Details
Test / test_snapshot_ec (push) Successful in 23s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_rm (push) Successful in 15s Details
Test / test_snapshot_chain (push) Successful in 2m23s Details
Test / test_snapshot_chain_ec (push) Successful in 3m1s Details
Test / test_snapshot_down (push) Successful in 31s Details
Test / test_snapshot_down_ec (push) Successful in 32s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_rebalance_verify (push) Successful in 3m34s Details
Test / test_rebalance_verify_imm (push) Successful in 3m31s Details
Test / test_rebalance_verify_ec (push) Successful in 5m14s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m14s Details
Test / test_write (push) Successful in 44s Details
Test / test_write_xor (push) Successful in 54s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m38s Details
Test / test_heal_ec (push) Successful in 3m56s Details
Test / test_scrub (push) Successful in 36s Details
Test / test_scrub_zero_osd_2 (push) Successful in 35s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 46s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 25s Details
Test / test_scrub_ec (push) Successful in 24s Details
- Fix vitastor-disk partition zeroing (sometimes it was writing garbage instead of zeroes)
- Fix incorrect EC space statistics in `vitastor-cli status`
- Several bug fixes for NFS:
  - Add . and .. in NFS directory listings
  - Return FILE_SYNC from NFS writes if immediate_commit is enabled
  - Return the same "verifier" in NFS COMMIT as in NFS WRITE
  - Make parallel NFS extending writes work correctly, without conflicts
  - Handle parallel NFS extending writes without imposing extra load on etcd
- Support UTF-8 in vitastor-cli table output
- Also allow "0" and "no" as false for inmemory_metadata and inmemory_journal
- Use HDD defaults for HDD-only in automatic `vitastor-disk prepare` mode
2023-07-29 10:54:00 +03:00
Vitaliy Filippov 988e90be69 Fix vitastor-disk partition zeroing (it was writing random garbage instead of zeroes :D)
Test / test_interrupted_rebalance (push) Successful in 1m59s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m38s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m12s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m18s Details
Test / test_failure_domain (push) Successful in 8s Details
Test / test_snapshot (push) Successful in 19s Details
Test / test_snapshot_ec (push) Successful in 20s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 18s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m3s Details
Test / test_snapshot_chain_ec (push) Successful in 1m25s Details
Test / test_snapshot_down (push) Successful in 20s Details
Test / test_snapshot_down_ec (push) Successful in 20s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_rebalance_verify (push) Successful in 3m16s Details
Test / test_rebalance_verify_imm (push) Successful in 3m40s Details
Test / test_rebalance_verify_ec (push) Successful in 4m17s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m20s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_xor (push) Successful in 33s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m54s Details
Test / test_heal_ec (push) Successful in 4m7s Details
Test / test_scrub (push) Successful in 58s Details
Test / test_scrub_zero_osd_2 (push) Successful in 53s Details
Test / test_scrub_xor (push) Successful in 40s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m6s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 37s Details
Test / test_scrub_ec (push) Successful in 39s Details
2023-07-28 12:29:07 +03:00
Vitaliy Filippov 700e0e9bff Handle parallel NFS extending writes without imposing extra load on etcd 2023-07-27 02:26:17 +00:00
Vitaliy Filippov ab0ca7c00f Return FILE_SYNC from NFS writes if immediate_commit is enabled 2023-07-26 02:09:47 +03:00
Vitaliy Filippov f153bc950b Return the same "verifier" in NFS COMMIT as in NFS WRITE
This fixes buffered (not O_DIRECT) NFS writes in Linux - previously they were
hanging in an infinite loop because COMMIT didn't return the same verifier as
previous WRITEs, and NFS kernel client was infinitely retrying the same writes.

Also this probably allows for correct NFS failover, at least for the same
buffered writes, because NFS clients repeat all write requests until a COMMIT
confirms them.
2023-07-26 02:09:47 +03:00
Vitaliy Filippov 425ff8818d Add . and .. in NFS directory listings
MC, for example, hangs with infinite listing retries without them
2023-07-26 02:09:47 +03:00
Vitaliy Filippov 9e287a7778 Handle extending writes correctly in NFS proxy
Previously, multiple parallel writes extending file size through NFS were
racing with each other and triggering deletions of part of the written data

I.e. if you mounted vitastor-nfs and just copied a file into it in MC then
you could end up with only a part of the file actually written
2023-07-26 02:09:43 +03:00
Vitaliy Filippov f52f58b9e9 Support UTF-8 in vitastor-cli table output 2023-07-25 01:48:57 +00:00
Vitaliy Filippov 1fe6b0c0e2 Also allow "0" and "no" as false for inmemory_metadata and inmemory_journal 2023-07-25 01:48:57 +00:00
Vitaliy Filippov e4237e9ed8 Enable HDD defaults for HDD-only in automatic `vitastor-disk prepare` mode
Test / test_interrupted_rebalance (push) Successful in 2m20s Details
Test / test_interrupted_rebalance_imm (push) Failing after 10m5s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m4s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m19s Details
Test / test_failure_domain (push) Successful in 39s Details
Test / test_snapshot (push) Successful in 34s Details
Test / test_snapshot_ec (push) Successful in 23s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m59s Details
Test / test_snapshot_chain_ec (push) Successful in 2m40s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m10s Details
Test / test_rebalance_verify_imm (push) Successful in 3m11s Details
Test / test_rebalance_verify_ec (push) Successful in 3m29s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m54s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_xor (push) Successful in 35s Details
Test / test_write_no_same (push) Successful in 18s Details
Test / test_heal_pg_size_2 (push) Successful in 3m59s Details
Test / test_heal_ec (push) Successful in 5m12s Details
Test / test_scrub (push) Successful in 59s Details
Test / test_scrub_zero_osd_2 (push) Successful in 47s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 38s Details
Test / test_scrub_ec (push) Successful in 22s Details
2023-07-23 02:33:22 +03:00
Vitaliy Filippov 10a5fd6abb Release 0.9.5
Test / test_etcd_fail (push) Successful in 50s Details
Test / test_interrupted_rebalance (push) Successful in 2m27s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m39s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m49s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m29s Details
Test / test_failure_domain (push) Successful in 19s Details
Test / test_snapshot (push) Successful in 31s Details
Test / test_snapshot_ec (push) Successful in 26s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot_chain (push) Successful in 1m49s Details
Test / test_snapshot_chain_ec (push) Successful in 2m51s Details
Test / test_snapshot_down (push) Successful in 26s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 18s Details
Test / test_rebalance_verify (push) Successful in 3m8s Details
Test / test_rebalance_verify_imm (push) Successful in 3m13s Details
Test / test_rebalance_verify_ec (push) Successful in 3m36s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m59s Details
Test / test_write (push) Successful in 48s Details
Test / test_write_xor (push) Successful in 37s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s Details
Test / test_heal_ec (push) Successful in 4m6s Details
Test / test_scrub (push) Successful in 35s Details
Test / test_scrub_zero_osd_2 (push) Successful in 34s Details
Test / test_scrub_xor (push) Successful in 42s Details
Test / test_scrub_pg_size_3 (push) Successful in 52s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 33s Details
Test / test_scrub_ec (push) Successful in 28s Details
A hotfix to 0.9.4 containing only one bugfix: 100% CPU usage in the new QEMU
driver caused by the lack of eventfd reset on io_uring event handling :)
2023-07-21 00:04:41 +03:00
Vitaliy Filippov 1c316ef350 Reset eventfd on every ringloop::loop() 2023-07-21 00:04:41 +03:00
Vitaliy Filippov 0b2d12eef1 Remove has_work, it was unnecessary 2023-07-21 00:04:37 +03:00
Vitaliy Filippov 1c10430ae1 Release 0.9.4
Test / test_interrupted_rebalance (push) Successful in 1m54s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m4s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m40s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m25s Details
Test / test_failure_domain (push) Successful in 15s Details
Test / test_snapshot (push) Successful in 25s Details
Test / test_snapshot_ec (push) Successful in 20s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 1m56s Details
Test / test_snapshot_chain_ec (push) Successful in 2m33s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m3s Details
Test / test_rebalance_verify_imm (push) Successful in 3m2s Details
Test / test_rebalance_verify_ec (push) Successful in 3m13s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 8m35s Details
Test / test_write (push) Successful in 33s Details
Test / test_write_xor (push) Successful in 40s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m25s Details
Test / test_heal_ec (push) Successful in 3m9s Details
Test / test_scrub (push) Successful in 1m0s Details
Test / test_scrub_zero_osd_2 (push) Successful in 46s Details
Test / test_scrub_xor (push) Successful in 1m1s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m55s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m25s Details
Test / test_scrub_ec (push) Successful in 52s Details
- Improve QEMU driver performance by integrating io_uring in it (up to 1.5x total iops improvement)
- Fix QEMU driver deadlocks which started to reproduce in qemu-img after iothread fixes
- Fix `vitastor-cli status` reporting more etcds than actually exists (fix etcd address duplication in config on reload)
- Fix `vitastor-cli ls` crashing on inodes in non-existing pools
- Delete old garbage /pool/stats/ keys for non-existing (deleted) pools
- Reduce memory usage of etcds initialized by make-etcd script
- Fix OSDs almost always crashing on etcd restart due to "revisions were compacted" (support reloading state from etcd)
- Fix a crash and a stall possible mostly in HDD setups with small journal and big (512k, 900k) random writes
- Add notes about HDDs to documentation. You are officially allowed to use HDD-only Vitastor with HGST/Toshiba/EXOS :)
2023-07-19 02:50:30 +03:00
Vitaliy Filippov d0e257ee81 Fix non-existing pool handling in `vitastor-cli ls`
Test / test_interrupted_rebalance (push) Successful in 2m6s Details
Test / test_interrupted_rebalance_imm (push) Successful in 3m11s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m6s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 2m20s Details
Test / test_failure_domain (push) Successful in 22s Details
Test / test_snapshot (push) Successful in 50s Details
Test / test_snapshot_ec (push) Successful in 33s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 1m23s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 2m22s Details
Test / test_snapshot_chain_ec (push) Successful in 3m6s Details
Test / test_snapshot_down (push) Successful in 24s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 19s Details
Test / test_rebalance_verify (push) Successful in 3m28s Details
Test / test_rebalance_verify_imm (push) Successful in 3m27s Details
Test / test_rebalance_verify_ec (push) Successful in 9m10s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 9m29s Details
Test / test_write (push) Successful in 1m36s Details
Test / test_write_xor (push) Successful in 2m17s Details
Test / test_write_no_same (push) Successful in 36s Details
Test / test_heal_pg_size_2 (push) Successful in 6m27s Details
Test / test_heal_ec (push) Successful in 5m53s Details
Test / test_scrub (push) Successful in 44s Details
Test / test_scrub_zero_osd_2 (push) Successful in 35s Details
Test / test_scrub_xor (push) Successful in 36s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m1s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 46s Details
Test / test_scrub_ec (push) Successful in 36s Details
2023-07-18 23:52:02 +03:00
Vitaliy Filippov 9815d70ffc It is impossible to use io_uring with older vitastor-client because it does not have vitastor_c_uring_has_work() 2023-07-18 23:37:53 +03:00
Vitaliy Filippov 4a4627dcab Do not use bool in C library 2023-07-18 23:37:53 +03:00
Vitaliy Filippov ba7427020e Fix deadlocks possible in qemu-img after fixing iothread
Deadlock was caused by switching QEMU coroutines directly inside
vitastor_co_read_bitmap_cb() callback. The correct way is to schedule a BH
/BH is a QEMU term for setImmediate() :)/, same as in read and write callbacks.
2023-07-18 23:32:16 +03:00
Vitaliy Filippov ac7b834af3 Disable journal_no_same_sector_overwrites by default for HDD-only
Test / test_interrupted_rebalance (push) Successful in 1m17s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m29s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m53s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m31s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_snapshot (push) Successful in 20s Details
Test / test_snapshot_ec (push) Successful in 21s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 39s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m10s Details
Test / test_snapshot_chain_ec (push) Successful in 2m9s Details
Test / test_snapshot_down (push) Successful in 21s Details
Test / test_snapshot_down_ec (push) Successful in 22s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_rebalance_verify (push) Successful in 2m56s Details
Test / test_rebalance_verify_imm (push) Successful in 2m53s Details
Test / test_rebalance_verify_ec (push) Successful in 5m39s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m44s Details
Test / test_write (push) Successful in 49s Details
Test / test_write_xor (push) Successful in 59s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m41s Details
Test / test_heal_ec (push) Successful in 3m49s Details
Test / test_scrub (push) Successful in 33s Details
Test / test_scrub_zero_osd_2 (push) Successful in 30s Details
Test / test_scrub_xor (push) Successful in 25s Details
Test / test_scrub_pg_size_3 (push) Successful in 40s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 35s Details
Test / test_scrub_ec (push) Successful in 33s Details
2023-07-10 00:34:35 +03:00
Vitaliy Filippov 57ad4c3636 Add a note about HDD, enable throttling only for hybrid OSDs
Test / test_interrupted_rebalance (push) Successful in 2m4s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m30s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m45s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m23s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_snapshot (push) Successful in 19s Details
Test / test_snapshot_ec (push) Successful in 19s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Failing after 1m32s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m8s Details
Test / test_snapshot_chain_ec (push) Successful in 2m2s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 21s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 2m52s Details
Test / test_rebalance_verify_imm (push) Successful in 2m46s Details
Test / test_rebalance_verify_ec (push) Successful in 5m9s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m5s Details
Test / test_write (push) Successful in 40s Details
Test / test_write_xor (push) Successful in 49s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s Details
Test / test_heal_ec (push) Successful in 4m35s Details
Test / test_scrub (push) Successful in 37s Details
Test / test_scrub_zero_osd_2 (push) Successful in 40s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 40s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 26s Details
Test / test_scrub_ec (push) Successful in 22s Details
2023-07-09 12:45:11 +03:00
Vitaliy Filippov b7e4d0c9bf Fix journal dirty_start position tracking and some debug prints
Fixes two bugs found during HDD testing :-)
1) OSD crashed with "BUG: Attempt to overwrite used offset of the journal" during
   `fio -bs=900k -iodepth=128` test with 16 MB journal
2) OSD stalled during `fio -bs=512k -iodepth=128` test with 64 MB journal
2023-07-09 01:17:55 +03:00
Vitaliy Filippov 161a23c966 Support reloading state when etcd says "revisions were compacted"
Test / test_interrupted_rebalance (push) Successful in 3m9s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m38s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m54s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m36s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_snapshot (push) Successful in 23s Details
Test / test_snapshot_ec (push) Successful in 22s Details
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_move_reappear (push) Successful in 19s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 2m2s Details
Test / test_snapshot_chain_ec (push) Successful in 2m38s Details
Test / test_snapshot_down (push) Successful in 21s Details
Test / test_snapshot_down_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m10s Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec (push) Successful in 3m27s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 6m2s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 45s Details
Test / test_write_no_same (push) Successful in 22s Details
Test / test_heal_pg_size_2 (push) Successful in 4m0s Details
Test / test_heal_ec (push) Successful in 3m52s Details
Test / test_scrub (push) Successful in 1m1s Details
Test / test_scrub_zero_osd_2 (push) Successful in 42s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 45s Details
Test / test_scrub_ec (push) Successful in 26s Details
Before this change, OSDs almost always died when one of the etcds was restarted,
even though the rest of them was still in quorum and the lease was still active
2023-07-07 01:33:48 +03:00
Vitaliy Filippov 45c0694853 Clear etcd_local addresses on reload and also skip duplicates 2023-07-06 00:39:39 +03:00
Vitaliy Filippov 30ac899074 Make QEMU driver compatible with older vitastor_client and with systems without io_uring
Test / test_interrupted_rebalance (push) Successful in 1m45s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m25s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m23s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m37s Details
Test / test_failure_domain (push) Successful in 8s Details
Test / test_snapshot (push) Successful in 20s Details
Test / test_snapshot_ec (push) Successful in 18s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 1m26s Details
Test / test_snapshot_chain_ec (push) Successful in 2m8s Details
Test / test_snapshot_down (push) Successful in 23s Details
Test / test_snapshot_down_ec (push) Successful in 21s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_rebalance_verify (push) Successful in 2m48s Details
Test / test_rebalance_verify_imm (push) Successful in 2m48s Details
Test / test_rebalance_verify_ec (push) Successful in 4m10s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m13s Details
Test / test_write (push) Successful in 1m16s Details
Test / test_write_xor (push) Successful in 38s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 4m10s Details
Test / test_heal_ec (push) Successful in 3m28s Details
Test / test_scrub (push) Successful in 43s Details
Test / test_scrub_zero_osd_2 (push) Successful in 41s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 44s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 38s Details
Test / test_scrub_ec (push) Successful in 36s Details
2023-07-04 15:51:43 +03:00
Vitaliy Filippov 2348d39cf4 Avoid repeated qemu_uring_handlers, add 2.0-2.7 compatibility 2023-07-04 00:28:23 +03:00
Vitaliy Filippov 3de7929fe5 Integrate v2 - direct epoll 2023-07-04 00:28:23 +03:00
Vitaliy Filippov 07b2196bc2 Integrate QEMU driver with io_uring 2023-07-04 00:28:23 +03:00
Vitaliy Filippov a612cdca47 Release 0.9.3
- Add patch for libvirt 9.0
- Add support for Proxmox VE 8.0
- Fix compatibility of the QEMU driver with iothread (QEMU rebuilds are coming)
- Fix vitastor-cli rm-data/rm/merge hanging when some OSDs are down.
  Allow deletions in unclean cluster at the cost of some data possibly
  "reappearing" when those OSDs start back. In that case you can just repeat
  the deletion request using rm-data.
- A bunch of bug fixes for snapshots:
  - Fix snapshot reads often not working at all with snapshot chain size > 2
  - Fix optimized snapshot data merge (children to parent)
  - Fix updating of image name index key during optimized merge
  - Fix auto-selection preventing the use of optimized merge with only 1 snapshot
  - Fix incorrect CAS retries during snapshot merge
  - Fix snapshot merge progress reporting
- Fix primary_read bitmap buffers use-after-free which could lead to
  incorrect allocation map reads
- Remove /usr/local/bin path from make-etcd
- Some documentation fixes
2023-07-01 00:25:58 +03:00
Vitaliy Filippov c8d61568b5 Fix primary_read bitmap buffers being freed too early (use-after-free)
Test / test_etcd_fail (push) Successful in 1m43s Details
Test / test_interrupted_rebalance (push) Successful in 1m23s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m33s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m39s Details
Test / test_failure_domain (push) Successful in 10s Details
Test / test_snapshot (push) Successful in 19s Details
Test / test_snapshot_ec (push) Successful in 17s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 1m7s Details
Test / test_snapshot_chain_ec (push) Successful in 2m1s Details
Test / test_snapshot_down (push) Successful in 22s Details
Test / test_snapshot_down_ec (push) Successful in 20s Details
Test / test_splitbrain (push) Successful in 12s Details
Test / test_rebalance_verify (push) Successful in 2m51s Details
Test / test_rebalance_verify_imm (push) Successful in 2m49s Details
Test / test_rebalance_verify_ec (push) Successful in 4m6s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m3s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_xor (push) Successful in 34s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_heal_pg_size_2 (push) Successful in 3m11s Details
Test / test_heal_ec (push) Successful in 3m54s Details
Test / test_scrub (push) Successful in 44s Details
Test / test_scrub_zero_osd_2 (push) Successful in 30s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 37s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 32s Details
Test / test_scrub_ec (push) Successful in 1m31s Details
2023-06-30 12:47:45 +03:00
Vitaliy Filippov 84ed3c6395 Fix CAS retries during snapshot merge
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Failing after 10m10s Details
Test / test_interrupted_rebalance (push) Successful in 2m46s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m35s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m45s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m53s Details
Test / test_failure_domain (push) Successful in 7s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 29s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_move_reappear (push) Failing after 49s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m35s Details
Test / test_snapshot_chain_ec (push) Successful in 2m9s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_rebalance_verify (push) Successful in 3m0s Details
Test / test_rebalance_verify_imm (push) Successful in 3m5s Details
Test / test_rebalance_verify_ec (push) Successful in 3m20s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m24s Details
Test / test_write (push) Successful in 34s Details
Test / test_write_xor (push) Successful in 38s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m0s Details
Test / test_heal_ec (push) Successful in 3m58s Details
Test / test_scrub (push) Successful in 58s Details
Test / test_scrub_zero_osd_2 (push) Successful in 38s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 35s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 30s Details
Test / test_scrub_ec (push) Successful in 23s Details
2023-06-30 02:30:23 +03:00
Vitaliy Filippov a7b57386c0 Do not print last subcommand result twice during "inverse" snapshot merge
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 1m28s Details
Test / test_interrupted_rebalance (push) Successful in 1m45s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m46s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m44s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m30s Details
Test / test_failure_domain (push) Successful in 8s Details
Test / test_snapshot (push) Successful in 30s Details
Test / test_snapshot_ec (push) Successful in 29s Details
Test / test_minsize_1 (push) Failing after 19s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot_chain (push) Successful in 1m57s Details
Test / test_snapshot_chain_ec (push) Successful in 2m42s Details
Test / test_splitbrain (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m23s Details
Test / test_rebalance_verify_imm (push) Successful in 3m17s Details
Test / test_rebalance_verify_ec (push) Successful in 4m5s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m50s Details
Test / test_write (push) Successful in 41s Details
Test / test_write_xor (push) Failing after 3m17s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m12s Details
Test / test_heal_ec (push) Successful in 4m13s Details
Test / test_scrub (push) Successful in 38s Details
Test / test_scrub_zero_osd_2 (push) Successful in 36s Details
Test / test_scrub_xor (push) Successful in 33s Details
Test / test_scrub_pg_size_3 (push) Successful in 46s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 35s Details
Test / test_scrub_ec (push) Successful in 33s Details
2023-06-30 02:07:10 +03:00
Vitaliy Filippov 9d4ea5f764 Fix inverse parent selection which prevented the use of optimized merge in case of only 1 snapshot 2023-06-30 01:39:11 +03:00
Vitaliy Filippov 000e4944ec Remove "inverse parent" image name index key from etcd during snapshot merge 2023-06-30 01:23:30 +03:00
Vitaliy Filippov 8426616d89 Warn about unfinished deletions in rm-data
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 1m46s Details
Test / test_interrupted_rebalance (push) Successful in 2m34s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m50s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m35s Details
Test / test_interrupted_rebalance_ec_imm (push) Failing after 10m6s Details
Test / test_failure_domain (push) Successful in 12s Details
Test / test_snapshot (push) Successful in 21s Details
Test / test_snapshot_ec (push) Successful in 27s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Failing after 48s Details
Test / test_rm (push) Successful in 13s Details
Test / test_snapshot_chain (push) Successful in 1m35s Details
Test / test_snapshot_chain_ec (push) Successful in 2m11s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_rebalance_verify (push) Successful in 3m4s Details
Test / test_rebalance_verify_imm (push) Successful in 3m0s Details
Test / test_rebalance_verify_ec (push) Successful in 3m25s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m58s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 34s Details
Test / test_write_no_same (push) Successful in 12s Details
Test / test_heal_pg_size_2 (push) Successful in 4m3s Details
Test / test_heal_ec (push) Successful in 3m53s Details
Test / test_scrub (push) Successful in 26s Details
Test / test_scrub_zero_osd_2 (push) Successful in 35s Details
Test / test_scrub_xor (push) Successful in 35s Details
Test / test_scrub_pg_size_3 (push) Successful in 45s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 32s Details
Test / test_scrub_ec (push) Successful in 25s Details
2023-06-30 01:18:25 +03:00
Vitaliy Filippov 1a841344ec Print progress of all operations during snapshot merge 2023-06-30 01:13:47 +03:00
Vitaliy Filippov 8603b5cb1d Do not hang on inactive OSDs during delete, report and skip them instead
Test / test_create_nomaxid (push) Successful in 8s Details
Test / test_etcd_fail (push) Successful in 46s Details
Test / test_interrupted_rebalance (push) Successful in 3m13s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m30s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m18s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m34s Details
Test / test_failure_domain (push) Successful in 7s Details
Test / test_snapshot (push) Successful in 29s Details
Test / test_snapshot_ec (push) Successful in 32s Details
Test / test_minsize_1 (push) Successful in 12s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot_chain (push) Successful in 1m37s Details
Test / test_snapshot_chain_ec (push) Successful in 2m21s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 3m14s Details
Test / test_rebalance_verify_imm (push) Successful in 3m8s Details
Test / test_rebalance_verify_ec (push) Successful in 3m18s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m20s Details
Test / test_write (push) Successful in 32s Details
Test / test_write_xor (push) Successful in 43s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m55s Details
Test / test_heal_ec (push) Successful in 4m2s Details
Test / test_scrub (push) Successful in 56s Details
Test / test_scrub_zero_osd_2 (push) Successful in 51s Details
Test / test_scrub_xor (push) Successful in 29s Details
Test / test_scrub_pg_size_3 (push) Successful in 1m2s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 37s Details
Test / test_scrub_ec (push) Successful in 32s Details
2023-06-30 00:15:16 +03:00
Vitaliy Filippov 878ccbb6ea Fix snapshot chain "down-merge" ("up-merge" worked well...)
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_interrupted_rebalance (push) Successful in 7m46s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m31s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m42s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m22s Details
Test / test_failure_domain (push) Successful in 7s Details
Test / test_snapshot (push) Successful in 18s Details
Test / test_snapshot_ec (push) Successful in 20s Details
Test / test_minsize_1 (push) Successful in 11s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rm (push) Successful in 11s Details
Test / test_snapshot_chain (push) Successful in 1m11s Details
Test / test_snapshot_chain_ec (push) Successful in 2m13s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 3m26s Details
Test / test_rebalance_verify_imm (push) Successful in 3m27s Details
Test / test_rebalance_verify_ec (push) Successful in 4m23s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m32s Details
Test / test_write (push) Successful in 31s Details
Test / test_write_xor (push) Successful in 33s Details
Test / test_write_no_same (push) Successful in 13s Details
Test / test_heal_pg_size_2 (push) Successful in 3m37s Details
Test / test_heal_ec (push) Successful in 4m43s Details
Test / test_scrub (push) Successful in 45s Details
Test / test_scrub_zero_osd_2 (push) Successful in 37s Details
Test / test_scrub_xor (push) Successful in 39s Details
Test / test_scrub_pg_size_3 (push) Successful in 55s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 33s Details
Test / test_scrub_ec (push) Successful in 32s Details
2023-06-29 00:47:21 +03:00
Vitaliy Filippov 63c2b9832c Fix chained (snapshot) reads often not working at all with chain size > 2
Test / test_change_pg_count (push) Successful in 42s Details
Test / test_change_pg_count_ec (push) Successful in 36s Details
Test / test_change_pg_size (push) Successful in 9s Details
Test / test_create_nomaxid (push) Successful in 8s Details
Test / test_etcd_fail (push) Successful in 1m6s Details
Test / test_failure_domain (push) Successful in 10s Details
Test / test_interrupted_rebalance (push) Successful in 1m52s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m49s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m2s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m19s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 3m10s Details
Test / test_rebalance_verify_imm (push) Successful in 2m56s Details
Test / test_rebalance_verify_ec (push) Successful in 3m6s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m9s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 27s Details
Test / test_splitbrain (push) Successful in 25s Details
Test / test_write (push) Successful in 1m30s Details
Test / test_write_xor (push) Successful in 2m14s Details
Test / test_write_no_same (push) Successful in 20s Details
Test / test_heal_pg_size_2 (push) Successful in 4m5s Details
Test / test_heal_ec (push) Successful in 4m11s Details
Test / test_scrub (push) Successful in 48s Details
Test / test_scrub_zero_osd_2 (push) Successful in 32s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 37s Details
Test / test_scrub_ec (push) Successful in 41s Details
2023-06-28 18:54:03 +03:00
Vitaliy Filippov a11ca56fb1 Fix compatibility of the QEMU driver with iothread
Test / test_change_pg_count_ec (push) Successful in 55s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 57s Details
Test / test_failure_domain (push) Successful in 10s Details
Test / test_interrupted_rebalance (push) Successful in 1m53s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m31s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m39s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m46s Details
Test / test_minsize_1 (push) Successful in 55s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 3m13s Details
Test / test_rebalance_verify_imm (push) Successful in 2m52s Details
Test / test_rebalance_verify_ec (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m25s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot (push) Successful in 36s Details
Test / test_snapshot_ec (push) Successful in 26s Details
Test / test_splitbrain (push) Successful in 13s Details
Test / test_write (push) Successful in 38s Details
Test / test_write_xor (push) Successful in 1m50s Details
Test / test_write_no_same (push) Successful in 14s Details
Test / test_heal_pg_size_2 (push) Successful in 4m17s Details
Test / test_heal_ec (push) Successful in 4m18s Details
Test / test_scrub (push) Successful in 50s Details
Test / test_scrub_zero_osd_2 (push) Successful in 51s Details
Test / test_scrub_xor (push) Successful in 39s Details
Test / test_scrub_pg_size_3 (push) Successful in 53s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 42s Details
Test / test_scrub_ec (push) Successful in 28s Details
2023-06-21 02:11:28 +03:00
Vitaliy Filippov b84927b340 Fix \n in nbd_proxy
Test / test_change_pg_count (push) Successful in 41s Details
Test / test_change_pg_count_ec (push) Successful in 37s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 9s Details
Test / test_etcd_fail (push) Successful in 1m9s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_interrupted_rebalance (push) Successful in 1m54s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m47s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m17s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m26s Details
Test / test_minsize_1 (push) Successful in 19s Details
Test / test_move_reappear (push) Successful in 50s Details
Test / test_rebalance_verify (push) Successful in 3m6s Details
Test / test_rebalance_verify_imm (push) Successful in 3m3s Details
Test / test_rebalance_verify_ec (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m55s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 24s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_write (push) Successful in 1m38s Details
Test / test_write_xor (push) Successful in 2m13s Details
Test / test_write_no_same (push) Successful in 23s Details
Test / test_heal_pg_size_2 (push) Successful in 4m5s Details
Test / test_scrub (push) Successful in 42s Details
Test / test_scrub_zero_osd_2 (push) Successful in 33s Details
Test / test_scrub_xor (push) Successful in 31s Details
Test / test_scrub_pg_size_3 (push) Successful in 40s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 34s Details
Test / test_scrub_ec (push) Successful in 30s Details
2023-06-19 01:48:58 +03:00
Vitaliy Filippov 926be372fd Release 0.9.2
Test / test_change_pg_count (push) Successful in 35s Details
Test / test_change_pg_count_ec (push) Successful in 34s Details
Test / test_change_pg_size (push) Successful in 9s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 1m1s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_interrupted_rebalance (push) Successful in 1m26s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m15s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m31s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m32s Details
Test / test_minsize_1 (push) Successful in 13s Details
Test / test_rebalance_verify (push) Successful in 2m49s Details
Test / test_rebalance_verify_imm (push) Successful in 2m45s Details
Test / test_rebalance_verify_ec (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m3s Details
Test / test_rm (push) Successful in 12s Details
Test / test_snapshot (push) Successful in 24s Details
Test / test_snapshot_ec (push) Successful in 35s Details
Test / test_splitbrain (push) Successful in 21s Details
Test / test_write (push) Successful in 1m10s Details
Test / test_write_xor (push) Successful in 2m23s Details
Test / test_write_no_same (push) Successful in 19s Details
Test / test_heal_pg_size_2 (push) Successful in 3m58s Details
Test / test_heal_ec (push) Successful in 4m8s Details
Test / test_scrub (push) Successful in 1m3s Details
Test / test_scrub_zero_osd_2 (push) Successful in 36s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 51s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 32s Details
Test / test_scrub_ec (push) Successful in 43s Details
- Measure and report scrub I/O statistics in vitastor-cli status
- Make aggregated statistics in vitastor-cli status much smoother
  (first derive, then sum instead of first summing and then deriving)
- Fix an old rare bug leading to journal corruption
  (try to use scrub if you think you're affected...)
- Do not start EC PGs without at least <data chunks> OSDs in each old set
  (prevents spurious read errors with EC during reconnections/restarts)
- Fix failed assert(!scrub_list_op) on OSD restart with pending scrubs
- Fix future planned scrubs not starting because of incorrect time comparison
- Build packages for Debian 12 (Bookworm)
2023-06-18 19:44:33 +03:00
Vitaliy Filippov c74a424930 Report scrub I/O in vitastor-cli status 2023-06-17 21:11:21 +03:00
Vitaliy Filippov 32f2c4dd27 Measure scrub statistics 2023-06-17 20:56:26 +03:00
Vitaliy Filippov 3ad16b9a1a Fix auto_scrubs not starting because of < vs <= =))
Test / test_change_pg_count (push) Successful in 41s Details
Test / test_change_pg_count_ec (push) Successful in 36s Details
Test / test_change_pg_size (push) Successful in 9s Details
Test / test_create_nomaxid (push) Successful in 9s Details
Test / test_etcd_fail (push) Successful in 1m20s Details
Test / test_failure_domain (push) Successful in 12s Details
Test / test_interrupted_rebalance (push) Successful in 2m1s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m55s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m48s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m32s Details
Test / test_move_reappear (push) Successful in 51s Details
Test / test_rebalance_verify (push) Successful in 3m19s Details
Test / test_rebalance_verify_imm (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec (push) Successful in 3m21s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m4s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot (push) Successful in 23s Details
Test / test_snapshot_ec (push) Successful in 26s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_write (push) Successful in 1m35s Details
Test / test_write_xor (push) Successful in 2m29s Details
Test / test_write_no_same (push) Successful in 29s Details
Test / test_heal_pg_size_2 (push) Successful in 4m11s Details
Test / test_heal_ec (push) Successful in 5m4s Details
Test / test_scrub (push) Successful in 55s Details
Test / test_scrub_zero_osd_2 (push) Successful in 41s Details
Test / test_scrub_xor (push) Successful in 37s Details
Test / test_scrub_pg_size_3 (push) Successful in 57s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 46s Details
Test / test_scrub_ec (push) Successful in 31s Details
2023-06-17 17:32:21 +03:00
Vitaliy Filippov 1c2df841c2 Fix failed assert(!scrub_list_op) on OSD restart with pending scrubs
Test / test_change_pg_count (push) Successful in 38s Details
Test / test_change_pg_count_ec (push) Successful in 38s Details
Test / test_change_pg_size (push) Successful in 9s Details
Test / test_create_nomaxid (push) Successful in 8s Details
Test / test_etcd_fail (push) Successful in 53s Details
Test / test_failure_domain (push) Successful in 15s Details
Test / test_interrupted_rebalance (push) Successful in 2m3s Details
Test / test_interrupted_rebalance_imm (push) Successful in 2m55s Details
Test / test_interrupted_rebalance_ec (push) Successful in 2m43s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m31s Details
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_move_reappear (push) Successful in 17s Details
Test / test_rebalance_verify (push) Successful in 3m9s Details
Test / test_rebalance_verify_imm (push) Successful in 3m10s Details
Test / test_rebalance_verify_ec (push) Successful in 3m15s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 4m45s Details
Test / test_rm (push) Successful in 18s Details
Test / test_snapshot (push) Successful in 35s Details
Test / test_snapshot_ec (push) Successful in 18s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_write (push) Successful in 35s Details
Test / test_write_xor (push) Successful in 1m12s Details
Test / test_write_no_same (push) Successful in 15s Details
Test / test_heal_pg_size_2 (push) Successful in 3m41s Details
Test / test_scrub (push) Successful in 33s Details
Test / test_scrub_zero_osd_2 (push) Successful in 29s Details
Test / test_scrub_xor (push) Successful in 30s Details
Test / test_scrub_pg_size_3 (push) Successful in 51s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 31s Details
Test / test_scrub_ec (push) Successful in 26s Details
2023-06-17 17:02:54 +03:00
Vitaliy Filippov aa5dacc7a9 Do not start EC PGs without at least pg_data_size connections to old OSDs from each set
Test / test_change_pg_count (push) Successful in 36s Details
Test / test_change_pg_count_ec (push) Successful in 38s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 9s Details
Test / test_etcd_fail (push) Successful in 1m13s Details
Test / test_failure_domain (push) Successful in 11s Details
Test / test_interrupted_rebalance (push) Successful in 1m51s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m43s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m47s Details
Test / test_minsize_1 (push) Successful in 43s Details
Test / test_move_reappear (push) Successful in 43s Details
Test / test_rebalance_verify (push) Successful in 3m16s Details
Test / test_rebalance_verify_imm (push) Successful in 3m9s Details
Test / test_rebalance_verify_ec (push) Successful in 3m8s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 3m10s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot (push) Successful in 22s Details
Test / test_snapshot_ec (push) Successful in 25s Details
Test / test_splitbrain (push) Successful in 15s Details
Test / test_write (push) Successful in 1m44s Details
Test / test_write_xor (push) Successful in 2m29s Details
Test / test_write_no_same (push) Successful in 22s Details
Test / test_heal_pg_size_2 (push) Successful in 4m37s Details
Test / test_heal_ec (push) Successful in 4m4s Details
Test / test_scrub (push) Successful in 48s Details
Test / test_scrub_zero_osd_2 (push) Successful in 41s Details
Test / test_scrub_xor (push) Successful in 39s Details
Test / test_scrub_pg_size_3 (push) Successful in 47s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 41s Details
Test / test_scrub_ec (push) Successful in 34s Details
2023-06-17 02:16:30 +03:00
Vitaliy Filippov 4fdc49bdc7 Add another assert-type check (it does not fire, just as a safety measure for the future) 2023-06-17 00:07:22 +03:00
Vitaliy Filippov 86b4682975 Put get_trim_pos into the "critical section". Fixes rare journal corruption issue
The consequence of this issue was that in some very rare cases (only reproduced
under load in CI when running 4+ tests in parallel) small write data written to
journal could overwrite journal entries.

Also add an assert-type safety check to be able to catch this issue in the
future again in case of a regression.
2023-06-17 00:06:42 +03:00
Vitaliy Filippov bdd48e4cf1 Release 0.9.1
Test / test_change_pg_count (push) Successful in 35s Details
Test / test_change_pg_count_ec (push) Successful in 32s Details
Test / test_change_pg_size (push) Successful in 8s Details
Test / test_create_nomaxid (push) Successful in 6s Details
Test / test_etcd_fail (push) Successful in 49s Details
Test / test_failure_domain (push) Successful in 9s Details
Test / test_interrupted_rebalance (push) Successful in 1m4s Details
Test / test_interrupted_rebalance_imm (push) Successful in 57s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m2s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 53s Details
Test / test_minsize_1 (push) Successful in 14s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 1m45s Details
Test / test_rebalance_verify_imm (push) Successful in 1m41s Details
Test / test_rebalance_verify_ec (push) Successful in 1m53s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 1m48s Details
Test / test_rm (push) Successful in 10s Details
Test / test_snapshot (push) Successful in 14s Details
Test / test_snapshot_ec (push) Successful in 16s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_write (push) Successful in 49s Details
Test / test_write_xor (push) Successful in 1m1s Details
Test / test_write_no_same (push) Successful in 11s Details
Test / test_heal_pg_size_2 (push) Successful in 3m12s Details
Test / test_scrub (push) Successful in 28s Details
Test / test_scrub_zero_osd_2 (push) Successful in 22s Details
Test / test_scrub_xor (push) Successful in 34s Details
Test / test_scrub_pg_size_3 (push) Successful in 25s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 26s Details
Test / test_scrub_ec (push) Successful in 24s Details
- Fix "Client XX command out of sync" messages sometimes happening on OSD reconnections
- Fix a bug where EC reads parallel with writes to the same object failed with -ERANGE error
- Slightly reduce the amount of metadata writes during journal flushing
- Correctly unmap NBD volumes when Proxmox forces map_volume use (with SWTPM and maybe some other cases)
2023-06-10 11:42:49 +03:00
Vitaliy Filippov f9fbea25a4 Remove double write when old and new locations are in the same metadata block
Also add another metadata entry fool-safety check which, ideally, will never fire %)
2023-06-03 00:47:10 +03:00
Vitaliy Filippov 2c9a10d081 Fix an idiotic bug leading to failed reads with -ERANGE with EC :D 2023-06-03 00:44:52 +03:00
Vitaliy Filippov 150968070f Slightly improve some debug prints
Test / test_change_pg_count (push) Successful in 30s Details
Test / test_change_pg_count_ec (push) Successful in 31s Details
Test / test_change_pg_size (push) Successful in 7s Details
Test / test_create_nomaxid (push) Successful in 7s Details
Test / test_etcd_fail (push) Successful in 45s Details
Test / test_failure_domain (push) Successful in 8s Details
Test / test_interrupted_rebalance (push) Successful in 1m3s Details
Test / test_interrupted_rebalance_imm (push) Successful in 55s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m30s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 57s Details
Test / test_minsize_1 (push) Successful in 20s Details
Test / test_move_reappear (push) Successful in 16s Details
Test / test_rebalance_verify (push) Successful in 1m49s Details
Test / test_rebalance_verify_imm (push) Successful in 1m40s Details
Test / test_rebalance_verify_ec (push) Successful in 2m4s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 1m51s Details
Test / test_rm (push) Successful in 14s Details
Test / test_snapshot (push) Successful in 16s Details
Test / test_snapshot_ec (push) Successful in 17s Details
Test / test_splitbrain (push) Successful in 14s Details
Test / test_write (push) Successful in 41s Details
Test / test_write_xor (push) Successful in 49s Details
Test / test_write_no_same (push) Successful in 10s Details
Test / test_heal_pg_size_2 (push) Successful in 2m58s Details
Test / test_scrub (push) Successful in 23s Details
Test / test_scrub_zero_osd_2 (push) Successful in 19s Details
Test / test_scrub_xor (push) Successful in 17s Details
Test / test_scrub_pg_size_3 (push) Successful in 24s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 28s Details
Test / test_scrub_ec (push) Successful in 25s Details
2023-05-29 01:04:16 +03:00
Vitaliy Filippov cdfc74665b Close client FDs only when destroying the client, after handling all async reads/writes
Test / test_change_pg_count (push) Successful in 50s Details
Test / test_change_pg_count_ec (push) Successful in 58s Details
Test / test_change_pg_size (push) Successful in 17s Details
Test / test_create_nomaxid (push) Successful in 19s Details
Test / test_etcd_fail (push) Successful in 58s Details
Test / test_failure_domain (push) Successful in 14s Details
Test / test_interrupted_rebalance (push) Successful in 1m31s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m0s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m23s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m16s Details
Test / test_minsize_1 (push) Successful in 38s Details
Test / test_move_reappear (push) Successful in 54s Details
Test / test_rebalance_verify (push) Successful in 2m26s Details
Test / test_rebalance_verify_imm (push) Successful in 2m7s Details
Test / test_rebalance_verify_ec (push) Successful in 2m51s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m15s Details
Test / test_rm (push) Successful in 22s Details
Test / test_snapshot (push) Successful in 40s Details
Test / test_snapshot_ec (push) Successful in 34s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_write (push) Successful in 1m7s Details
Test / test_write_xor (push) Successful in 2m13s Details
Test / test_write_no_same (push) Successful in 18s Details
Test / test_heal_pg_size_2 (push) Successful in 5m14s Details
Test / test_scrub (push) Successful in 36s Details
Test / test_scrub_zero_osd_2 (push) Successful in 40s Details
Test / test_scrub_xor (push) Successful in 54s Details
Test / test_scrub_pg_size_3 (push) Successful in 54s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 1m12s Details
Test / test_scrub_ec (push) Successful in 1m10s Details
Fixes "Client XX command out of sync" sometimes happening on reconnections
2023-05-25 00:52:43 +03:00
Vitaliy Filippov 3b4cf29e65 Release 0.9.0
New features:
- Scrubbing! Check documentation: [auto_scrub](src/branch/master/docs/config/osd.en.md#auto_scrub)
- Document online-updatable configuration parameters

Bug fixes:
- Fix NaN during PG optimisation if there are nonexisting OSDs in node_placement
- Fix monitor crash on pool deletion
- Clear journal_device and meta_device before initialising the next OSD in automatic mode
- Sync unsynced deletes before overwriting them with a lower version
  (reproducted mostly/only after scrubbing)
2023-05-21 15:07:14 +03:00
Vitaliy Filippov ce02f47de6 Allow to disable scrub_find_best 2023-05-21 12:33:38 +03:00
Vitaliy Filippov f1961157f0 Fix brute-force error locator for EC n+k with k > 2
Test / test_change_pg_count_ec (push) Successful in 2m23s Details
Test / test_change_pg_size (push) Successful in 20s Details
Test / test_create_nomaxid (push) Successful in 16s Details
Test / test_etcd_fail (push) Successful in 55s Details
Test / test_failure_domain (push) Successful in 12s Details
Test / test_interrupted_rebalance (push) Successful in 1m18s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m9s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m23s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m8s Details
Test / test_minsize_1 (push) Successful in 22s Details
Test / test_move_reappear (push) Successful in 28s Details
Test / test_rebalance_verify (push) Successful in 2m17s Details
Test / test_rebalance_verify_imm (push) Successful in 2m19s Details
Test / test_rebalance_verify_ec (push) Successful in 3m4s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m22s Details
Test / test_rm (push) Successful in 23s Details
Test / test_snapshot (push) Successful in 20s Details
Test / test_snapshot_ec (push) Successful in 34s Details
Test / test_splitbrain (push) Successful in 33s Details
Test / test_write (push) Successful in 1m15s Details
Test / test_write_xor (push) Successful in 2m6s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_heal_pg_size_2 (push) Successful in 5m22s Details
Test / test_heal_ec (push) Successful in 5m31s Details
Test / test_scrub (push) Successful in 29s Details
Test / test_scrub_zero_osd_2 (push) Successful in 27s Details
Test / test_scrub_xor (push) Successful in 22s Details
Test / test_scrub_pg_size_3 (push) Failing after 37s Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 29s Details
Test / test_scrub_ec (push) Failing after 33s Details
2023-05-21 00:57:14 +03:00
Vitaliy Filippov 88c1ba0790 Fix compile errors with gcc 10
Test / build (push) Has started running Details
Test / buildenv (push) Successful in 11s Details
Test / make_test (push) Has been cancelled Details
Test / test_add_osd (push) Has been cancelled Details
Test / test_cas (push) Has been cancelled Details
Test / test_change_pg_count (push) Has been cancelled Details
Test / test_change_pg_count_ec (push) Has been cancelled Details
Test / test_change_pg_size (push) Has been cancelled Details
Test / test_create_nomaxid (push) Has been cancelled Details
Test / test_etcd_fail (push) Has been cancelled Details
Test / test_failure_domain (push) Has been cancelled Details
Test / test_interrupted_rebalance (push) Has been cancelled Details
Test / test_interrupted_rebalance_imm (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec_imm (push) Has been cancelled Details
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot (push) Has been cancelled Details
Test / test_snapshot_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
2023-05-20 23:20:09 +03:00
Vitaliy Filippov fa90b5a4e7 Schedule automatic scrubs correctly (not just after previous scrub) 2023-05-20 23:20:09 +03:00
Vitaliy Filippov 8d40ad99a6 Add scrub documentation 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 6ca20aa194 Allow scrub to fix corrupted object states 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 4bfd994341 Sync unsynced deletes before overwriting them with a lower version 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 59e959dcbb Do not die when "different versions are returned from subops" 2023-05-20 23:19:39 +03:00
Vitaliy Filippov a9581f0739 Handle dirty deletes during read correctly O_o 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 105a405b0a Implement vitastor-cli fix 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 0e5d0e02a9 Add "vitastor-cli describe" command 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 0439981a66 Implement "describe object(s)" operation
Required to implement fixing inconsistent objects in vitastor-cli
2023-05-20 23:19:39 +03:00
Vitaliy Filippov 6648f6bb6e Implement ambiguity detection during scrub 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 281be547eb Implement brute-force error locator for EC 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 0c78dd7178 Add no_scrub flag 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 3c924397e7 Store next scrub timestamp instead of last scrub timestamp 2023-05-20 23:19:39 +03:00
Vitaliy Filippov c3bd26193d Implement PG scrub runner 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 43b77d7619 Implement scrubbing "data path" - OSD_OP_SCRUB 2023-05-20 23:19:39 +03:00
Vitaliy Filippov a6d846863b Add min/max stripe and limit to OP_LIST 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 8dc427b43c Retry failed reads (including chained and RMW) from other replicas 2023-05-20 23:19:39 +03:00
Vitaliy Filippov bf2112653b Refcount object_states 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 0538a484b3 Add corrupted object state 2023-05-20 23:19:39 +03:00
Vitaliy Filippov 97720fa6b4 Remove unused capture
Test / buildenv (push) Successful in 12s Details
Test / build (push) Has started running Details
Test / make_test (push) Has been cancelled Details
Test / test_add_osd (push) Has been cancelled Details
Test / test_cas (push) Has been cancelled Details
Test / test_change_pg_count (push) Has been cancelled Details
Test / test_change_pg_count_ec (push) Has been cancelled Details
Test / test_change_pg_size (push) Has been cancelled Details
Test / test_create_nomaxid (push) Has been cancelled Details
Test / test_etcd_fail (push) Has been cancelled Details
Test / test_failure_domain (push) Has been cancelled Details
Test / test_interrupted_rebalance (push) Has been cancelled Details
Test / test_interrupted_rebalance_imm (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec (push) Has been cancelled Details
Test / test_interrupted_rebalance_ec_imm (push) Has been cancelled Details
Test / test_minsize_1 (push) Has been cancelled Details
Test / test_move_reappear (push) Has been cancelled Details
Test / test_rebalance_verify (push) Has been cancelled Details
Test / test_rebalance_verify_imm (push) Has been cancelled Details
Test / test_rebalance_verify_ec (push) Has been cancelled Details
Test / test_rebalance_verify_ec_imm (push) Has been cancelled Details
Test / test_rm (push) Has been cancelled Details
Test / test_snapshot (push) Has been cancelled Details
Test / test_snapshot_ec (push) Has been cancelled Details
Test / test_splitbrain (push) Has been cancelled Details
Test / test_write (push) Has been cancelled Details
Test / test_write_xor (push) Has been cancelled Details
Test / test_write_no_same (push) Has been cancelled Details
Test / test_heal_pg_size_2 (push) Has been cancelled Details
Test / test_heal_ec (push) Has been cancelled Details
2023-05-20 22:58:51 +03:00
Vitaliy Filippov e60e352df6 Improve vitastor-nbd documentation 2023-05-20 22:58:51 +03:00
Vitaliy Filippov 629999f789 Clear journal_device and meta_device before initialising the next OSD in automatic mode 2023-05-15 23:58:55 +03:00
Vitaliy Filippov 5a9e1ede52 Release 0.8.9
Test / buildenv (push) Successful in 9s Details
Test / build (push) Successful in 2m31s Details
Test / test_cas (push) Successful in 12s Details
Test / make_test (push) Successful in 33s Details
Test / test_change_pg_size (push) Successful in 19s Details
Test / test_change_pg_count (push) Successful in 55s Details
Test / test_create_nomaxid (push) Successful in 21s Details
Test / test_change_pg_count_ec (push) Successful in 58s Details
Test / test_failure_domain (push) Successful in 13s Details
Test / test_etcd_fail (push) Successful in 1m4s Details
Test / test_interrupted_rebalance (push) Successful in 1m13s Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m7s Details
Test / test_add_osd (push) Successful in 2m59s Details
Test / test_move_reappear (push) Successful in 24s Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m22s Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m1s Details
Test / test_rebalance_verify (push) Successful in 2m12s Details
Test / test_minsize_1 (push) Successful in 15s Details
Test / test_rebalance_verify_imm (push) Successful in 2m4s Details
Test / test_rebalance_verify_ec_imm (push) Successful in 2m9s Details
Test / test_rm (push) Successful in 17s Details
Test / test_snapshot (push) Successful in 23s Details
Test / test_rebalance_verify_ec (push) Successful in 2m31s Details
Test / test_splitbrain (push) Successful in 23s Details
Test / test_snapshot_ec (push) Successful in 30s Details
Test / test_write_no_same (push) Successful in 16s Details
Test / test_write (push) Successful in 53s Details
Test / test_write_xor (push) Successful in 1m19s Details
Test / test_heal_pg_size_2 (push) Successful in 4m30s Details
Test / test_heal_ec (push) Successful in 4m32s Details
- The tests are now stable and run in a CI system based on Gitea CI
- The release includes final bug fixes for EC:
  - Implement missing EC recovery of allocation bitmap when built with ISA-L
  - Fix broken snapshot export with EC (allocation bitmap reads were giving incorrect results previously)
- Also fixed bugs manifesting under heavy load:
  - Fix monitor possibly applying incorrect PG history on retries
  - Fix monitor incorrectly changing PG count when last_clean_pgs contains less PGs than the new number
  - Allow writes to wait for free space again, but now correctly (previously dropped in 0.8.2)
  - Fix a rare segfault in client (handle client stop during incoming stream handling in 1 more place)
  - Make monitor correctly handle etcd connection errors - it could die instead of connecting to another etcd
  - Fix OSD rarely being unable to report PG states after a PG was taken over by another OSD
- Fixed return code for incomplete EC objects (now EIO) and made cluster client retry this error
- Made other small changes for tests: timeouts, nice/ionice for etcd, waiting conditions, NBD device checks and so on
2023-05-14 01:25:09 +03:00
Vitaliy Filippov de3e609166 Add a FIXME about QEMU driver thread safety 2023-05-14 00:06:09 +03:00
Vitaliy Filippov 11481170f5 Add a FIXME about ENOSPC 2023-05-13 23:59:44 +03:00
Vitaliy Filippov 6442010f93 Skip offline PGs during state reporting when the state is already deleted or taken over by another OSD
This fixes OSDs being unable to report PG states in rare conditions
2023-05-12 23:17:45 +03:00
Vitaliy Filippov ce4a8067b5 Handle client stop during incoming stream handling in 1 more place 2023-05-11 01:53:41 +03:00
Vitaliy Filippov 8cac795445 Return EIO instead of EINVAL for incomplete EC objects 2023-05-11 01:15:23 +03:00
Vitaliy Filippov a409598b16 Wait for free space again, but count on big_write flushes instead of just flusher activity 2023-05-10 01:51:02 +03:00
Vitaliy Filippov f4c6765522 Ignore ENOENT in epoll_ctl 2023-05-08 20:39:20 +03:00
Vitaliy Filippov 5da1d8e1b5 Fix EC just-bitmap reads (len=0) (fixes SCHEME=ec test_snapshot.sh) 2023-05-07 14:00:08 +03:00
Vitaliy Filippov 44f86f1999 Add a basic EC 2+2 recovery test (not really required, but let it be there) 2023-05-07 11:26:27 +03:00
Vitaliy Filippov 2d9a80c6f6 Implement missing bitmap recovery with ISA-L \(°□°)/ 2023-05-07 11:25:51 +03:00
Vitaliy Filippov ab615849d6 Release 0.8.8
- Fix vitastor-cli rm/rm-data broken in 0.8.6 (missing messenger initialization)
- Prepare OSD read handler for upcoming version with scrub - allow "secondary reads" to return errors
- Fix OSDs re-peering PGs infinitely with a big number of PGs (reproduced in test_add_osd)
- Fix another variant of flusher sync-waiting stall (reproduced in test_write)
- Fix other tests in tests/ (will add them to Gitea CI soon)
- Add patches for QEMU 6.2-8.0
- Fix QEMU driver compatibility with QEMU 8.0
- Build packages for RHEL 9 clones (based on AlmaLinux 9)
2023-04-28 11:22:00 +03:00
Vitaliy Filippov b94587ef0e Fix some build warnings 2023-04-28 00:44:27 +03:00
Vitaliy Filippov c768a9015f Fix QEMU driver compatibility with QEMU 8.0 2023-04-25 11:20:21 +03:00
Vitaliy Filippov b74ccb613c Fix another variant of flusher sync-waiting stall 2023-04-24 00:44:41 +03:00
Vitaliy Filippov a04dab0840 Initialize messenger in cluster_client listings 2023-04-24 00:44:41 +03:00
Vitaliy Filippov 160863f707 Print op pointer values in slow log 2023-04-23 17:54:00 +03:00
Vitaliy Filippov 2877cd0adb Allow OP_SEC_READ to return errors (do not hang the connection) 2023-04-23 17:54:00 +03:00
Vitaliy Filippov 480509f5b9 Fix pg_data_size > 1 for replicas (harmless bug) 2023-04-23 01:50:42 +03:00
Vitaliy Filippov 46462da45e Preload own PG history updates to fix PG state loop possibly applying the old metadata version 2023-04-23 01:50:30 +03:00
Vitaliy Filippov 7e958afeda Release 0.8.7
This release includes a bunch of important bugfixes for erasure-coded setups
with disabled immediate_commit. After these fixes, "test_heal" OSD killing test
now passes fine with EC:

- Fix cluster write stalls with "Error while doing flush on OSD xx: -16 (Device or resource busy)"
  in OSD logs possible in EC setups with disabled immediate_commit by selectively
  syncing nonsynced objects on STABILIZE/ROLLBACK (https://github.com/vitalif/vitastor/issues/51)
- Fix other EC + disabled immediate_commit problems:
  - Fix "opcode=5 retval=-2" errors happening on SYNC retries
  - Fix non-working "pagination" during PG dirty object flushing
  - Fix write operations not continued correctly after dirty object flushing
- Fix incorrect parity read-modify-write calculation when writing into a lost chunk
- Fix OSDs losing left_on_dead PG state of non-clean PGs and thus not removing junk data in the cluster
- Fix a small memory leak caused by bad indexing of EC recovery matrices
- Fix a rare use-after-free in cluster_client caused by a reenterability issue
- Fix vitastor-cli create command syntax in the CSI driver
- Allow to start OSDs without local store for tests
- Fix memory allocation error in disk_tool_meta for non-standard metadata block sizes
- Fix delete operations received before loading pool metadata crashing OSDs with "null pointer exception"
- Improve "theoretical performance" Russian documentation

New features:

- Implement online configuration update for some parameters. Documentation is coming soon :)
2023-04-11 02:11:57 +03:00
Vitaliy Filippov 2f5e769a29 Fix a small memory leak caused by bad indexing of EC recovery matrices 2023-04-11 00:30:36 +03:00
Vitaliy Filippov 3237014608 Fix incorrect parity read-modify-write calculation when writing into a lost chunk 2023-04-09 02:06:10 +03:00
Vitaliy Filippov baaf8f6f44 Fix write operations not continued correctly after flush 2023-04-09 02:06:10 +03:00
Vitaliy Filippov 1d83fdcd17 Add debug logs to osd_flush 2023-04-09 02:06:10 +03:00
Vitaliy Filippov 0ddd787c38 Fix non-working "pagination" during PG dirty object flushing 2023-04-08 02:44:02 +03:00
Vitaliy Filippov 6eff3a60a5 Do not lose left_on_dead PG state of non-clean PGs 2023-04-08 02:44:02 +03:00
Vitaliy Filippov 888a6975ab Fix a rare use-after-free in cluster_client caused by a reenterability issue 2023-04-08 02:44:02 +03:00
Vitaliy Filippov cd1e890bd4 Fix "opcode=5 retval=-2" errors sometimes possible with EC 2023-04-08 02:44:02 +03:00
Vitaliy Filippov 0fbf4c6a08 Selectively sync nonsynced objects on STABILIZE/ROLLBACK (fix for github issue #51) 2023-04-08 02:44:02 +03:00
Vitaliy Filippov d06ed2b0e7 Implement online config update 2023-03-26 19:21:50 +03:00
Vitaliy Filippov 2fb0c85618 Allow to start OSDs without local store (only for tests) 2023-03-15 01:13:59 +03:00
Vitaliy Filippov d81a6c04fc Update cmake min version so it does not complain about deprecation 2023-03-15 01:08:23 +03:00
Vitaliy Filippov 7b35801647 Fix possible bad realloc in disk_tool_meta for non-standard metadata block sizes 2023-03-15 01:08:23 +03:00
Vitaliy Filippov f3228d5c07 Fix typo (did not affect execution though) 2023-03-15 01:08:23 +03:00
Vitaliy Filippov 18366f5055 Fix read/write return type in rw_blocking 2023-03-15 01:08:14 +03:00
Vitaliy Filippov 851507c147 Add missing close() in test stubs 2023-03-15 00:23:56 +03:00
Vitaliy Filippov 9aaad28488 Fix "null pointer exception" for unhandled OSD_OP_DELETEs (when pool is not loaded yet) 2023-03-02 11:16:39 +03:00
Vitaliy Filippov 8810eae8fb Release 0.8.6
Important fixes:

- Fix possibly incorrect EC parity chunk updates with EC n+k, k > 1 and when
  the first parity chunk is missing

Minor fixes and improvements:

- Fix incorrect EC free space statistics in vitastor-cli df output
- Speedup vitastor-cli startup in clusters with RDMA
- Remove unused PG "peered" state (previously used to update PG epoch)
- Use sfdisk with just --json in vitastor-disk (--dump --json isn't needed)
- Allow trailing comma in sfdisk output (fixes sfdisk 2.36 compatibility)
- Slightly improve RDMA send/receive code
- Reduce RDMA memory consumption by default (rdma_max_recv/send = 16/8)
- Use vitastor-cli instead of direct etcd interaction in the CSI driver
2023-02-28 11:18:48 +03:00
Vitaliy Filippov 14d6acbcba Set default rdma_max_recv/send to 16/8, fix documentation 2023-02-28 11:00:56 +03:00
Vitaliy Filippov 1e307069bc Fix missing parity chunk calculation for EC n+k, k > 1 and first parity chunk missing 2023-02-28 02:40:19 +03:00
Vitaliy Filippov c3e80abad7 Allow to send more than 1 operation at a time 2023-02-26 02:01:04 +03:00
Vitaliy Filippov 138ffe4032 Reuse incoming RDMA buffers 2023-02-26 00:55:01 +03:00
Vitaliy Filippov 4ab630b44d Use just sfdisk --json, --dump is not needed 2023-02-23 00:55:47 +03:00
Vitaliy Filippov 2c8241b7db Remove PG "peered" state 2023-02-21 01:30:42 +03:00
Vitaliy Filippov 36a7dd3671 Move tests to "make test" 2023-02-21 01:30:42 +03:00
Vitaliy Filippov 936122bbcf Initialize msgr lazily in client to speedup vitastor-cli with RDMA enabled 2023-02-19 18:59:07 +03:00
Vitaliy Filippov 1a1ba0d1e7 Add set_immediate to ringloop and use it for bs/osd ops to prevent reenterability issues 2023-02-09 17:37:26 +03:00
Vitaliy Filippov 3d09c9cec7 Remove unused wait_sqe() from ringloop 2023-02-09 17:37:26 +03:00
Vitaliy Filippov 3d08a1ad6c Fix cluster_client test after last reenterability fixes 2023-02-05 01:47:32 +03:00
Vitaliy Filippov aba93b951b Fix incorrect EC free space statistics in vitastor-cli df output 2023-01-26 02:04:29 +03:00
Vitaliy Filippov d125fb1f30 Release 0.8.5
- Fix a possible "double free" bug in the client library happening on OSD restart
- Fix a possible write hang on PG history update when only epoch is changed
- Fix incorrect systemd target "local.target" in mon/make-etcd
- Allow "content" option in PVE storage plugin to allow to enable containers
- Build client library without tcmalloc which fixes "attempt to free invalid pointer"
  errors when, for example, trying to run QEMU with both Vitastor and Ceph RBD disks
2023-01-25 01:43:49 +03:00
Vitaliy Filippov 8b552a01f9 Do not retry successful operation parts in client (could lead to "double free" bugs) 2023-01-25 01:30:36 +03:00
Vitaliy Filippov 0385b2f9e8 Fix write hangs on PG epoch update - always set pg.history_changed to true 2023-01-25 01:30:15 +03:00
Vitaliy Filippov 9f4e34a8cc Build client library without tcmalloc
Fixes "[src/tcmalloc.cc:332] Attempt to free invalid pointer ..." when trying
to run QEMU with both Vitastor and Ceph RBD disks and other possible allocator
collisions.
2023-01-15 00:01:11 +03:00
Vitaliy Filippov 81fc8bb94c Release 0.8.4
New features:
- Implement QCOW2 image/snapshot export via qemu-img (bdrv_co_block_status in the driver)
- Remove OSDs from PG history during `vitastor-cli rm-osd` to prevent `left_on_dead` PG states after deletion
- Add a new recovery_pg_switch setting to mix all PGs during recovery, to almost
  fully reduce the probability of ENOSPC during rebalance
- Introduce partial ENOSPC ("OSD is full") handling - now ENOSPC doesn't turn
  into cascades of crashes
- Add migration support to Proxmox VE Vitastor driver
- Track last_clean_pgs on a per-pool basis thus reducing data movement in a cluster
  with pools remaining unclean/degraded for a long time

Bug fixes:
- Fix a bug where monitor could generate degraded PGs if one of the hosts had no OSDs
- Fix a bug where monitor could skip PG redistribution with a lot of OSDs in cluster
- Report PG history synchronously on the first write, which improves PG consistency
  and availability at the same time, because history now gets reported correctly
  and doesn't get reported without the need for it
- Fix possible write and recovery stalls which could happen in a cluster with both EC and replicated pools
- Make OSD and monitors sanitize & deduplicate PG history items in etcd
- Fix non-working OSD peer config safety check
- Fix a rare journal flush stall where flushing wasn't activated with full journal, but with empty flush queue
- Fix builds without ISA-L (jerasure-only) crashing with EC N+K, K>=2 due to the lack of 16-byte buffer alignment
- Fix a possible crash for EC N+K, K>=2 when calculating a parity chunk with previous parity chunk missing
- Fix a bug where vitastor-disk purge with suppressed warnings didn't work
2023-01-13 23:59:54 +03:00
Vitaliy Filippov bc465c16de Fix arithmetic on void* for clang 2023-01-13 23:58:42 +03:00
Vitaliy Filippov 8763e9211c Fix qemu driver compilation warning/error 2023-01-13 23:44:39 +03:00
Vitaliy Filippov fe87b4076b Fix backwards compatibility in cluster_client 2023-01-12 02:37:31 +03:00
Vitaliy Filippov 137309cf29 Implement bdrv_co_block_status for snapshot export support 2023-01-07 17:06:58 +03:00
Vitaliy Filippov 373f9d0387 Try to re-peer PGs on history change 2023-01-06 12:46:44 +03:00
Vitaliy Filippov c4516ea971 Also remove deleted OSD from PG configuration and last_clean_pgs 2023-01-06 12:46:44 +03:00
Vitaliy Filippov 91065c80fc Try to prevent left_on_dead when deleting OSDs by removing them from PG history 2023-01-06 12:46:43 +03:00
Vitaliy Filippov 02e7be7dc9 Prevent reenterability side effects during PG history operation resume 2023-01-03 02:20:50 +03:00
Vitaliy Filippov 73940adf07 Prioritize EC (non-instantly-stable) operations under journal pressure
This reduces the probability of hitting OSD stalls with EC due to "deadlocks"
where two parallel write operations wait for each other to complete
2023-01-03 00:05:45 +03:00
Vitaliy Filippov e950c024d3 Do not sync peer OSDs before listing
Sync before listing was added to wait for all PG writes possibly left in queue
from the previous master to finish before listing it

But in fact it may block the cluster when EC is used and some unstable writes
are left in the queue - they block journal flushing, rollback/stabilize is
required to unblock them, but rollback/stabilize may only happen after PG is
peered. But peering needs listings, listings are requested only after sync, and
sync itself waits for currently blocked writes waiting in the queue
2023-01-03 00:05:45 +03:00
Vitaliy Filippov 71d6d9f868 Fix possible crash on ENOSPC during operation cancel in blockstore 2023-01-03 00:05:45 +03:00
Vitaliy Filippov a4dfa519af Report PG history synchronously during write
This has 2 effects:
1) OSD sets aren't added into PG history until actual write attempts anymore
   which removes unneeded extra osd_sets in PG history
2) New OSD sets are reported synchronously and can't be lost on PG restarts
   happening at the same time with reconfiguration
2023-01-01 23:41:05 +03:00
Vitaliy Filippov 67019f5b02 Make OSD sort & sanitize PG history items 2023-01-01 23:17:42 +03:00
Vitaliy Filippov 0593e5c21c Fix OSD peer config safety check 2022-12-31 02:24:42 +03:00
Vitaliy Filippov 998e24adf8 Add a new recovery_pg_switch setting to mix all PGs during recovery 2022-12-30 02:03:33 +03:00
Vitaliy Filippov d7bd36dc32 Fix another rare journal flush stall 2022-12-30 02:03:33 +03:00
Vitaliy Filippov cf5c562800 Log all object locations when peering PGs 2022-12-30 02:03:33 +03:00
Vitaliy Filippov 629200b0cc Return ENOSPC as the primary OSD 2022-12-30 02:03:33 +03:00
Vitaliy Filippov 3589ccec22 Do not disconnect peer on ENOSPC during write 2022-12-30 01:54:25 +03:00
Vitaliy Filippov 8d55a1e780 Build osd_rmw_test both with and without ISA-L 2022-12-29 19:13:57 +03:00
Vitaliy Filippov 65f6b3a4eb Fix jerasure crashing on bitmap calculation/restoration due to the lack of 16-byte alignment 2022-12-29 19:13:57 +03:00
Vitaliy Filippov fd216eac77 Add a test for missing parity chunk calculation 2022-12-29 19:13:57 +03:00
Vitaliy Filippov 61fca7c426 Fix crash when calculating a parity chunk with previous parity chunk missing (test coming shortly) 2022-12-29 19:13:57 +03:00
Vitaliy Filippov 68f3fb795e Suppress warnings in vitastor-disk purge correctly 2022-12-27 11:09:19 +03:00
Vitaliy Filippov fa90f287da Release 0.8.3
- Implement a new "vitastor-disk purge" command to remove OSDs with safety checks
- Implement a new "vitastor-cli rm-osd" command to only remove OSD metadata from etcd
- Fix a bug where the monitor could ignore OSD removal and other /osd/stats key changes
- Fix a bug where garbage could be returned when reading objects being written at the same time
- Fix a rare write stall where journal space could be not reclaimed where there
  were no new operations in the flush queue
- Fix a rare peering stall caused by a previous long listing operations queues limiting attempt
- Fix total object count statistic in OSD on object creation
- Add missing offset&len into vitastor-disk dump-journal for big_writes, fix JSON format
- Make vitastor-cli print help on missing command
- Make vitastor-cli translate all '-' to '_' in CLI options
2022-12-27 02:40:55 +03:00
Vitaliy Filippov 795020674d Loop journal flusher when the queue is empty but there is a trim request 2022-12-27 02:28:20 +03:00
Vitaliy Filippov 8e12285629 Fix vitastor-disk purge (now it works) 2022-12-27 02:28:20 +03:00
Vitaliy Filippov b9b50ab4cc Implement vitastor-disk purge command 2022-12-26 02:48:48 +03:00
Vitaliy Filippov 0d8625f92d Make vitastor-cli print help on missing command 2022-12-26 02:48:48 +03:00
Vitaliy Filippov 2f3c2c5140 Implement safety check for OSD removal, translate all '-' to '_' in cli options
'-' to '_' translation fixes a bug with create --image_meta
2022-12-26 02:48:48 +03:00
Vitaliy Filippov 4ebdd02b0f Remove LIST op limiter
It doesn't prevent OSD slow ops but may itself lead to stalls :)
2022-12-26 02:48:48 +03:00
Vitaliy Filippov c2244331e6 Add vitastor-cli rm-osd command 2022-12-26 02:48:48 +03:00
Vitaliy Filippov 31bd1ec145 Fix object creation check for statistics 2022-12-21 02:51:11 +03:00
Vitaliy Filippov c08d1f2dfe Add missing offset&len into big_writes journal dump, fix commas again 2022-12-21 02:51:11 +03:00
Vitaliy Filippov 1d80bcc8d0 Fix blockstore returning garbage for unstable reads if there is an in-flight version
"In-flight" versions are added into dirty_db when writes are enqueued. And they
weren't ignored by subsequent reads even though they didn't have data location yet.
This bug was leading to test_heal.sh not passing sometimes with replicated setups.
2022-12-21 02:48:24 +03:00
Vitaliy Filippov 5ef8bed75f Release 0.8.2
- Fix QEMU driver compatibility with QEMU 7.0 and < 2.9
- Add patches for pve-qemu-kvm 7.1 (PVE 7.3) and pve-qemu-kvm 6.2 (PVE 7.2)
- Fix Proxmox driver location in the pve-storage-vitastor package
- Disable HDD autodetection in non-hybrid mode
- Explicitly warn about a buggy kernels on -EAGAIN in io_uring
- Final fix for the lack of zeroing out of old metadata entries
  (do not crash with "big_write journal_entry was allocated over another object"
  in some cases after an unclean OSD shutdown)
- Wait for data writes before fsyncing data if data fsync is enabled
- Never try to wait for free space inside blockstore thus stalling OSDs
- Fix a rare crash in osd_peering due to callback ordering
- Fix a rare duplication of ping & op message IDs
- Fix a rare use-after-free during pings
- Add --force to vitastor-disk read-sb
- Make vitastor-disk dump metadata object IDs in hex, add forgotten commas
- Fix vitastor-disk SCSI disk cache check
2022-12-17 17:54:13 +03:00
Vitaliy Filippov 8669998e5e Fix discard_list_subop() for local ops 2022-12-17 17:54:13 +03:00
Vitaliy Filippov b457327e77 Oops. Fix metadata read after fixes :-) 2022-12-17 17:31:57 +03:00
Vitaliy Filippov f7fa9d5e34 Fix SCSI device cache type check 2022-12-17 17:31:57 +03:00
Vitaliy Filippov 49b88b01f9 Fix clang build 2022-12-17 16:25:26 +03:00
Vitaliy Filippov 71688bcb59 Disable HDD autodetection in non-hybrid mode 2022-12-17 16:12:15 +03:00
Vitaliy Filippov 552e207d2b Explicitly print errors about -EAGAIN in io_uring 2022-12-17 15:49:49 +03:00
Vitaliy Filippov 5464821fa5 Final fix for the lack of zeroing out of old metadata entries
If a crash occurs during flushing a redirect-write it may happen so that
the disk contains both old and new metadata entries. This is OK, but prior
to 0.8.0 after this situation OSDs started without problem, but then they
crashed after some more overwrites with a "tried to overwrite non-zero
metadata entry" error. 0.8.0 introduced a change that was intended to fix
this situation, but rather than fixing it it prevented OSDs from starting,
now because of a "big_write journal_entry was allocated over another object"
error... :-)

This change finally fixes the original issue.

Followup to 54ef2c389f
2022-12-17 14:50:31 +03:00
Vitaliy Filippov 6917a32ca8 Add --force to vitastor-disk read-sb 2022-12-17 02:47:15 +03:00
Vitaliy Filippov f8722a8bd5 Dump meta in hex 2022-12-17 01:50:38 +03:00
Vitaliy Filippov 9c2f69c9fa Add forgotten commas to vitastor-disk dump-journal 2022-12-17 01:22:58 +03:00
Vitaliy Filippov 1a93e3f33a Wait for data writes before fsyncing data if data fsync is enabled 2022-12-16 20:46:55 +03:00
Vitaliy Filippov 3f35744052 Fix compatibility with QEMU aio_set_fd_handler signatures in 7.0 and < 2.9 2022-12-15 19:17:17 +03:00
Vitaliy Filippov cb437913d3 Never try to wait for free space inside blockstore 2022-12-12 00:27:05 +03:00