Correct vitastor-cli help for rm, raw-ls, fix and rm-osd

- rm: the --exact and --matching descriptions were swapped. - raw-ls: show that --pool_id is required, and --pg_num unless --offset is given, and when the pool configuration fills in the rest. - fix: show --bad-osds as required. - rm-osd: describe -f, --allow-up and --allow-data-loss. Up OSDs are refused too, -f covers them and degraded/offline PGs, and --allow-data-loss covers incomplete ones and implies -f. Apply the same corrections to the CLI documentation.
2026-10-25 16:10:29 +03:00
9 changed files with 46 additions and 220 deletions
--- a/docs/usage/cli.en.md
+++ b/docs/usage/cli.en.md
@@ -231,8 +231,8 @@ Options:
 * `--writers-stopped` allows optimised removal in case of a single 'slim' read-write
  child and 'fat' removed parent: the child is merged into parent and parent is renamed
  to child in that case. In other cases parent layers are always merged into children.
-* `--exact` - remove multiple images with names matching given glob patterns.
-* `--matching` - remove multiple images with given names
+* `--exact` - remove multiple images with given names.
+* `--matching` - remove multiple images with names matching given glob patterns.
 * `--down-ok` - continue deletion/merging even if some data will be left on unavailable OSDs.

 ## flatten
@@ -287,7 +287,7 @@ Describe unclean object locations in the cluster. Options:

 ## fix

-`vitastor-cli fix [--objects <objects>] [--bad-osds <osds>] [--part <part>] [--check no]`
+`vitastor-cli fix [--objects <objects>] --bad-osds <osds> [--part <part>] [--check no]`

 Fix inconsistent objects in the cluster by deleting some copies.

@@ -298,7 +298,7 @@ Fix inconsistent objects in the cluster by deleting some copies.
    Plain text format: 0x<inode>:0x<stripe> <any delimiter> 0x<inode>:0x<stripe> ...
    JSON format: [{"inode":"0x...","stripe":"0x..."},...]
 --bad-osds <osds>
-    Remove inconsistent copies/parts of objects from these OSDs, effectively
+    Required. Remove inconsistent copies/parts of objects from these OSDs, effectively
    marking them bad and allowing Vitastor to recover objects from other copies.
 --part <number>
    Only remove EC part <number> (from 0 to pg_size-1), required for extreme
@@ -316,14 +316,17 @@ Allocate a new OSD number and reserve it by creating empty `/osd/stats/<n>` key.

 ## rm-osd

-`vitastor-cli rm-osd [--force] [--allow-data-loss] [--dry-run] <osd_id> [osd_id...]`
+`vitastor-cli rm-osd [-f|--force] [--allow-data-loss] [--allow-up] [--dry-run] <osd_id> [osd_id...]`

 Remove metadata and configuration for specified OSD(s) from etcd.

-Refuses to remove OSDs with data without `--force` and `--allow-data-loss`.
+Refuses to remove OSDs that are up or whose removal makes PGs degraded, offline or incomplete.

-With `--dry-run` only checks if deletion is possible without data loss and
-redundancy degradation.
+* `-f|--force` - remove OSDs even if they are up or PGs become degraded or offline.
+* `--allow-data-loss` - remove OSDs even if PGs or objects become incomplete (DATA LOSS). Implies `--force`.
+* `--allow-up` - remove OSDs even if they are up.
+* `--dry-run` - only check if deletion is possible without data loss and
+  redundancy degradation.

 ## osd-tree

--- a/docs/usage/cli.ru.md
+++ b/docs/usage/cli.ru.md
@@ -239,8 +239,8 @@ vitastor-cli dd [iimg=<image> | if=<file>] [oimg=<image> | of=<file>] [bs=1M] \
  у удаляемой цепочки есть только один дочерний образ, содержащий небольшой объём данных.
  В этом случае дочерний образ вливается в родительский и удаляется, а родительский
  переименовывается в дочерний.
-* `--exact` - удалить все образы с именами, подходящими под переданные glob-шаблоны.
-* `--matching` - удалить все образы с точно заданными именами.
+* `--exact` - удалить все образы с точно заданными именами.
+* `--matching` - удалить все образы с именами, подходящими под переданные glob-шаблоны.
 * `--down-ok` - продолжать удаление/слияние, даже если часть данных останется неудалённой на недоступных OSD.

 ## flatten
@@ -302,7 +302,7 @@ vitastor-cli dd [iimg=<image> | if=<file>] [oimg=<image> | of=<file>] [bs=1M] \

 ## fix

-`vitastor-cli fix [--objects <объекты>] [--bad-osds <osds>] [--part <номер>] [--check no]`
+`vitastor-cli fix [--objects <объекты>] --bad-osds <osds> [--part <номер>] [--check no]`

 Исправить неконсистентные (неоднозначные) объекты путём удаления части копий.

@@ -313,9 +313,9 @@ vitastor-cli dd [iimg=<image> | if=<file>] [oimg=<image> | of=<file>] [bs=1M] \
    Простой формат: 0x<инод>:0x<смещение> <любой разделитель> 0x<инод>:0x<смещение> ...
    Формат JSON: [{"inode":"0x<инод>","stripe":"0x<смещение>"},...]
 --bad-osds <osds>
-    Удалить неконсистентные копии/части объектов с данных OSD, таким образом
-    признавая потерю этих копий и позволяя Vitastor-у восстановить объекты из
-    других копий.
+    Обязательная опция. Удалить неконсистентные копии/части объектов с данных
+    OSD, таким образом признавая потерю этих копий и позволяя Vitastor-у
+    восстановить объекты из других копий.
 --part <номер>
    Удалить только части EC с заданным номером (от 0 до pg_size-1). Нужно только
    в редких граничных случаях, когда один и тот же OSD содержит несколько частей
@@ -334,14 +334,19 @@ vitastor-cli dd [iimg=<image> | if=<file>] [oimg=<image> | of=<file>] [bs=1M] \

 ## rm-osd

-`vitastor-cli rm-osd [--force] [--allow-data-loss] [--dry-run] <osd_id> [osd_id...]`
+`vitastor-cli rm-osd [-f|--force] [--allow-data-loss] [--allow-up] [--dry-run] <osd_id> [osd_id...]`

 Удалить метаданные и конфигурацию для заданных OSD из etcd.

-Отказывается удалять OSD с данными без опций `--force` и `--allow-data-loss`.
+Отказывается удалять работающие OSD, а также OSD, после удаления которых PG станут
+деградированными, недоступными (offline) или неполными (incomplete).

-С опцией `--dry-run` только проверяет, возможно ли удаление без потери данных и деградации
-избыточности.
+* `-f|--force` - удалить OSD, даже если они работают или PG станут деградированными или недоступными.
+* `--allow-data-loss` - удалить OSD, даже если PG или объекты станут неполными (ПОТЕРЯ ДАННЫХ).
+  Включает действие `--force`.
+* `--allow-up` - удалить OSD, даже если они работают.
+* `--dry-run` - только проверить, возможно ли удаление без потери данных и деградации
+  избыточности.

 ## osd-tree

--- a/src/cmd/cli.cpp
+++ b/src/cmd/cli.cpp
@@ -108,8 +108,8 @@ static const char* help_text =
    "  --writers-stopped allows optimised removal in case of a single 'slim' read-write\n"
    "  child and 'fat' removed parent: the child is merged into parent and parent is renamed\n"
    "  to child in that case. In other cases parent layers are always merged into children.\n"
-    "  --exact            Remove multiple images with names matching given glob patterns.\n"
-    "  --matching         Remove multiple images with given names\n"
+    "  --exact            Remove multiple images with given names.\n"
+    "  --matching         Remove multiple images with names matching given glob patterns.\n"
    "  --writers-stopped  Allow renaming inodes over their read/write children.\n"
    "  --down-ok          Continue deletion/merging even if some data will be left on unavailable OSDs.\n"
    "\n"
@@ -145,12 +145,15 @@ static const char* help_text =
    "  --min-offset, --max-offset\n"
    "      Restrict listing to specific offsets inside inodes.\n"
    "\n"
-    "vitastor-cli raw-ls [OPTIONS]\n"
+    "vitastor-cli raw-ls --pool_id ID (--pg_num NUM|--offset NUM) [OPTIONS]\n"
    "  Find object(s) in the cluster using raw secondary listing operations. Options:\n"
    "  [--min_inode NUM] [--max_inode NUM] [--offset NUM] [--pg_num NUM] [--pg_count COUNT]\n"
    "  [--pg_stripe_size NUM] [--osds 1,2,3,...]\n"
+    "  --pool_id (numeric pool ID) is required. --pg_num is required unless --offset is given,\n"
+    "  in which case the PG is calculated from the offset. --pg_count, --pg_stripe_size and\n"
+    "  --osds default to the pool configuration and are required if the pool does not exist.\n"
    "\n"
-    "vitastor-cli fix [--objects <objects>] [--bad-osds <osds>] [--part <part>] [--check no]\n"
+    "vitastor-cli fix [--objects <objects>] --bad-osds <osds> [--part <part>] [--check no]\n"
    "  Fix inconsistent objects in the cluster by deleting some copies.\n"
    "  --objects <objects>\n"
    "      Objects to fix, either in plain text or JSON format. If not specified,\n"
@@ -158,7 +161,7 @@ static const char* help_text =
    "      Plain text format: 0x<inode>:0x<stripe> <any delimiter> 0x<inode>:0x<stripe> ...\n"
    "      JSON format: [{\"inode\":\"0x...\",\"stripe\":\"0x...\"},...]\n"
    "  --bad-osds <osds>\n"
-    "      Remove inconsistent copies/parts of objects from these OSDs, effectively\n"
+    "      Required. Remove inconsistent copies/parts of objects from these OSDs, effectively\n"
    "      marking them bad and allowing Vitastor to recover objects from other copies.\n"
    "  --part <number>\n"
    "      Only remove EC part <number> (from 0 to pg_size-1), required for extreme\n"
@@ -170,11 +173,14 @@ static const char* help_text =
    "vitastor-cli alloc-osd\n"
    "  Allocate a new OSD number and reserve it by creating empty /osd/stats/<n> key.\n"
    "\n"
-    "vitastor-cli rm-osd [--force] [--allow-data-loss] [--dry-run] <osd_id> [osd_id...]\n"
+    "vitastor-cli rm-osd [-f|--force] [--allow-data-loss] [--allow-up] [--dry-run] <osd_id> [osd_id...]\n"
    "  Remove metadata and configuration for specified OSD(s) from etcd.\n"
-    "  Refuses to remove OSDs with data without --force and --allow-data-loss.\n"
-    "  With --dry-run only checks if deletion is possible without data loss and\n"
-    "  redundancy degradation.\n"
+    "  Refuses to remove OSDs that are up or whose removal makes PGs degraded, offline or incomplete.\n"
+    "  -f|--force         Remove OSDs even if they are up or PGs become degraded or offline.\n"
+    "  --allow-data-loss  Remove OSDs even if PGs or objects become incomplete (DATA LOSS). Implies --force.\n"
+    "  --allow-up         Remove OSDs even if they are up.\n"
+    "  --dry-run          Only check if deletion is possible without data loss and\n"
+    "                     redundancy degradation.\n"
    "\n"
    "vitastor-cli osd-tree [-l|--long]\n"
    "  Show current OSD tree, optionally with I/O statistics if -l is specified.\n"
@@ -687,7 +693,7 @@ static int run(cli_tool_t *p, json11::Json::object cfg)
    {
        printf("%s\n", result.data.dump().c_str());
    }
-    else if (p->json_output && result.err && !p->json_printed)
+    else if (p->json_output && result.err)
    {
        printf("%s\n", json11::Json(json11::Json::object {
            { "error_code", result.err },
--- a/src/cmd/cli.h
+++ b/src/cmd/cli.h
@@ -38,7 +38,6 @@ public:
    bool json_output = false;
    int log_level = 0;
    bool is_command_line = false;
-    bool json_printed = false;
    bool color = false;

    std::shared_ptr<user_info_t> user; // for http mode
--- a/src/cmd/cli_describe.cpp
+++ b/src/cmd/cli_describe.cpp
@@ -33,13 +33,6 @@ std::vector<uint64_t> parse_uint64_list(json11::Json val)
    return ret;
 }

-bool is_osd_up(cli_tool_t *parent, osd_num_t osd_num)
-{
-    auto & peer_states = parent->cli->st_cli->peer_states;
-    auto peer_it = peer_states.find(osd_num);
-    return peer_it != peer_states.end() && !peer_it->second.is_null();
-}
-
 struct cli_describe_t
 {
    uint64_t object_state = 0;
@@ -48,8 +41,6 @@ struct cli_describe_t
    std::vector<osd_num_t> only_osds;
    uint64_t min_inode = 0, max_inode = 0;
    uint64_t min_offset = 0, max_offset = 0;
-    std::map<osd_num_t, std::string> skipped_pgs;
-    bool incomplete = false;

    cli_tool_t *parent = NULL;
    int state = 0;
@@ -92,18 +83,19 @@ struct cli_describe_t
        }
        only_pg = cfg["pg"].uint64_value();
        min_inode = cfg["inode"].uint64_value();
-        min_offset = stoull_full(cfg["min_offset"].string_value(), 0);
-        max_offset = stoull_full(cfg["max_offset"].string_value(), 0);
        if (min_inode)
        {
            if (!INODE_POOL(min_inode))
                min_inode |= (uint64_t)only_pool << (64-POOL_ID_BITS);
            max_inode = min_inode;
+            min_offset = max_offset = 0;
        }
        else
        {
            min_inode = stoull_full(cfg["min_inode"].string_value(), 0); // to support 0x...
            max_inode = stoull_full(cfg["max_inode"].string_value(), 0);
+            min_offset = stoull_full(cfg["min_offset"].string_value(), 0);
+            max_offset = stoull_full(cfg["max_offset"].string_value(), 0);
            if (!min_inode && !max_inode && only_pool)
            {
                min_inode = (uint64_t)only_pool << (64-POOL_ID_BITS);
@@ -111,9 +103,6 @@ struct cli_describe_t
                    (((uint64_t)1 << (64-POOL_ID_BITS)) - 1);
            }
        }
-        // The OSD ends the listing at (max_inode, max_offset) inclusive
-        if (max_inode && cfg["max_offset"].is_null())
-            max_offset = UINT64_MAX;
        only_osds = parse_uint64_list(cfg["osds"]);
        object_state = stoull_full(cfg["object_state"].string_value(), 0);
        if (!object_state && cfg["object_state"].is_string())
@@ -169,49 +158,15 @@ struct cli_describe_t
                if (pp.first >= min_pool && (!max_pool || pp.first <= max_pool))
                {
                    for (auto & pgp: pp.second.pg_config)
-                    {
-                        if (only_pool && only_pg && (pp.first != only_pool || pgp.first != only_pg))
-                            continue;
-                        if (is_osd_up(parent, pgp.second.cur_primary))
-                            only_osds.push_back(pgp.second.cur_primary);
-                        else
-                        {
-                            auto & pgs = skipped_pgs[pgp.second.cur_primary];
-                            pgs += (pgs == "" ? "" : ", ")+std::to_string(pp.first)+"/"+std::to_string(pgp.first);
-                        }
-                    }
+                        only_osds.push_back(pgp.second.cur_primary);
                }
            }
        }
-        else
-        {
-            for (auto osd_num: only_osds)
-            {
-                if (!is_osd_up(parent, osd_num))
-                    skipped_pgs[osd_num] = "";
-            }
-            only_osds.erase(std::remove_if(only_osds.begin(), only_osds.end(), [this](osd_num_t osd_num)
-            {
-                return !is_osd_up(parent, osd_num);
-            }), only_osds.end());
-        }
-        // Requests to an OSD which is down are never answered
-        for (auto & sk: skipped_pgs)
-        {
-            if (!sk.first)
-                fprintf(stderr, "No primary OSD for PG %s, skipping\n", sk.second.c_str());
-            else if (sk.second != "")
-                fprintf(stderr, "OSD %ju is down, skipping PG %s\n", sk.first, sk.second.c_str());
-            else
-                fprintf(stderr, "OSD %ju is down, skipping it\n", sk.first);
-            incomplete = true;
-        }
        remove_duplicates(only_osds);
        parent->cli->init_msgr();
        if (parent->json_output && parent->is_command_line)
        {
            printf("[\n");
-            parent->json_printed = true;
        }
        for (int i = 0; i < only_osds.size(); i++)
        {
@@ -235,7 +190,6 @@ struct cli_describe_t
            {
                if (op->reply.hdr.retval < 0)
                {
-                    incomplete = true;
                    fprintf(
                        stderr, "Failed to describe objects on OSD %ju (retval=%jd)\n",
                        osd_num, op->reply.hdr.retval
@@ -243,7 +197,6 @@ struct cli_describe_t
                }
                else if (op->reply.describe.result_bytes != op->reply.hdr.retval * sizeof(osd_reply_describe_item_t))
                {
-                    incomplete = true;
                    fprintf(
                        stderr, "Invalid response size from OSD %ju (expected %ju bytes, got %ju bytes)\n",
                        osd_num, op->reply.hdr.retval * sizeof(osd_reply_describe_item_t), op->reply.describe.result_bytes
@@ -310,11 +263,6 @@ struct cli_describe_t
        {
            result.data = describe_items;
        }
-        if (incomplete)
-        {
-            result.err = EIO;
-            result.text = "The listing is incomplete, see the messages above";
-        }
        state = 100;
        describe_items.clear();
    }
--- a/src/cmd/cli_fix.cpp
+++ b/src/cmd/cli_fix.cpp
@@ -21,7 +21,6 @@ struct cli_fix_t
    json11::Json options;
    cli_result_t result;
    json11::Json::array fix_result;
-    bool incomplete = false;

    bool is_done()
    {
@@ -124,20 +123,6 @@ struct cli_fix_t
            state = 100;
            return;
        }
-        {
-            std::string down_osds;
-            for (auto osd_num: bad_osds)
-            {
-                if (!is_osd_up(parent, osd_num))
-                    down_osds += (down_osds == "" ? "" : ", ")+std::to_string(osd_num);
-            }
-            if (down_osds != "")
-            {
-                result = (cli_result_t){ .err = EAGAIN, .text = "Bad OSD(s) "+down_osds+" are down, can't remove object copies from them" };
-                state = 100;
-                return;
-            }
-        }
        remove_duplicates(objects);
        parent->cli->init_msgr();
    resume_1:
@@ -153,7 +138,6 @@ struct cli_fix_t
            if (pool_cfg_it == parent->cli->st_cli->pool_config.end())
            {
                fprintf(stderr, "Object %jx:%jx is from unknown pool\n", obj.inode, obj.stripe);
-                incomplete = true;
                continue;
            }
            auto & pool_cfg = pool_cfg_it->second;
@@ -166,20 +150,9 @@ struct cli_fix_t
                    stderr, "Object %jx:%jx is from PG %u/%u which is not currently active\n",
                    obj.inode, obj.stripe, pool_cfg_it->first, pg_num
                );
-                incomplete = true;
                continue;
            }
            osd_num_t primary_osd = pg_it->second.cur_primary;
-            if (!is_osd_up(parent, primary_osd))
-            {
-                // Requests to an OSD which is down are never answered
-                fprintf(
-                    stderr, "Object %jx:%jx is from PG %u/%u whose primary OSD %ju is down\n",
-                    obj.inode, obj.stripe, pool_cfg_it->first, pg_num, primary_osd
-                );
-                incomplete = true;
-                continue;
-            }
            // Describe -> Remove some copies -> Scrub again
            osd_op_t *op = new osd_op_t;
            op->req = (osd_any_op_t){
@@ -199,7 +172,6 @@ struct cli_fix_t
                if (op->reply.hdr.retval < 0 || op->reply.describe.result_bytes != op->reply.hdr.retval * sizeof(osd_reply_describe_item_t))
                {
                    fprintf(stderr, "Failed to describe objects on OSD %ju (retval=%jd)\n", primary_osd, op->reply.hdr.retval);
-                    incomplete = true;
                    parent->waiting--;
                    loop();
                }
@@ -236,7 +208,6 @@ struct cli_fix_t
                                (*rm_count)--;
                                if (rm_op->reply.hdr.retval < 0)
                                {
-                                    incomplete = true;
                                    fprintf(
                                        stderr, "Failed to remove object %jx:%jx from OSD %ju (retval=%jd)\n",
                                        rm_op->req.sec_del.oid.inode, rm_op->req.sec_del.oid.stripe,
@@ -281,7 +252,6 @@ struct cli_fix_t
                                    {
                                        if (scrub_op->reply.hdr.retval < 0 && scrub_op->reply.hdr.retval != -ENOENT)
                                        {
-                                            incomplete = true;
                                            fprintf(
                                                stderr, "Failed to scrub %jx:%jx on OSD %ju (retval=%jd)\n",
                                                obj.inode, obj.stripe, primary_osd, scrub_op->reply.hdr.retval
@@ -319,11 +289,6 @@ struct cli_fix_t
        {
            result.data = fix_result;
        }
-        if (incomplete)
-        {
-            result.err = EIO;
-            result.text = "Some objects were not fixed, see the messages above";
-        }
        state = 100;
    }
 };
--- a/src/cmd/cli_fix.h
+++ b/src/cmd/cli_fix.h
@@ -7,7 +7,6 @@
 #include <algorithm>

 std::vector<uint64_t> parse_uint64_list(json11::Json val);
-bool is_osd_up(cli_tool_t *parent, osd_num_t osd_num);

 template<class T> void remove_duplicates(std::vector<T> & ret)
 {
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -65,7 +65,6 @@ OLD=1 ./test_move_reappear.sh
 ./test_rm.sh

 ./test_rm_degraded.sh
-./test_describe.sh

 ./test_snapshot_chain.sh
 SCHEME=ec ./test_snapshot_chain.sh
--- a/tests/test_describe.sh
+++ b/tests/test_describe.sh
@@ -1,98 +0,0 @@
-#!/bin/bash -ex
-
-# Test vitastor-cli describe and fix on degraded objects and with down OSDs
-
-PG_COUNT=4
-PG_SIZE=3
-PG_MINSIZE=2
-# Keep degraded objects until the checks are done
-GLOBAL_CONFIG=',"no_recovery":true'
-. `dirname $0`/run_3osds.sh
-
-$VITASTOR_CLI create -s 32M testimg
-
-# Data written while OSD 3 is down is degraded when it comes back
-kill -9 $OSD3_PID
-$ETCDCTL del /vitastor/osd/state/3
-wait_condition 60 "$ETCDCTL get --prefix /vitastor/pg/state/ --print-value-only | jq -s -e '[ .[] | select(.state | index(\"degraded\")) ] | length == $PG_COUNT'" "PG peering without OSD 3"
-$VITASTOR_CLI dd if=/dev/urandom oimg=testimg bs=1M count=32
-start_osd 3
-wait_condition 60 "$ETCDCTL get --prefix /vitastor/pg/state/ --print-value-only | jq -s -e '[ .[] | select((.state | index(\"active\")) and (.state | index(\"has_degraded\"))) ] | length == $PG_COUNT'" "PG peering with OSD 3"
-
-INODE_ID=$($VITASTOR_CLI ls --json | jq -r '.[] | select(.name == "testimg") | .inode_id')
-INODE=$(printf '0x%x' $INODE_ID)
-INODE_NUM=$((INODE_ID & 0xffffffffffff))
-
-# --inode lists the whole inode, not only its first object
-all=$($VITASTOR_CLI describe --json | jq -c '[ .[] | select(.inode == "'$INODE'") | .stripe ] | unique')
-if [[ $(echo "$all" | jq length) -le 1 ]]; then
-    format_error "describe should list more than one degraded object of testimg: $all"
-fi
-one=$($VITASTOR_CLI describe --pool 1 --inode $INODE_NUM --json | jq -c '[ .[].stripe ] | unique')
-if [[ "$one" != "$all" ]]; then
-    format_error "describe --inode lists objects $one instead of $all"
-fi
-
-# --min-offset and --max-offset narrow --inode, and --max-inode includes its whole inode
-second=$(echo "$all" | jq -r '.[1]')
-part=$($VITASTOR_CLI describe --pool 1 --inode $INODE_NUM --min-offset $second --max-offset $second --json | jq -c '[ .[].stripe ] | unique')
-if [[ "$part" != "[\"$second\"]" ]]; then
-    format_error "describe --inode --min-offset $second --max-offset $second lists objects $part"
-fi
-range=$($VITASTOR_CLI describe --min-inode $INODE --max-inode $INODE --json | jq -c '[ .[].stripe ] | unique')
-if [[ "$range" != "$all" ]]; then
-    format_error "describe --min-inode $INODE --max-inode $INODE lists objects $range instead of $all"
-fi
-
-# describe and fix must not wait for an OSD which is down.
-# Stop the monitor so that PGs keep the down OSD as their primary,
-# and stop a primary OSD which has the latest data, so other PGs stay degraded.
-VICTIM=$($VITASTOR_CLI ls-pgs --json | jq -r '[ .[].cur_primary | select(. == 1 or . == 2) ] | first')
-VICTIM_PG=$($VITASTOR_CLI ls-pgs --json | jq -r '[ .[] | select(.cur_primary == '$VICTIM') | .pg_num ] | first')
-OTHER_PG=$($VITASTOR_CLI ls-pgs --json | jq -r '[ .[] | select(.cur_primary != '$VICTIM') | .pg_num ] | first')
-OTHER_PG_COUNT=$($VITASTOR_CLI ls-pgs --json | jq '[ .[] | select(.cur_primary != '$VICTIM') ] | length')
-if [[ "$VICTIM" = null || "$OTHER_PG_COUNT" = 0 ]]; then
-    format_error "Expected PGs with primary OSD 1 or 2 and PGs with other primary OSDs"
-fi
-VICTIM_OBJ=$($VITASTOR_CLI describe --pool 1 --pg $VICTIM_PG --json | jq -r '.[0] | .inode+":"+.stripe')
-OTHER_OBJ=$($VITASTOR_CLI describe --pool 1 --pg $OTHER_PG --json | jq -r '.[0] | .inode+":"+.stripe')
-
-kill -9 $MON_PID
-eval kill -9 \$OSD${VICTIM}_PID
-$ETCDCTL del /vitastor/osd/state/$VICTIM
-wait_condition 60 "$ETCDCTL get --prefix /vitastor/pg/state/ --print-value-only | jq -s -e '[ .[] | select(.primary != $VICTIM and (.state | index(\"degraded\"))) ] | length == $OTHER_PG_COUNT'" "PG peering without OSD $VICTIM"
-# Keep PG 1/$VICTIM_PG active with the down primary, as it is until the OSD's etcd lease expires
-$ETCDCTL put /vitastor/pg/state/1/$VICTIM_PG '{"peers":[1,2,3],"primary":'$VICTIM',"state":["active"]}'
-
-# A listing with skipped OSDs or PGs is incomplete: EIO, but still valid JSON on stdout
-for args in "" "--osds 1,2,3" "--pool 1 --pg $VICTIM_PG"; do
-    rc=0
-    timeout 30 $VITASTOR_CLI describe $args --json > ./testdata/describe.json 2> ./testdata/describe.err || rc=$?
-    [[ $rc == 5 ]] || format_error "describe $args exited with $rc instead of EIO with OSD $VICTIM down"
-    grep -q "OSD $VICTIM is down, skipping \(it\|PG 1/$VICTIM_PG\)" ./testdata/describe.err || \
-        format_error "describe $args should warn about OSD $VICTIM: $(cat ./testdata/describe.err)"
-    grep -q "The listing is incomplete" ./testdata/describe.err || \
-        format_error "describe $args should say that the listing is incomplete: $(cat ./testdata/describe.err)"
-    count=$(jq length ./testdata/describe.json) || format_error "describe $args printed invalid JSON"
-    if [[ "$args" == --pool* ]]; then
-        [[ $count == 0 ]] || format_error "describe $args should list nothing"
-    else
-        [[ $count -gt 0 ]] || format_error "describe $args should still list objects from the other OSDs"
-    fi
-done
-
-if timeout 30 $VITASTOR_CLI fix --bad-osds $VICTIM --objects $OTHER_OBJ 2> ./testdata/fix.err; then
-    format_error "fix should refuse to work with bad OSD $VICTIM down"
-fi
-grep -q "Bad OSD(s) $VICTIM are down" ./testdata/fix.err || format_error "fix printed: $(cat ./testdata/fix.err)"
-
-# An object which is skipped is not fixed: EIO
-rc=0
-timeout 30 $VITASTOR_CLI fix --bad-osds 3 --objects $VICTIM_OBJ 2> ./testdata/fix.err || rc=$?
-[[ $rc == 5 ]] || format_error "fix exited with $rc instead of EIO on an object from PG 1/$VICTIM_PG with primary OSD $VICTIM down"
-grep -q "is from PG 1/$VICTIM_PG whose primary OSD $VICTIM is down" ./testdata/fix.err || \
-    format_error "fix should skip the object from PG 1/$VICTIM_PG: $(cat ./testdata/fix.err)"
-grep -q "Some objects were not fixed" ./testdata/fix.err || \
-    format_error "fix should say that it skipped objects: $(cat ./testdata/fix.err)"
-
-format_green OK