Add a note about HDD, enable throttling only for hybrid OSDs
Test / test_add_osd (push) Successful in 2m21s
Details
Test / buildenv (push) Successful in 10s
Details
Test / build (push) Successful in 2m39s
Details
Test / make_test (push) Successful in 34s
Details
Test / test_cas (push) Successful in 7s
Details
Test / test_change_pg_count (push) Successful in 43s
Details
Test / test_change_pg_count_ec (push) Successful in 38s
Details
Test / test_change_pg_size (push) Successful in 7s
Details
Test / test_create_nomaxid (push) Successful in 7s
Details
Test / test_etcd_fail (push) Successful in 1m30s
Details
Test / test_interrupted_rebalance (push) Successful in 2m4s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m30s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m45s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m23s
Details
Test / test_failure_domain (push) Successful in 9s
Details
Test / test_snapshot (push) Successful in 19s
Details
Test / test_snapshot_ec (push) Successful in 19s
Details
Test / test_minsize_1 (push) Successful in 12s
Details
Test / test_move_reappear (push) Failing after 1m32s
Details
Test / test_rm (push) Successful in 12s
Details
Test / test_snapshot_chain (push) Successful in 1m8s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m2s
Details
Test / test_snapshot_down (push) Successful in 22s
Details
Test / test_snapshot_down_ec (push) Successful in 21s
Details
Test / test_splitbrain (push) Successful in 13s
Details
Test / test_rebalance_verify (push) Successful in 2m52s
Details
Test / test_rebalance_verify_imm (push) Successful in 2m46s
Details
Test / test_rebalance_verify_ec (push) Successful in 5m9s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m5s
Details
Test / test_write (push) Successful in 40s
Details
Test / test_write_xor (push) Successful in 49s
Details
Test / test_write_no_same (push) Successful in 15s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s
Details
Test / test_heal_ec (push) Successful in 4m35s
Details
Test / test_scrub (push) Successful in 37s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 40s
Details
Test / test_scrub_xor (push) Successful in 31s
Details
Test / test_scrub_pg_size_3 (push) Successful in 40s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 26s
Details
Test / test_scrub_ec (push) Successful in 22s
Details
Test / test_add_osd (push) Successful in 2m21s
Details
Test / buildenv (push) Successful in 10s
Details
Test / build (push) Successful in 2m39s
Details
Test / make_test (push) Successful in 34s
Details
Test / test_cas (push) Successful in 7s
Details
Test / test_change_pg_count (push) Successful in 43s
Details
Test / test_change_pg_count_ec (push) Successful in 38s
Details
Test / test_change_pg_size (push) Successful in 7s
Details
Test / test_create_nomaxid (push) Successful in 7s
Details
Test / test_etcd_fail (push) Successful in 1m30s
Details
Test / test_interrupted_rebalance (push) Successful in 2m4s
Details
Test / test_interrupted_rebalance_imm (push) Successful in 1m30s
Details
Test / test_interrupted_rebalance_ec (push) Successful in 1m45s
Details
Test / test_interrupted_rebalance_ec_imm (push) Successful in 1m23s
Details
Test / test_failure_domain (push) Successful in 9s
Details
Test / test_snapshot (push) Successful in 19s
Details
Test / test_snapshot_ec (push) Successful in 19s
Details
Test / test_minsize_1 (push) Successful in 12s
Details
Test / test_move_reappear (push) Failing after 1m32s
Details
Test / test_rm (push) Successful in 12s
Details
Test / test_snapshot_chain (push) Successful in 1m8s
Details
Test / test_snapshot_chain_ec (push) Successful in 2m2s
Details
Test / test_snapshot_down (push) Successful in 22s
Details
Test / test_snapshot_down_ec (push) Successful in 21s
Details
Test / test_splitbrain (push) Successful in 13s
Details
Test / test_rebalance_verify (push) Successful in 2m52s
Details
Test / test_rebalance_verify_imm (push) Successful in 2m46s
Details
Test / test_rebalance_verify_ec (push) Successful in 5m9s
Details
Test / test_rebalance_verify_ec_imm (push) Successful in 5m5s
Details
Test / test_write (push) Successful in 40s
Details
Test / test_write_xor (push) Successful in 49s
Details
Test / test_write_no_same (push) Successful in 15s
Details
Test / test_heal_pg_size_2 (push) Successful in 3m43s
Details
Test / test_heal_ec (push) Successful in 4m35s
Details
Test / test_scrub (push) Successful in 37s
Details
Test / test_scrub_zero_osd_2 (push) Successful in 40s
Details
Test / test_scrub_xor (push) Successful in 31s
Details
Test / test_scrub_pg_size_3 (push) Successful in 40s
Details
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 26s
Details
Test / test_scrub_ec (push) Successful in 22s
Details
parent
b7e4d0c9bf
commit
57ad4c3636
|
@ -15,7 +15,7 @@ Vitastor архитектурно похож на Ceph, что означает
|
|||
и автоматическое распределение данных по любому числу дисков любого размера с настраиваемыми схемами
|
||||
избыточности - репликацией или с произвольными кодами коррекции ошибок.
|
||||
|
||||
Vitastor нацелен на SSD и SSD+HDD кластеры с как минимум 10 Гбит/с сетью, поддерживает
|
||||
Vitastor нацелен в первую очередь на SSD и SSD+HDD кластеры с как минимум 10 Гбит/с сетью, поддерживает
|
||||
TCP и RDMA и на хорошем железе может достигать задержки 4 КБ чтения и записи на уровне ~0.1 мс,
|
||||
что примерно в 10 раз быстрее, чем Ceph и другие популярные программные СХД.
|
||||
|
||||
|
|
|
@ -14,8 +14,8 @@ Vitastor is architecturally similar to Ceph which means strong consistency,
|
|||
primary-replication, symmetric clustering and automatic data distribution over any
|
||||
number of drives of any size with configurable redundancy (replication or erasure codes/XOR).
|
||||
|
||||
Vitastor targets SSD and SSD+HDD clusters with at least 10 Gbit/s network, supports
|
||||
TCP and RDMA and may achieve 4 KB read and write latency as low as ~0.1 ms
|
||||
Vitastor targets primarily SSD and SSD+HDD clusters with at least 10 Gbit/s network,
|
||||
supports TCP and RDMA and may achieve 4 KB read and write latency as low as ~0.1 ms
|
||||
with proper hardware which is ~10 times faster than other popular SDS's like Ceph
|
||||
or internal systems of public clouds.
|
||||
|
||||
|
|
|
@ -19,6 +19,9 @@
|
|||
- Get some SATA or NVMe SSDs with capacitors (server-grade drives). You can use desktop SSDs
|
||||
with lazy fsync, but prepare for inferior single-thread latency. Read more about capacitors
|
||||
[here](../config/layout-cluster.en.md#immediate_commit).
|
||||
- If you want to use HDDs, get modern HDDs with Media Cache or SSD Cache: HGST Ultrastar,
|
||||
Toshiba MG08, Seagate EXOS or something similar. If your drives don't have such cache then
|
||||
you also need small SSDs for journal and metadata (even 2 GB per 1 TB of HDD space is enough).
|
||||
- Get a fast network (at least 10 Gbit/s). Something like Mellanox ConnectX-4 with RoCEv2 is ideal.
|
||||
- Disable CPU powersaving: `cpupower idle-set -D 0 && cpupower frequency-set -g performance`.
|
||||
- [Install Vitastor packages](../installation/packages.en.md).
|
||||
|
@ -45,9 +48,10 @@ On the monitor hosts:
|
|||
}
|
||||
```
|
||||
- Initialize OSDs:
|
||||
- SSD-only: `vitastor-disk prepare /dev/sdXXX [/dev/sdYYY ...]`. You can add
|
||||
`--disable_data_fsync off` to leave disk cache enabled if you use desktop
|
||||
SSDs without capacitors.
|
||||
- SSD-only or HDD-only: `vitastor-disk prepare /dev/sdXXX [/dev/sdYYY ...]`.
|
||||
Add `--disable_data_fsync off` to leave disk write cache enabled if you use
|
||||
desktop SSDs without capacitors. Do NOT add `--disable_data_fsync off` if you
|
||||
use HDDs or SSD+HDD.
|
||||
- Hybrid, SSD+HDD: `vitastor-disk prepare --hybrid /dev/sdXXX [/dev/sdYYY ...]`.
|
||||
Pass all your devices (HDD and SSD) to this script — it will partition disks and initialize journals on its own.
|
||||
This script skips HDDs which are already partitioned so if you want to use non-empty disks for
|
||||
|
|
|
@ -19,6 +19,9 @@
|
|||
- Возьмите серверы с SSD (SATA или NVMe), желательно с конденсаторами (серверные SSD). Можно
|
||||
использовать и десктопные SSD, включив режим отложенного fsync, но производительность будет хуже.
|
||||
О конденсаторах читайте [здесь](../config/layout-cluster.ru.md#immediate_commit).
|
||||
- Если хотите использовать HDD, берите современные модели с Media или SSD кэшем - HGST Ultrastar,
|
||||
Toshiba MG08, Seagate EXOS или что-то похожее. Если такого кэша у ваших дисков нет,
|
||||
обязательно возьмите SSD под метаданные и журнал (маленькие, буквально 2 ГБ на 1 ТБ HDD-места).
|
||||
- Возьмите быструю сеть, минимум 10 гбит/с. Идеал - что-то вроде Mellanox ConnectX-4 с RoCEv2.
|
||||
- Для лучшей производительности отключите энергосбережение CPU: `cpupower idle-set -D 0 && cpupower frequency-set -g performance`.
|
||||
- [Установите пакеты Vitastor](../installation/packages.ru.md).
|
||||
|
@ -45,9 +48,10 @@
|
|||
}
|
||||
```
|
||||
- Инициализуйте OSD:
|
||||
- SSD: `vitastor-disk prepare /dev/sdXXX [/dev/sdYYY ...]`. Если вы используете
|
||||
десктопные SSD без конденсаторов, можете оставить кэш включённым, добавив
|
||||
опцию `--disable_data_fsync off`.
|
||||
- Только SSD или только HDD: `vitastor-disk prepare /dev/sdXXX [/dev/sdYYY ...]`.
|
||||
Если вы используете десктопные SSD без конденсаторов, добавьте опцию `--disable_data_fsync off`,
|
||||
чтобы оставить кэш записи диска включённым. НЕ добавляйте эту опцию, если используете
|
||||
жёсткие диски (HDD).
|
||||
- Гибридные, SSD+HDD: `vitastor-disk prepare --hybrid /dev/sdXXX [/dev/sdYYY ...]`.
|
||||
Передайте все ваши SSD и HDD скрипту в командной строке подряд, скрипт автоматически выделит
|
||||
разделы под журналы на SSD и данные на HDD. Скрипт пропускает HDD, на которых уже есть разделы
|
||||
|
|
|
@ -99,7 +99,7 @@ int disk_tool_t::prepare_one(std::map<std::string, std::string> options, int is_
|
|||
if (options["journal_size"] == "")
|
||||
{
|
||||
if (options["journal_device"] == "")
|
||||
options["journal_size"] = "32M";
|
||||
options["journal_size"] = is_hdd ? "128M" : "32M";
|
||||
else if (is_hdd)
|
||||
options["journal_size"] = DEFAULT_HYBRID_JOURNAL;
|
||||
}
|
||||
|
@ -107,7 +107,7 @@ int disk_tool_t::prepare_one(std::map<std::string, std::string> options, int is_
|
|||
{
|
||||
if (options["block_size"] == "")
|
||||
options["block_size"] = "1M";
|
||||
if (options["throttle_small_writes"] == "")
|
||||
if (options["journal_device"] != "" && options["throttle_small_writes"] == "")
|
||||
options["throttle_small_writes"] = "1";
|
||||
}
|
||||
json11::Json::object sb;
|
||||
|
|
Loading…
Reference in New Issue