Client: Retry a failed config reload instead of failing the running client #117

Merged
vitalif merged 1 commits from koolkhel/vitastor:client-reload-retry into master 2026-11-16 18:21:37 +03:00
Contributor

Description

A client reloads the global config when etcd cancels its watch on a compacted revision (on_reload_hook). With client_infinite_start off, which is the default, one etcd error during that reload goes to fatal_error() and exits the process (or aborts the client when exit_on_error is off), although the client is already serving I/O with a valid configuration. Seen with an iSCSI gateway built on the client while etcd was slow (one member down, 600 ms latency on another).

The fix: a reload only happens after start, so the reload hook switches the client to infinite_start and the reload is retried by timer, as load_global_config() already does in that mode.

Tested: builds; test_cluster_client passes.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description A client reloads the global config when etcd cancels its watch on a compacted revision (`on_reload_hook`). With `client_infinite_start` off, which is the default, one etcd error during that reload goes to `fatal_error()` and exits the process (or aborts the client when `exit_on_error` is off), although the client is already serving I/O with a valid configuration. Seen with an iSCSI gateway built on the client while etcd was slow (one member down, 600 ms latency on another). The fix: a reload only happens after start, so the reload hook switches the client to `infinite_start` and the reload is retried by timer, as `load_global_config()` already does in that mode. Tested: builds; `test_cluster_client` passes. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-11-16 18:13:28 +03:00
Client: Retry a failed config reload instead of failing the running client
All checks were successful
Test / test_heal_csum_4k_dj (push) Successful in 2m34s
Test / test_enospc_imm_xor (push) Successful in 22s
Test / test_old_enospc_imm (push) Successful in 18s
Test / test_old_enospc_xor (push) Successful in 21s
Test / test_old_enospc (push) Successful in 26s
Test / test_old_enospc_imm_xor (push) Successful in 25s
Test / test_scrub_zero_osd_2 (push) Successful in 23s
Test / test_scrub (push) Successful in 27s
Test / test_scrub_pg_size_3 (push) Successful in 20s
Test / test_scrub_xor (push) Successful in 28s
Test / test_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 25s
Test / test_scrub_ec (push) Successful in 28s
Test / test_old_scrub (push) Successful in 27s
Test / test_old_scrub_zero_osd_2 (push) Successful in 26s
Test / test_old_scrub_xor (push) Successful in 29s
Test / test_old_scrub_pg_size_3 (push) Successful in 23s
Test / test_old_scrub_pg_size_6_pg_minsize_4_osd_count_6_ec (push) Successful in 26s
Test / test_old_scrub_ec (push) Successful in 27s
Test / test_old_partwr_csum (push) Successful in 25s
Test / test_heal_old_csum_32k_dj (push) Successful in 2m23s
Test / test_heal_old_csum_32k (push) Successful in 2m30s
Test / test_heal_old_csum_32k_dmj (push) Successful in 2m37s
Test / test_heal_old_csum_4k_dmj (push) Successful in 2m22s
Test / test_nfs (push) Successful in 18s
Test / test_nfs_unaligned_append (push) Successful in 21s
Test / test_write_encrypted (push) Successful in 30s
Test / test_nfs_long_readdir (push) Successful in 1m0s
Test / test_write_encrypted_ec (push) Successful in 40s
Test / test_heal_old_csum_4k_dj (push) Successful in 2m27s
Test / test_heal_old_csum_4k (push) Successful in 2m22s
beb52fdf16
A client reloads the global config when etcd cancels its watch on a
compacted revision. With client_infinite_start off, which is the default,
one etcd error during that reload goes to fatal_error() and exits the
process (or aborts the client when exit_on_error is off), although the
client is already serving I/O with a valid configuration. Retry the
reload by timer instead, as with infinite_start.
vitalif approved these changes 2026-11-16 18:15:56 +03:00
vitalif merged commit beb52fdf16 into master 2026-11-16 18:21:37 +03:00
Sign in to join this conversation.
No Reviewers
No Label
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#117
No description provided.