osd: Wait for the old state lease to expire instead of exiting #140
Open
koolkhel
wants to merge 1 commits from
koolkhel/vitastor:osd-wait-old-lease into master
pull from: koolkhel/vitastor:osd-wait-old-lease
merge into: vitalif:master
vitalif:master
vitalif:primary-stickiness
vitalif:libvfn
vitalif:canceling
vitalif:immediate-repeering
vitalif:rdma-close-tcp
vitalif:use-degraded-writes
vitalif:fix-write-delete-flow
vitalif:openssl-3.0.12
vitalif:openssl-capture-secrets
vitalif:ignore-small-rdma-end
vitalif:encrypt-batch
vitalif:faster-peering-calc
vitalif:http-hello
vitalif:xxh3-bench
vitalif:create-user-etcd-based
vitalif:aesxts_streaming_nosupport
vitalif:refactor-heap-continue-write
vitalif:ston3lu-patch-0127
vitalif:rdma-credits
vitalif:heap-meta
vitalif:sendmsg-zc-no-more-error
vitalif:fix-clear-refs
vitalif:veeam-trick
vitalif:test-etcd-schizo
vitalif:test
vitalif:test1
vitalif:test2
vitalif:cow-meta
vitalif:kv-readahead
vitalif:qemu-fix-crash-idea
vitalif:eventloop-auto-init-trigger
vitalif:hugo-docs
vitalif:node-mutex-and-postpone
vitalif:check-writeback
vitalif:msgr-iothreads-v2
vitalif:antietcd
vitalif:rel-1.4
vitalif:test-fix-ec-unknown-state-51
vitalif:kv-update
vitalif:kv-debug
vitalif:hotfix-1.2.0
vitalif:rdma-simple-nodp
vitalif:zerocopy-tcp-send
vitalif:csi-staging
vitalif:hotfix-1.1.0
vitalif:hotfix-1.0.0
vitalif:fsync-feedback
vitalif:cached-reads
vitalif:qemu-send-loop
vitalif:test-double-alloc
vitalif:hier-failure-domains
vitalif:mon-self-restart
vitalif:epoch-deletions
vitalif:rdma-flow-control
vitalif:csi-use-vitastor-cli
vitalif:rdma-v2
vitalif:etcd-hide-base64
vitalif:lrc-matrix
vitalif:rm-left-on-dead
vitalif:nfs-proxy-old
vitalif:non-odp-rdma
vitalif:test-zctr
vitalif:separate-data-connections
vitalif:sec_osd_msgr
vitalif:test-assert
vitalif:nbd-vmsplice
vitalif:allow-etcd-address-option
vitalif:rdma-zerocopy
vitalif:rel-0.5
vitalif:openonload
vitalif:sync-io-test
vitalif:trace-sync
vitalif:trace-sqes
vitalif:test-submit-and-wait
vitalif:test-sq-poll
vitalif:blocking-uring-test
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
No description provided.
Delete Branch "koolkhel/vitastor:osd-wait-old-lease"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
An OSD killed with SIGKILL or by a power loss can't revoke its etcd lease, so its
/osd/statekey stays until the lease expires. An OSD restarted before that printsKey ... already exists in etcd, OSD N is still upand exits. systemd starts it again 10seconds later, and each start loads the whole blockstore before it gets to this check.
After a power loss of the whole cluster this happens to most OSDs: etcd doesn't store lease
expiry times, and a newly elected leader gives every lease its full TTL again
(
lessor.Promote()), so the keys of the previous boot live for one more TTL, and againafter every further leader change.
The change:
bind_socket()andacquire_lease(), the OSDreads its state key. While the key exists and has a lease, it checks it again every
etcd_quick_timeout(1 second by default). It logs one line when the wait starts:[OSD 1] Key /vitastor/osd/state/1 already exists in etcd, waiting up to 45 seconds for its lease to expire.create_osd_state()finds the key and exits with code 1 and the old message, becauseanother OSD with the same number really is up. A key without a lease is not waited for.
writes nothing to etcd.
acquire_lease()andcreate_osd_state()work as before.Tested:
tests/test_osd_restart.sh:kill -9an OSD and start it again at once, withoutdeleting the state key by hand. It has to wait without listening sockets, then register
and listen. Then a second OSD with the same number has to exit with an error and leave
the state key of the first one alone. The test fails on master (the restarted OSD exits)
and passes with the change, with etcd 3.5.15 and with
ANTIETCD=1(about a minute each).osd_test,test_switch_primary.sh,test_rm_degraded.shpass.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA
В принципе идея неплохая, но нужно тогда ещё все bind_socket запускать после получения lease, чтобы случайно запросы не начать принимать.
выше отписал
Переделал: ожидание теперь идёт до
bind_socket().wait_old_osd_state()вызывается послеinit_blockstoreвместо парыbind_socket()+acquire_lease(). Она читает ключ/osd/state/N. Если ключ есть и у него есть lease, спрашивает/lease/timetolive, ждёт и проверяет снова, всего не больше 3 TTL. Когда ключа нет, вызываетbind_socket()иacquire_lease(), как раньше.bind_port=0его назначает ядро. Поэтому порядок такой: дождаться, пока старый ключ исчезнет → bind → lease → ключ. Окно между bind и ключом осталось таким же, как в master.acquire_lease()иcreate_osd_state()остались как в master, а правки про «свой lease в ключе» и про таймер статистики стали не нужны, я их убрал. Если после 3 TTL ключ всё ещё есть,create_osd_state()пишет «OSD N is still up» и выходит, как раньше.tests/test_osd_restart.shтеперь проверяет черезss, что у ждущего OSD нет слушающих сокетов, а после регистрации они есть. На прошлой версии ветки эта проверка падает.Ветку перебазировал на текущий master (force-push). Описание PR выше осталось от первой версии.
dace9b6392to15db867957@@ -530,0 +565,4 @@{ "ID", old_lease }}, st_cli->etcd_quick_timeout, 0, 0, [this, state_key, waited, max_wait](std::string err, json11::Json data){// Wait a full TTL if etcd can't tell how much of it is left (antietcd)Может быть не пытаться спать полный интервал при неудаче, а просто ретраить получение лизы до успеха с равными интервалами типа 1 секунды или более?
@@ -530,0 +565,4 @@{ "ID", old_lease }}, st_cli->etcd_quick_timeout, 0, 0, [this, state_key, waited, max_wait](std::string err, json11::Json data){// Wait a full TTL if etcd can't tell how much of it is left (antietcd)Сделал так: убрал
/lease/timetoliveи расчёт интервала совсем. Теперь OSD просто перепроверяет ключ раз вetcd_quick_timeout(1 с по умолчанию), пока он не исчезнет, одинаково для etcd и antietcd. Ограничение в 3 TTL оставил: после негоcreate_osd_state()находит ключ и выходит с «OSD N is still up», как в master.15db867957to2edaa05bd0View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.