osd: Wait for the old state lease to expire instead of exiting #140
Open
koolkhel
wants to merge 1 commits from
koolkhel/vitastor:osd-wait-old-lease into master
pull from: koolkhel/vitastor:osd-wait-old-lease
merge into: vitalif:master
vitalif:master
vitalif:primary-stickiness
vitalif:libvfn
vitalif:canceling
vitalif:immediate-repeering
vitalif:rdma-close-tcp
vitalif:use-degraded-writes
vitalif:fix-write-delete-flow
vitalif:openssl-3.0.12
vitalif:openssl-capture-secrets
vitalif:ignore-small-rdma-end
vitalif:encrypt-batch
vitalif:faster-peering-calc
vitalif:http-hello
vitalif:xxh3-bench
vitalif:create-user-etcd-based
vitalif:aesxts_streaming_nosupport
vitalif:refactor-heap-continue-write
vitalif:ston3lu-patch-0127
vitalif:rdma-credits
vitalif:heap-meta
vitalif:sendmsg-zc-no-more-error
vitalif:fix-clear-refs
vitalif:veeam-trick
vitalif:test-etcd-schizo
vitalif:test
vitalif:test1
vitalif:test2
vitalif:cow-meta
vitalif:kv-readahead
vitalif:qemu-fix-crash-idea
vitalif:eventloop-auto-init-trigger
vitalif:hugo-docs
vitalif:node-mutex-and-postpone
vitalif:check-writeback
vitalif:msgr-iothreads-v2
vitalif:antietcd
vitalif:rel-1.4
vitalif:test-fix-ec-unknown-state-51
vitalif:kv-update
vitalif:kv-debug
vitalif:hotfix-1.2.0
vitalif:rdma-simple-nodp
vitalif:zerocopy-tcp-send
vitalif:csi-staging
vitalif:hotfix-1.1.0
vitalif:hotfix-1.0.0
vitalif:fsync-feedback
vitalif:cached-reads
vitalif:qemu-send-loop
vitalif:test-double-alloc
vitalif:hier-failure-domains
vitalif:mon-self-restart
vitalif:epoch-deletions
vitalif:rdma-flow-control
vitalif:csi-use-vitastor-cli
vitalif:rdma-v2
vitalif:etcd-hide-base64
vitalif:lrc-matrix
vitalif:rm-left-on-dead
vitalif:nfs-proxy-old
vitalif:non-odp-rdma
vitalif:test-zctr
vitalif:separate-data-connections
vitalif:sec_osd_msgr
vitalif:test-assert
vitalif:nbd-vmsplice
vitalif:allow-etcd-address-option
vitalif:rdma-zerocopy
vitalif:rel-0.5
vitalif:openonload
vitalif:sync-io-test
vitalif:trace-sync
vitalif:trace-sqes
vitalif:test-submit-and-wait
vitalif:test-sq-poll
vitalif:blocking-uring-test
1 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 2edaa05bd0 |
osd: Wait for the old state lease to expire instead of exiting
An OSD killed with SIGKILL or by a power loss can't revoke its etcd lease, so its /osd/state key stays until the lease expires. An OSD restarted before that printed "OSD N is still up" and exited, to be started again by systemd after one more full blockstore load. etcd also gives every lease a full TTL again when its leader changes, so after a power loss of the whole cluster most OSDs go through this, some of them several times. Check the state key before binding the sockets and acquiring the lease. While it exists and has a lease, check it again every etcd_quick_timeout (1 second by default). After 3 lease TTLs in total go on as before: create_osd_state() then finds the key and exits, because another OSD with this number really is up. The OSD doesn't listen and holds no lease while it waits, so it accepts no requests and writes nothing to etcd. |