osd: Stop on SIGINT/SIGTERM from the main loop, not the signal handler #141
Reference in New Issue
Block a user
No description provided.
Delete Branch "koolkhel/vitastor:osd-signal-stop"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
The OSD's SIGINT/SIGTERM handler calls
osd_t::force_stop(), which builds a JSON object,allocates strings and starts an HTTP request to etcd. None of that is async-signal-safe. The
OSD is single-threaded, so when the signal interrupts
malloc()orfree()in the main loop,the handler re-enters the allocator and corrupts the heap. The OSD then dies in
mallocwithSIGSEGV or SIGABRT instead of exiting with 0. With glibc 2.28 it shows as a SIGSEGV in
mallocor asmalloc_consolidate(): invalid chunk sizewithin a millisecond of the stopsignal.
The fix follows
aefaa2ee(NBD, UBLK, NFS): the handler only sets a flag, and the main loopcalls
force_stop(0). A second signal still exits at once, now with_exit().One limit, the same as in the NBD loop: a signal that arrives between the flag check and
io_uring_wait_cqe()is handled at the next ring event. On an idle OSD that is at most 1second later (the messenger keepalive timer).
Tested (glibc 2.39, x86-64):
malloc()'s tcache fast path (after the free chunkis read, before the list head is updated). Before:
malloc(): unaligned tcache chunk detected, SIGABRT. After: the lease is revoked and the OSD exits with 0.vitastor-cli ddwith bs=4k iodepth=64): all exit with 0, in 22 ms on average and 108 ms at most. The
unpatched OSD did not crash in 300 such stops, which is why the crash is shown with gdb.
Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA