osd: Stop on SIGINT/SIGTERM from the main loop, not the signal handler #141

Merged
vitalif merged 1 commits from koolkhel/vitastor:osd-signal-stop into master 2026-11-20 19:10:55 +03:00
Contributor

Description

The OSD's SIGINT/SIGTERM handler calls osd_t::force_stop(), which builds a JSON object,
allocates strings and starts an HTTP request to etcd. None of that is async-signal-safe. The
OSD is single-threaded, so when the signal interrupts malloc() or free() in the main loop,
the handler re-enters the allocator and corrupts the heap. The OSD then dies in malloc with
SIGSEGV or SIGABRT instead of exiting with 0. With glibc 2.28 it shows as a SIGSEGV in
malloc or as malloc_consolidate(): invalid chunk size within a millisecond of the stop
signal.

The fix follows aefaa2ee (NBD, UBLK, NFS): the handler only sets a flag, and the main loop
calls force_stop(0). A second signal still exits at once, now with _exit().

One limit, the same as in the NBD loop: a signal that arrives between the flag check and
io_uring_wait_cqe() is handled at the next ring event. On an idle OSD that is at most 1
second later (the messenger keepalive timer).

Tested (glibc 2.39, x86-64):

  • gdb, SIGTERM delivered in the middle of malloc()'s tcache fast path (after the free chunk
    is read, before the list head is updated). Before: malloc(): unaligned tcache chunk detected, SIGABRT. After: the lease is revoked and the OSD exits with 0.
  • SIGTERM at random moments on an idle OSD, 10 times: exit 0 in 12–21 ms.
  • 200 SIGTERM stops of OSDs under write load (3 OSDs, a replicated pool, vitastor-cli dd
    with bs=4k iodepth=64): all exit with 0, in 22 ms on average and 108 ms at most. The
    unpatched OSD did not crash in 300 such stops, which is why the crash is shown with gdb.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description The OSD's SIGINT/SIGTERM handler calls `osd_t::force_stop()`, which builds a JSON object, allocates strings and starts an HTTP request to etcd. None of that is async-signal-safe. The OSD is single-threaded, so when the signal interrupts `malloc()` or `free()` in the main loop, the handler re-enters the allocator and corrupts the heap. The OSD then dies in `malloc` with SIGSEGV or SIGABRT instead of exiting with 0. With glibc 2.28 it shows as a SIGSEGV in `malloc` or as `malloc_consolidate(): invalid chunk size` within a millisecond of the stop signal. The fix follows aefaa2ee (NBD, UBLK, NFS): the handler only sets a flag, and the main loop calls `force_stop(0)`. A second signal still exits at once, now with `_exit()`. One limit, the same as in the NBD loop: a signal that arrives between the flag check and `io_uring_wait_cqe()` is handled at the next ring event. On an idle OSD that is at most 1 second later (the messenger keepalive timer). Tested (glibc 2.39, x86-64): - gdb, SIGTERM delivered in the middle of `malloc()`'s tcache fast path (after the free chunk is read, before the list head is updated). Before: `malloc(): unaligned tcache chunk detected`, SIGABRT. After: the lease is revoked and the OSD exits with 0. - SIGTERM at random moments on an idle OSD, 10 times: exit 0 in 12–21 ms. - 200 SIGTERM stops of OSDs under write load (3 OSDs, a replicated pool, `vitastor-cli dd` with bs=4k iodepth=64): all exit with 0, in 22 ms on average and 108 ms at most. The unpatched OSD did not crash in 300 such stops, which is why the crash is shown with gdb. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-11-20 16:29:20 +03:00
The handler called osd_t::force_stop(), which builds a JSON object,
allocates strings and starts an HTTP request to etcd. None of that is
async-signal-safe. When the signal interrupts malloc() or free() in the
main loop, the handler re-enters the allocator and corrupts the heap,
and the OSD dies in malloc with SIGSEGV or SIGABRT instead of exiting
with 0.

Only set a flag in the handler, like the NBD, UBLK and NFS servers do,
and call force_stop() from the main loop. A second signal still exits
at once, now with _exit().
vitalif merged commit 971d45def6 into master 2026-11-20 19:10:55 +03:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#141
No description provided.