Fix vitastor-nbd spinning forever after NBD_CMD_DISC #142

Merged
vitalif merged 1 commits from koolkhel/vitastor:nbd-disc-exit into master 2026-11-21 11:28:20 +03:00
Contributor

Description

After vitastor-nbd unmap, the server process could stay forever at 100 % CPU with its cluster connections open, ignoring SIGTERM. The NBD_CMD_DISC branch of handle_finished_read() closed the socket and returned without preparing the next read. The parse loop in handle_read() then never advanced over the bytes that followed the disconnect request in the same recvmsg(). stop and the signals are only checked in the main loop, which was never reached again.

The kernel does send requests after the disconnect request in ioctl mode. NBD_DISCONNECT only queues it, and the next open of the device re-reads the partition table. In the 64k write runs below one such read was in flight after unmap, besides the test's own write, and the kernel log has nbd0: unable to read partition table.

The closed descriptor also stayed in use. When the read that carried the disconnect request filled the receive buffer, the next recvmsg() went to the closed descriptor, and the process exited with code 1 (Socket disconnected: Bad file descriptor) in the middle of the final sync.

Now nbd_fd is set to -1 once the socket is closed, on NBD_CMD_DISC and on hang-up. Parsing, submit_read() and submit_send() stop after that, and a completion that arrives after the close no longer ends in exit(1).

Tested as root against a local 3-OSD cluster (tests/run_3osds.sh, with IMMEDIATE_COMMIT=1) on master cf5c466e. The order of requests is forced: map --foreground 1, open the device, kill -STOP the server, unmap, start a direct dd through the descriptor opened before and wait until /sys/block/nbdN/inflight counts it, kill -CONT. The server then gets the disconnect request and the next request in one read.

4k read after the disconnect 64k write after the disconnect
master still running after 30 s: state R, 95 % CPU, ignores SIGTERM the same
this PR exits with code 0 within 1 s exits with code 0 within 1 s

The second problem shows with immediate_commit=none and an unsynced write before the disconnect. A build with only the loop fixed (while (result > 0 && !stop)) then exits with code 1 and Socket disconnected: Bad file descriptor in the 64k case. This PR exits with code 0.

It touches the same file as #120, but not the same lines: the two merge cleanly.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description After `vitastor-nbd unmap`, the server process could stay forever at 100 % CPU with its cluster connections open, ignoring SIGTERM. The `NBD_CMD_DISC` branch of `handle_finished_read()` closed the socket and returned without preparing the next read. The parse loop in `handle_read()` then never advanced over the bytes that followed the disconnect request in the same `recvmsg()`. `stop` and the signals are only checked in the main loop, which was never reached again. The kernel does send requests after the disconnect request in ioctl mode. `NBD_DISCONNECT` only queues it, and the next open of the device re-reads the partition table. In the 64k write runs below one such read was in flight after `unmap`, besides the test's own write, and the kernel log has `nbd0: unable to read partition table`. The closed descriptor also stayed in use. When the read that carried the disconnect request filled the receive buffer, the next `recvmsg()` went to the closed descriptor, and the process exited with code 1 (`Socket disconnected: Bad file descriptor`) in the middle of the final sync. Now `nbd_fd` is set to -1 once the socket is closed, on `NBD_CMD_DISC` and on hang-up. Parsing, `submit_read()` and `submit_send()` stop after that, and a completion that arrives after the close no longer ends in `exit(1)`. Tested as root against a local 3-OSD cluster (`tests/run_3osds.sh`, with `IMMEDIATE_COMMIT=1`) on master cf5c466e. The order of requests is forced: `map --foreground 1`, open the device, `kill -STOP` the server, `unmap`, start a direct `dd` through the descriptor opened before and wait until `/sys/block/nbdN/inflight` counts it, `kill -CONT`. The server then gets the disconnect request and the next request in one read. | | 4k read after the disconnect | 64k write after the disconnect | |---|---|---| | master | still running after 30 s: state R, 95 % CPU, ignores SIGTERM | the same | | this PR | exits with code 0 within 1 s | exits with code 0 within 1 s | The second problem shows with `immediate_commit=none` and an unsynced write before the disconnect. A build with only the loop fixed (`while (result > 0 && !stop)`) then exits with code 1 and `Socket disconnected: Bad file descriptor` in the 64k case. This PR exits with code 0. It touches the same file as #120, but not the same lines: the two merge cleanly. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-11-21 10:57:01 +03:00
The NBD_CMD_DISC branch closed the socket and returned without
preparing the next read, so handle_read() looped forever on any bytes
that followed the disconnect request in the same recvmsg(), ignoring
`stop` and SIGTERM. The kernel sends such requests after the
NBD_DISCONNECT ioctl, for example when the device is opened and its
partition table is re-read.

The closed descriptor also stayed in use: a read or a reply submitted
after it failed with EBADF, and the process exited with code 1 in the
middle of the final sync. Forget the descriptor once it is closed and
stop parsing, reading and sending after that.
vitalif merged commit 7cc4553e01 into master 2026-11-21 11:28:20 +03:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#142
No description provided.