Fix vitastor-nbd spinning forever after NBD_CMD_DISC #142
Reference in New Issue
Block a user
No description provided.
Delete Branch "koolkhel/vitastor:nbd-disc-exit"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
After
vitastor-nbd unmap, the server process could stay forever at 100 % CPU with its cluster connections open, ignoring SIGTERM. TheNBD_CMD_DISCbranch ofhandle_finished_read()closed the socket and returned without preparing the next read. The parse loop inhandle_read()then never advanced over the bytes that followed the disconnect request in the samerecvmsg().stopand the signals are only checked in the main loop, which was never reached again.The kernel does send requests after the disconnect request in ioctl mode.
NBD_DISCONNECTonly queues it, and the next open of the device re-reads the partition table. In the 64k write runs below one such read was in flight afterunmap, besides the test's own write, and the kernel log hasnbd0: unable to read partition table.The closed descriptor also stayed in use. When the read that carried the disconnect request filled the receive buffer, the next
recvmsg()went to the closed descriptor, and the process exited with code 1 (Socket disconnected: Bad file descriptor) in the middle of the final sync.Now
nbd_fdis set to -1 once the socket is closed, onNBD_CMD_DISCand on hang-up. Parsing,submit_read()andsubmit_send()stop after that, and a completion that arrives after the close no longer ends inexit(1).Tested as root against a local 3-OSD cluster (
tests/run_3osds.sh, withIMMEDIATE_COMMIT=1) on mastercf5c466e. The order of requests is forced:map --foreground 1, open the device,kill -STOPthe server,unmap, start a directddthrough the descriptor opened before and wait until/sys/block/nbdN/inflightcounts it,kill -CONT. The server then gets the disconnect request and the next request in one read.The second problem shows with
immediate_commit=noneand an unsynced write before the disconnect. A build with only the loop fixed (while (result > 0 && !stop)) then exits with code 1 andSocket disconnected: Bad file descriptorin the 64k case. This PR exits with code 0.It touches the same file as #120, but not the same lines: the two merge cleanly.
Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA