client: resend only the unsent tail after a short io_uring send #122

Open
lei.xianqiang wants to merge 2 commits from lei.xianqiang/vitastor:fix_io_uring_short_send into master
First-time contributor

Description

A short io_uring send (sendmsg/sendmsg_zc with MSG_WAITALL) used to
disconnect the peer. With large replies such as SEC_LIST filling the
socket buffer this happened repeatedly and caused endless re-peering.
Now the already accepted prefix stays on the zerocopy path and waits for
its release notification as usual, while only the unsent tail is resent
with a copying sendmsg before any new ops are appended. -EAGAIN/-EINTR
completions are handled the same way, as a zero-byte short send, instead
of leaving the batch stuck in send_list.
The send_free range of the short-written batch is recorded once, on its
first short send, and released only after both the tail is fully sent
and, for a zerocopy send, its release notification arrives.
Notifications of other batches are still released immediately, so none
of them can be lost and pin send_free.

Contributor Name

Xianqiang Lei (Datatom, https://www.datatom.com/)

CLA

### Description A short io_uring send (sendmsg/sendmsg_zc with MSG_WAITALL) used to disconnect the peer. With large replies such as SEC_LIST filling the socket buffer this happened repeatedly and caused endless re-peering. Now the already accepted prefix stays on the zerocopy path and waits for its release notification as usual, while only the unsent tail is resent with a copying sendmsg before any new ops are appended. -EAGAIN/-EINTR completions are handled the same way, as a zero-byte short send, instead of leaving the batch stuck in send_list. The send_free range of the short-written batch is recorded once, on its first short send, and released only after both the tail is fully sent and, for a zerocopy send, its release notification arrives. Notifications of other batches are still released immediately, so none of them can be lost and pin send_free. ### Contributor Name Xianqiang Lei (Datatom, https://www.datatom.com/) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
lei.xianqiang added 2 commits 2026-11-17 10:23:09 +03:00
A short io_uring send (sendmsg/sendmsg_zc with MSG_WAITALL) used to
disconnect the peer. With large replies such as SEC_LIST filling the
socket buffer this happened repeatedly and caused endless re-peering.
Now the already accepted prefix stays on the zerocopy path and waits for
its release notification as usual, while only the unsent tail is resent
with a copying sendmsg before any new ops are appended. -EAGAIN/-EINTR
completions are handled the same way, as a zero-byte short send, instead
of leaving the batch stuck in send_list.
The send_free range of the short-written batch is recorded once, on its
first short send, and released only after both the tail is fully sent
and, for a zerocopy send, its release notification arrives.
Notifications of other batches are still released immediately, so none
of them can be lost and pin send_free.
Author
First-time contributor
  1. Env Info
[root@node01 ~]# uname -a
Linux node01 6.6.0-28.0.0.34.oe2403.x86_64 #1 SMP Mon May 27 22:22:46 CST 2024 x86_64 x86_64 x86_64 GNU/Linu()

Net: 2×25GbE, Bond 4
  1. Test without 0f04c5a3ac (client: resend only the unsent tail after a short io_uring send)
# /root/test_msgr_send_reply server --ip 10.0.40.115 --net tcp --zerocopy
 
# /root/test_msgr_send_reply client --ip 10.0.40.115 --net tcp --send_size 300 --reply_size 1 --zerocopy
[client] filling 314572800 byte pattern
[client] osd=1 send=300 MiB (314572800 bytes) reply=1 MiB (1048576 bytes) transport=tcp zerocopy=1 sendmsg_zc=1 rdma_ready=0
Connecting to OSD 2 at 10.0.40.115:11223 (client 1, FD 6)
[client] connecting to 10.0.40.115:11223
[OSD 1] Connected with peer OSD 2 (client 1)
[client] connected to OSD 2 client_id=1 mode=tcp
[client] sending one SEC_WRITE of 314572800 bytes
Client 1 socket write error: expected to send 314572936 bytes with MSG_WAITALL but sent 240441708. Disconnecting client
[OSD 1] Stopping client 1 (OSD peer 2)
[client] SEC_WRITE retval=-32 (Broken pipe), 120.6 ms
[client] FAIL: SEC_WRITE was not accepted
  1. Test with 0f04c5a3ac (client: resend only the unsent tail after a short io_uring send)
# /root/test_msgr_send_reply server --ip 10.0.40.115 --net tcp --zerocopy

# /root/test_msgr_send_reply client --ip 10.0.40.115 --net tcp --send_size 300 --reply_size 1 --zerocopy
[client] filling 314572800 byte pattern
[client] osd=1 send=300 MiB (314572800 bytes) reply=1 MiB (1048576 bytes) transport=tcp zerocopy=1 sendmsg_zc=1 rdma_ready=0
Connecting to OSD 2 at 10.0.40.115:11223 (client 1, FD 6)
[client] connecting to 10.0.40.115:11223
[OSD 1] Connected with peer OSD 2 (client 1)
[client] connected to OSD 2 client_id=1 mode=tcp
[client] sending one SEC_WRITE of 314572800 bytes
[client] SEC_WRITE 314572800 bytes ok, 198.8 ms, 1509.0 MiB/s
[client] sending one SEC_READ of 1048576 bytes
[client] SEC_READ 1048576 bytes ok, 3.2 ms, 316.9 MiB/s
[client] PASS
[OSD 1] Stopping client 1 (OSD peer 2)
1. Env Info ``` [root@node01 ~]# uname -a Linux node01 6.6.0-28.0.0.34.oe2403.x86_64 #1 SMP Mon May 27 22:22:46 CST 2024 x86_64 x86_64 x86_64 GNU/Linu() Net: 2×25GbE, Bond 4 ``` 2. Test without 0f04c5a3ac (client: resend only the unsent tail after a short io_uring send) ``` # /root/test_msgr_send_reply server --ip 10.0.40.115 --net tcp --zerocopy # /root/test_msgr_send_reply client --ip 10.0.40.115 --net tcp --send_size 300 --reply_size 1 --zerocopy [client] filling 314572800 byte pattern [client] osd=1 send=300 MiB (314572800 bytes) reply=1 MiB (1048576 bytes) transport=tcp zerocopy=1 sendmsg_zc=1 rdma_ready=0 Connecting to OSD 2 at 10.0.40.115:11223 (client 1, FD 6) [client] connecting to 10.0.40.115:11223 [OSD 1] Connected with peer OSD 2 (client 1) [client] connected to OSD 2 client_id=1 mode=tcp [client] sending one SEC_WRITE of 314572800 bytes Client 1 socket write error: expected to send 314572936 bytes with MSG_WAITALL but sent 240441708. Disconnecting client [OSD 1] Stopping client 1 (OSD peer 2) [client] SEC_WRITE retval=-32 (Broken pipe), 120.6 ms [client] FAIL: SEC_WRITE was not accepted ``` 3. Test with 0f04c5a3ac (client: resend only the unsent tail after a short io_uring send) ``` # /root/test_msgr_send_reply server --ip 10.0.40.115 --net tcp --zerocopy # /root/test_msgr_send_reply client --ip 10.0.40.115 --net tcp --send_size 300 --reply_size 1 --zerocopy [client] filling 314572800 byte pattern [client] osd=1 send=300 MiB (314572800 bytes) reply=1 MiB (1048576 bytes) transport=tcp zerocopy=1 sendmsg_zc=1 rdma_ready=0 Connecting to OSD 2 at 10.0.40.115:11223 (client 1, FD 6) [client] connecting to 10.0.40.115:11223 [OSD 1] Connected with peer OSD 2 (client 1) [client] connected to OSD 2 client_id=1 mode=tcp [client] sending one SEC_WRITE of 314572800 bytes [client] SEC_WRITE 314572800 bytes ok, 198.8 ms, 1509.0 MiB/s [client] sending one SEC_READ of 1048576 bytes [client] SEC_READ 1048576 bytes ok, 3.2 ms, 316.9 MiB/s [client] PASS [OSD 1] Stopping client 1 (OSD peer 2) ```
Author
First-time contributor

In issue, it is mentioned that this issue looks like a kernel bug. I have not found it in the kernel yet, and I have only modified the Vitastor code for testing. Could you run test_msgr_send_reply in your environments with different kernel versions to test whether short io_uring send also exists?

In [issue](https://github.com/vitalif/vitastor/issues/145#issuecomment-6848289391), it is mentioned that this issue looks like a kernel bug. I have not found it in the kernel yet, and I have only modified the Vitastor code for testing. Could you run test_msgr_send_reply in your environments with different kernel versions to test whether short io_uring send also exists?
lei.xianqiang force-pushed fix_io_uring_short_send from 0f04c5a3ac to 7d297971cb 2026-11-17 11:44:16 +03:00 Compare
lei.xianqiang force-pushed fix_io_uring_short_send from 7d297971cb to 38d25bf8cf 2026-11-17 11:55:20 +03:00 Compare
Author
First-time contributor

In issue, it is mentioned that this issue looks like a kernel bug. I have not found it in the kernel yet, and I have only modified the Vitastor code for testing. Could you run test_msgr_send_reply in your environments with different kernel versions to test whether short io_uring send also exists?

#define APOLL_MAX_RETRY 128

When a short write occurs, io_uring attempts to retry through asynchronous polling, with a maximum of 128 retries, before falling back to io-wq. Since subsequent retries may still fail, causing the request to complete with the number of bytes already sent, setting MSG_WAITALL does not guarantee that all data will be sent successfully.

ftrace:

......
test_msgr_send_-661221  [047] ..... 2336186.988149: io_uring_task_work_run: tctx 00000000264ae7e1, count 1, loops 1
          <idle>-0       [010] d.s.. 2336186.989184: io_uring_task_add: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 304
 test_msgr_send_-661221  [047] ..... 2336186.989822: sock_send_length: sk address = 00000000b7a345ff, family = AF_INET protocol = IPPROTO_TCP, length = 4259248, error = 0, flags = 0x0
 test_msgr_send_-661221  [047] ..... 2336186.989823: io_uring_poll_arm: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 0xc000010e, events 0xc000213e
 test_msgr_send_-661221  [047] ..... 2336186.989823: io_uring_task_work_run: tctx 00000000264ae7e1, count 1, loops 1
          <idle>-0       [010] d.s.. 2336186.990299: io_uring_task_add: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 304
 test_msgr_send_-661221  [047] ..... 2336186.990715: sock_send_length: sk address = 00000000b7a345ff, family = AF_INET protocol = IPPROTO_TCP, length = 2129624, error = 0, flags = 0x0
 test_msgr_send_-661221  [047] ..... 2336186.990716: io_uring_poll_arm: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 0xc000010e, events 0xc000213e
 test_msgr_send_-661221  [047] ..... 2336186.990716: io_uring_task_work_run: tctx 00000000264ae7e1, count 1, loops 1
          <idle>-0       [010] d.s.. 2336186.991147: io_uring_task_add: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 304
 test_msgr_send_-661221  [047] ..... 2336186.991424: sock_send_length: sk address = 00000000b7a345ff, family = AF_INET protocol = IPPROTO_TCP, length = 1941716, error = 0, flags = 0x0
 test_msgr_send_-661221  [047] ..... 2336186.991426: io_uring_queue_async_work: ring 00000000b6112414, request 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, flags 0x32246000, normal queue, work 000000007beffc4a
 test_msgr_send_-661221  [047] ..... 2336186.991471: io_uring_task_work_run: tctx 00000000264ae7e1, count 1, loops 1
  iou-wrk-661221-661274  [018] ..... 2336186.991518: sock_send_length: sk address = 00000000b7a345ff, family = AF_INET protocol = IPPROTO_TCP, length = 0, error = -11, flags = 0x0
  iou-wrk-661221-661274  [018] ..... 2336186.991531: io_uring_complete: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, result 353904504, cflags 0x0 extra1 0 extra2 0
> In [issue](https://github.com/vitalif/vitastor/issues/145#issuecomment-6848289391), it is mentioned that this issue looks like a kernel bug. I have not found it in the kernel yet, and I have only modified the Vitastor code for testing. Could you run test_msgr_send_reply in your environments with different kernel versions to test whether short io_uring send also exists? > #define APOLL_MAX_RETRY 128 When a short write occurs, io_uring attempts to retry through asynchronous polling, with a maximum of 128 retries, before falling back to io-wq. Since subsequent retries may still fail, causing the request to complete with the number of bytes already sent, setting `MSG_WAITALL` does not guarantee that all data will be sent successfully. ftrace: ``` ...... test_msgr_send_-661221 [047] ..... 2336186.988149: io_uring_task_work_run: tctx 00000000264ae7e1, count 1, loops 1 <idle>-0 [010] d.s.. 2336186.989184: io_uring_task_add: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 304 test_msgr_send_-661221 [047] ..... 2336186.989822: sock_send_length: sk address = 00000000b7a345ff, family = AF_INET protocol = IPPROTO_TCP, length = 4259248, error = 0, flags = 0x0 test_msgr_send_-661221 [047] ..... 2336186.989823: io_uring_poll_arm: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 0xc000010e, events 0xc000213e test_msgr_send_-661221 [047] ..... 2336186.989823: io_uring_task_work_run: tctx 00000000264ae7e1, count 1, loops 1 <idle>-0 [010] d.s.. 2336186.990299: io_uring_task_add: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 304 test_msgr_send_-661221 [047] ..... 2336186.990715: sock_send_length: sk address = 00000000b7a345ff, family = AF_INET protocol = IPPROTO_TCP, length = 2129624, error = 0, flags = 0x0 test_msgr_send_-661221 [047] ..... 2336186.990716: io_uring_poll_arm: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 0xc000010e, events 0xc000213e test_msgr_send_-661221 [047] ..... 2336186.990716: io_uring_task_work_run: tctx 00000000264ae7e1, count 1, loops 1 <idle>-0 [010] d.s.. 2336186.991147: io_uring_task_add: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, mask 304 test_msgr_send_-661221 [047] ..... 2336186.991424: sock_send_length: sk address = 00000000b7a345ff, family = AF_INET protocol = IPPROTO_TCP, length = 1941716, error = 0, flags = 0x0 test_msgr_send_-661221 [047] ..... 2336186.991426: io_uring_queue_async_work: ring 00000000b6112414, request 00000000670dce91, user_data 0xf3e1a0, opcode SENDMSG, flags 0x32246000, normal queue, work 000000007beffc4a test_msgr_send_-661221 [047] ..... 2336186.991471: io_uring_task_work_run: tctx 00000000264ae7e1, count 1, loops 1 iou-wrk-661221-661274 [018] ..... 2336186.991518: sock_send_length: sk address = 00000000b7a345ff, family = AF_INET protocol = IPPROTO_TCP, length = 0, error = -11, flags = 0x0 iou-wrk-661221-661274 [018] ..... 2336186.991531: io_uring_complete: ring 00000000b6112414, req 00000000670dce91, user_data 0xf3e1a0, result 353904504, cflags 0x0 extra1 0 extra2 0 ```
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u fix_io_uring_short_send:lei.xianqiang-fix_io_uring_short_send
git checkout lei.xianqiang-fix_io_uring_short_send
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#122
No description provided.