client: resend only the unsent tail after a short io_uring send #122
Open
lei.xianqiang
wants to merge 2 commits from
lei.xianqiang/vitastor:fix_io_uring_short_send into master
pull from: lei.xianqiang/vitastor:fix_io_uring_short_send
merge into: vitalif:master
vitalif:master
vitalif:tombstones
vitalif:primary-stickiness
vitalif:libvfn
vitalif:canceling
vitalif:immediate-repeering
vitalif:rdma-close-tcp
vitalif:use-degraded-writes
vitalif:fix-write-delete-flow
vitalif:openssl-3.0.12
vitalif:openssl-capture-secrets
vitalif:ignore-small-rdma-end
vitalif:encrypt-batch
vitalif:faster-peering-calc
vitalif:http-hello
vitalif:xxh3-bench
vitalif:create-user-etcd-based
vitalif:aesxts_streaming_nosupport
vitalif:refactor-heap-continue-write
vitalif:ston3lu-patch-0127
vitalif:rdma-credits
vitalif:heap-meta
vitalif:sendmsg-zc-no-more-error
vitalif:fix-clear-refs
vitalif:veeam-trick
vitalif:test-etcd-schizo
vitalif:test
vitalif:test1
vitalif:test2
vitalif:cow-meta
vitalif:kv-readahead
vitalif:qemu-fix-crash-idea
vitalif:eventloop-auto-init-trigger
vitalif:hugo-docs
vitalif:node-mutex-and-postpone
vitalif:check-writeback
vitalif:msgr-iothreads-v2
vitalif:antietcd
vitalif:rel-1.4
vitalif:test-fix-ec-unknown-state-51
vitalif:kv-update
vitalif:kv-debug
vitalif:hotfix-1.2.0
vitalif:rdma-simple-nodp
vitalif:zerocopy-tcp-send
vitalif:csi-staging
vitalif:hotfix-1.1.0
vitalif:hotfix-1.0.0
vitalif:fsync-feedback
vitalif:cached-reads
vitalif:qemu-send-loop
vitalif:test-double-alloc
vitalif:hier-failure-domains
vitalif:mon-self-restart
vitalif:epoch-deletions
vitalif:rdma-flow-control
vitalif:csi-use-vitastor-cli
vitalif:rdma-v2
vitalif:etcd-hide-base64
vitalif:lrc-matrix
vitalif:rm-left-on-dead
vitalif:nfs-proxy-old
vitalif:non-odp-rdma
vitalif:test-zctr
vitalif:separate-data-connections
vitalif:sec_osd_msgr
vitalif:test-assert
vitalif:nbd-vmsplice
vitalif:allow-etcd-address-option
vitalif:rdma-zerocopy
vitalif:rel-0.5
vitalif:openonload
vitalif:sync-io-test
vitalif:trace-sync
vitalif:trace-sqes
vitalif:test-submit-and-wait
vitalif:test-sq-poll
vitalif:blocking-uring-test
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
No description provided.
Delete Branch "lei.xianqiang/vitastor:fix_io_uring_short_send"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
A short io_uring send (sendmsg/sendmsg_zc with MSG_WAITALL) used to
disconnect the peer. With large replies such as SEC_LIST filling the
socket buffer this happened repeatedly and caused endless re-peering.
Now the already accepted prefix stays on the zerocopy path and waits for
its release notification as usual, while only the unsent tail is resent
with a copying sendmsg before any new ops are appended. -EAGAIN/-EINTR
completions are handled the same way, as a zero-byte short send, instead
of leaving the batch stuck in send_list.
The send_free range of the short-written batch is recorded once, on its
first short send, and released only after both the tail is fully sent
and, for a zerocopy send, its release notification arrives.
Notifications of other batches are still released immediately, so none
of them can be lost and pin send_free.
Contributor Name
Xianqiang Lei (Datatom, https://www.datatom.com/)
CLA
0f04c5a3ac(client: resend only the unsent tail after a short io_uring send)0f04c5a3ac(client: resend only the unsent tail after a short io_uring send)In issue, it is mentioned that this issue looks like a kernel bug. I have not found it in the kernel yet, and I have only modified the Vitastor code for testing. Could you run test_msgr_send_reply in your environments with different kernel versions to test whether short io_uring send also exists?
0f04c5a3acto7d297971cb7d297971cbto38d25bf8cfWhen a short write occurs, io_uring attempts to retry through asynchronous polling, with a maximum of 128 retries, before falling back to io-wq. Since subsequent retries may still fail, causing the request to complete with the number of bytes already sent, setting
MSG_WAITALLdoes not guarantee that all data will be sent successfully.ftrace:
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.