Subscribe to EPOLLOUT only while a send waits for it in no-io_uring mode #144
Open
koolkhel
wants to merge 1 commits from
koolkhel/vitastor:no-uring-epollout-on-demand into master
pull from: koolkhel/vitastor:no-uring-epollout-on-demand
merge into: vitalif:master
vitalif:master
vitalif:primary-stickiness
vitalif:libvfn
vitalif:canceling
vitalif:immediate-repeering
vitalif:rdma-close-tcp
vitalif:use-degraded-writes
vitalif:fix-write-delete-flow
vitalif:openssl-3.0.12
vitalif:openssl-capture-secrets
vitalif:ignore-small-rdma-end
vitalif:encrypt-batch
vitalif:faster-peering-calc
vitalif:http-hello
vitalif:xxh3-bench
vitalif:create-user-etcd-based
vitalif:aesxts_streaming_nosupport
vitalif:refactor-heap-continue-write
vitalif:ston3lu-patch-0127
vitalif:rdma-credits
vitalif:heap-meta
vitalif:sendmsg-zc-no-more-error
vitalif:fix-clear-refs
vitalif:veeam-trick
vitalif:test-etcd-schizo
vitalif:test
vitalif:test1
vitalif:test2
vitalif:cow-meta
vitalif:kv-readahead
vitalif:qemu-fix-crash-idea
vitalif:eventloop-auto-init-trigger
vitalif:hugo-docs
vitalif:node-mutex-and-postpone
vitalif:check-writeback
vitalif:msgr-iothreads-v2
vitalif:antietcd
vitalif:rel-1.4
vitalif:test-fix-ec-unknown-state-51
vitalif:kv-update
vitalif:kv-debug
vitalif:hotfix-1.2.0
vitalif:rdma-simple-nodp
vitalif:zerocopy-tcp-send
vitalif:csi-staging
vitalif:hotfix-1.1.0
vitalif:hotfix-1.0.0
vitalif:fsync-feedback
vitalif:cached-reads
vitalif:qemu-send-loop
vitalif:test-double-alloc
vitalif:hier-failure-domains
vitalif:mon-self-restart
vitalif:epoch-deletions
vitalif:rdma-flow-control
vitalif:csi-use-vitastor-cli
vitalif:rdma-v2
vitalif:etcd-hide-base64
vitalif:lrc-matrix
vitalif:rm-left-on-dead
vitalif:nfs-proxy-old
vitalif:non-odp-rdma
vitalif:test-zctr
vitalif:separate-data-connections
vitalif:sec_osd_msgr
vitalif:test-assert
vitalif:nbd-vmsplice
vitalif:allow-etcd-address-option
vitalif:rdma-zerocopy
vitalif:rel-0.5
vitalif:openonload
vitalif:sync-io-test
vitalif:trace-sync
vitalif:trace-sqes
vitalif:test-submit-and-wait
vitalif:test-sq-poll
vitalif:blocking-uring-test
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
No description provided.
Delete Branch "koolkhel/vitastor:no-uring-epollout-on-demand"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
QEMU runs the fd handlers of a client without io_uring in a level-triggered loop (no EPOLLET), so the permanent EPOLLOUT subscription added in
b88e2ea6calls the write handler on every pass while the socket is writable, and an idle client spins at 100% CPU. Subscribe to EPOLLOUT only when a send parks on a full socket, and unsubscribe on EPOLLOUT. A missed edge is not possible: epoll_manager_t::set_fd_handler() polls the fd when it registers it and delivers EPOLLOUT if the socket is already writable.Tested: new test_no_uring_epollout_on_demand in test_msgr_stop. It drives the messenger without a ringloop through a level-triggered poll() loop, the way QEMU calls the handlers. Without the fix, the write handler is called on all 100 idle passes and the test fails. With it, there are no idle calls, a send to a peer that doesn't read parks with EPOLLOUT subscribed, and it completes once the peer drains, with the subscription dropped again. Also fio with -no_io_uring=1 against a local 3-OSD cluster (run_3osds.sh): tests/test_no_uring.sh, and a 512M 4M-block write with -verify=crc32c, where sends parked on EAGAIN and resumed.
A separate issue in the same function, not fixed here: in handle_peer_epoll(), the EPOLLOUT branch looks the client up with
clients.find(cl->client_id)after read_requests(). read_requests() can stop the client, and stop_client() can deletecl(msgr_stop.cpp), so that reads freed memory. Savingcl->client_idbefore the EPOLLIN branch avoids it.Contributor Name
Yury Luneff (MIND Software LLC, https://mindsw.io)
CLA
Какой реальный юзкейс мы здесь решаем? Level-triggered как будто нет желания поддерживать.
Юзкейс — QEMU (или qemu-img) там, где io_uring_setup запрещён:
Режим без io_uring у вас поддерживается:
e4a7e9bf, test_no_uring.sh в run_tests.sh. Драйвер в этом случае сам уходит в запасной путь: «failed to create io_uring - I/O will be slower» иvitastor_c_create_qemu()(src/client/qemu_driver.c:695-699). Аvitastor_c_create_qemu()регистрирует каждый сокет OSD в AioContext QEMU через aio_set_fd_handler (vitastor_c.cpp:87-103, vitastor_c_create_qemu_common). AioContext работает по уровню, это не мы его выбрали. Послеb88e2ea6у сокета постоянно висит fd_write, поэтому QEMU зовёт его на каждом проходе цикла, пока сокет доступен на запись. Простаивающий процесс занимает 100 % ядра. Мерили на qemu-storage-daemon с экспортом vduse-blk и kernel.io_uring_disabled=2: 100 % в простое без этого PR, 0 % с ним, запись 1 ГиБ с проверкой crc32c проходит в обоих случаях. Доe4a7e9bfтот же сценарий зависал на первой отправке в OSD.Если не хочется поддерживать level-triggered в мессенджере, есть вариант меньше и в духе epoll_manager. В запасном пути драйвера создавать клиента через
vitastor_c_create_epoll_json()и регистрировать в AioContext один fd —vitastor_c_epoll_get_fd(), обработчик чтенияvitastor_c_epoll_handle_events(proxy, 0). Так уже сделано с uring_eventfd: qemu_driver.c:671-684, attach/detach — 807-814, 856-858, 880-882. Мессенджер тогда остаётся edge-triggered, EPOLLOUT изb88e2ea6трогать не нужно. QEMU видит только epoll fd, а он читаем, только пока есть события. Изменения только в qemu_driver.c, этот PR тогда закрою. Переделать так?View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.