Changes in opdef.c

3a4689ac - io_uring/cmd: add iovec cache for commands

Fri, 21 Mar 2025 18:04:33 +0000

io_uring/cmd: add iovec cache for commandsAdd iou_vec to commands and wire caching for it, but don't expose it tousers just yet. We need the vec cleared on initial alloc, but sincewe can't place it at the beginning at the moment, zero the entireasync_data. It's cached, and the performance effects only the initialallocation, and it might be not a bad idea since we're exposing thosebits to outside drivers.Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/c0f2145b75791bc6106eb4e72add2cf6a2c72a7a.1742579999.git.asml.silence@gmail.comSigned-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

5f14404b - io_uring/cmd: don't expose entire cmd async data

Wed, 19 Mar 2025 06:12:48 +0000

io_uring/cmd: don't expose entire cmd async dataio_uring needs private bits in cmd's ->async_data, and they should neverbe exposed to drivers as it'd certainly be abused. Leave structio_uring_cmd_data for the drivers but wrap it into a structure. It's aprep patch and doesn't do anything useful yet.Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/20250319061251.21452-3-sidong.yang@furiosa.aiSigned-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

bdabba04 - io_uring/rw: implement vectored registered rw

Fri, 07 Mar 2025 16:00:31 +0000

io_uring/rw: implement vectored registered rwImplement registered buffer vectored reads with new opcodesIORING_OP_WRITEV_FIXED and IORING_OP_READV_FIXED.Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/d7c89eb481e870f598edc91cc66ff4d1e4ae3788.1741362889.git.asml.silence@gmail.comSigned-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

ff92d824 - io_uring/rw: move fixed buffer import to issue path

Thu, 27 Feb 2025 22:39:12 +0000

io_uring/rw: move fixed buffer import to issue pathRegistered buffers may depend on a linked command, which makes the preppath too early to import. Move to the issue path when the node isactually needed like all the other users of fixed buffers.Signed-off-by: Keith Busch Link: https://lore.kernel.org/r/20250227223916.143006-3-kbusch@meta.comReviewed-by: Ming Lei Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

19f7e942 - io_uring/epoll: add support for IORING_OP_EPOLL_WAIT

Fri, 31 Jan 2025 21:29:09 +0000

io_uring/epoll: add support for IORING_OP_EPOLL_WAITFor existing epoll event loops that can't fully convert to io_uring,the used approach is usually to add the io_uring fd to the epollinstance and use epoll_wait() to wait on both "legacy" and io_uringevents. While this work, it isn't optimal as:1) epoll_wait() is pretty limited in what it can do. It does not support partial reaping of events, or waiting on a batch of events.2) When an io_uring ring is added to an epoll instance, it activates the io_uring "I'm being polled" logic which slows things down.Rather than use this approach, with EPOLL_WAIT support added to io_uring,event loops can use the normal io_uring wait logic for everything, aslong as an epoll wait request has been armed with io_uring.Note that IORING_OP_EPOLL_WAIT does NOT take a timeout value, as thisis an async request. Waiting on io_uring events in general has varioustimeout parameters, and those are the ones that should be used whenwaiting on any kind of request. If events are immediately available forreaping, then This opcode will return those immediately. If none areavailable, then it will post an async completion when they becomeavailable.cqe->res will contain either an error code (< 0 value) for a malformedrequest, invalid epoll instance, etc. It will return a positive resultindicating how many events were reaped.IORING_OP_EPOLL_WAIT requests may be canceled using the normal io_uringcancelation infrastructure.Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

11ed914b - io_uring/zcrx: add io_recvzc request

Sat, 15 Feb 2025 00:09:41 +0000

io_uring/zcrx: add io_recvzc requestAdd io_uring opcode OP_RECV_ZC for doing zero copy reads out of asocket. Only the connection should be land on the specific rx queue setup for zero copy, and the socket must be handled by the io_uringinstance that the rx queue was registered for zero copy with. That'sbecause neither net_iovs / buffers from our queue can be read by outsideapplications, nor zero copy is possible if traffic for the zero copyconnection goes to another queue. This coordination is outside of thescope of this patch series. Also, any traffic directed to the zero copyenabled queue is immediately visible to the application, which is whyCAP_NET_ADMIN is required at the registration step.Of course, no data is actually read out of the socket, it has alreadybeen copied by the netdev into userspace memory via DMA. OP_RECV_ZCreads skbs out of the socket and checks that its frags are indeednet_iovs that belong to io_uring. A cqe is queued for each one of thesefrags.Recall that each cqe is a big cqe, with the top half being anio_uring_zcrx_cqe. The cqe res field contains the len or error. Thelower IORING_ZCRX_AREA_SHIFT bits of the struct io_uring_zcrx_cqe::offfield contain the offset relative to the start of the zero copy area.The upper part of the off field is trivially zero, and will be usedto carry the area id.For now, there is no limit as to how much work each OP_RECV_ZC requestdoes. It will attempt to drain a socket of all available data. Thisrequest always operates in multishot mode.Reviewed-by: Jens Axboe Signed-off-by: David Wei Acked-by: Jakub Kicinski Link: https://lore.kernel.org/r/20250215000947.789731-7-dw@davidwei.ukSigned-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

dadf03cf - io_uring/cmd: rename struct uring_cache to io_uring_cmd_data

Fri, 03 Jan 2025 15:02:23 +0000

io_uring/cmd: rename struct uring_cache to io_uring_cmd_dataIn preparation for making this more generically available for->uring_cmd() usage that needs stable command data, rename it and moveit to io_uring/cmd.h instead.Signed-off-by: Jens Axboe Signed-off-by: David Sterba List of files: /linux-6.15/io_uring/opdef.c

743fb58a - io_uring/splice: open code 2nd direct file assignment

Mon, 28 Oct 2024 14:03:04 +0000

io_uring/splice: open code 2nd direct file assignmentIn preparation for not pinning the whole registered file table, opencode the second potential direct file assignment. This will be handledby appropriate helpers in the future, for now just do it manually.Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

3e05b222 - io_uring: Fix probe of disabled operations

Wed, 19 Jun 2024 02:06:18 +0000

io_uring: Fix probe of disabled operationsio_probe checks io_issue_def->not_supported, but we never really setthat field, as we mark non-supported functions through a specific ->prephandler. This means we end up returning IO_URING_OP_SUPPORTED, even fordisabled operations. Fix it by just checking the prep handler itself.Fixes: 66f4af93da57 ("io_uring: add support for probing opcodes")Signed-off-by: Gabriel Krisman Bertazi Link: https://lore.kernel.org/r/20240619020620.5301-2-krisman@suse.deSigned-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

ff140cc8 - io_uring: Introduce IORING_OP_LISTEN

Fri, 14 Jun 2024 16:30:47 +0000

io_uring: Introduce IORING_OP_LISTENIORING_OP_LISTEN provides the semantic of listen(2) via io_uring. Whilethis is an essentially synchronous system call, the main point is toenable a network path to execute fully with io_uring registered anddescriptorless files.Signed-off-by: Gabriel Krisman Bertazi Link: https://lore.kernel.org/r/20240614163047.31581-4-krisman@suse.deSigned-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

7481fd93 - io_uring: Introduce IORING_OP_BIND

Fri, 14 Jun 2024 16:30:46 +0000

io_uring: Introduce IORING_OP_BINDIORING_OP_BIND provides the semantic of bind(2) via io_uring. Whilethis is an essentially synchronous system call, the main point is toenable a network path to execute fully with io_uring registered anddescriptorless files.Signed-off-by: Gabriel Krisman Bertazi Link: https://lore.kernel.org/r/20240614163047.31581-3-krisman@suse.deSigned-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

e1123116 - io_uring/rw: Free iovec before cleaning async data

Thu, 30 May 2024 14:23:39 +0000

io_uring/rw: Free iovec before cleaning async datakmemleak shows that there is a memory leak in io_uring read operation,where a buffer is allocated at iovec import, but never de-allocated.The memory is allocated at io_async_rw->free_iovec, but, thenio_async_rw is kfreed, taking the allocated memory with it. I saw thishappening when the read operation fails with -11 (EAGAIN).This is the kmemleak splat. unreferenced object 0xffff8881da591c00 (size 256):... backtrace (crc 7a15bdee): [<00000000256f2de4>] __kmalloc+0x2d6/0x410 [<000000007a9f5fc7>] iovec_from_user.part.0+0xc6/0x160 [<00000000cecdf83a>] __import_iovec+0x50/0x220 [<00000000d1d586a2>] __io_import_iovec+0x13d/0x220 [<0000000054ee9bd2>] io_prep_rw+0x186/0x340 [<00000000a9c0372d>] io_prep_rwv+0x31/0x120 [<000000001d1170b9>] io_prep_readv+0xe/0x30 [<0000000070b8eb67>] io_submit_sqes+0x1bd/0x780 [<00000000812496d4>] __do_sys_io_uring_enter+0x3ed/0x5b0 [<0000000081499602>] do_syscall_64+0x5d/0x170 [<00000000de1c5a4d>] entry_SYSCALL_64_after_hwframe+0x76/0x7eThis occurs because the async data cleanup functions are not set forread/write operations. As a result, the potentially allocated iovec inthe rw async data is not freed before the async data is released,leading to a memory leak.With this following patch, kmemleak does not show the leaked memoryanymore, and all liburing tests pass.Fixes: a9165b83c193 ("io_uring/rw: always setup io_async_rw for read/write requests")Signed-off-by: Breno Leitao Link: https://lore.kernel.org/r/20240530142340.1248216-1-leitao@debian.orgSigned-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

ac5f71a3 - io_uring/net: add provided buffer support for IORING_OP_SEND

Mon, 19 Feb 2024 17:46:44 +0000

io_uring/net: add provided buffer support for IORING_OP_SENDIt's pretty trivial to wire up provided buffer support for the sendside, just like how it's done the receive side. This enables setting upa buffer ring that an application can use to push pending sends to,and then have a send pick a buffer from that ring.One of the challenges with async IO and networking sends is that youcan get into reordering conditions if you have more than one inflightat the same time. Consider the following scenario where everything isfine:1) App queues sendA for socket12) App queues sendB for socket13) App does io_uring_submit()4) sendA is issued, completes successfully, posts CQE5) sendB is issued, completes successfully, posts CQEAll is fine. Requests are always issued in-order, and both completeinline as most sends do.However, if we're flooding socket1 with sends, the following couldalso result from the same sequence:1) App queues sendA for socket12) App queues sendB for socket13) App does io_uring_submit()4) sendA is issued, socket1 is full, poll is armed for retry5) Space frees up in socket1, this triggers sendA retry via task_work6) sendB is issued, completes successfully, posts CQE7) sendA is retried, completes successfully, posts CQENow we've sent sendB before sendA, which can make things unhappy. Ifboth sendA and sendB had been using provided buffers, then it would lookas follows instead:1) App queues dataA for sendA, queues sendA for socket12) App queues dataB for sendB queues sendB for socket13) App does io_uring_submit()4) sendA is issued, socket1 is full, poll is armed for retry5) Space frees up in socket1, this triggers sendA retry via task_work6) sendB is issued, picks first buffer (dataA), completes successfully, posts CQE (which says "I sent dataA")7) sendA is retried, picks first buffer (dataB), completes successfully, posts CQE (which says "I sent dataB")Now we've sent the data in order, and everybody is happy.It's worth noting that this also opens the door for supporting multishotsends, as provided buffers would be a prerequisite for that. Those cantrigger either when new buffers are added to the outgoing ring, or (ifstalled due to lack of space) when space frees up in the socket.Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

e10677a8 - io_uring: drop ->prep_async()

Tue, 19 Mar 2024 02:48:38 +0000

io_uring: drop ->prep_async()It's now unused, drop the code related to it. This includes theio_issue_defs->manual alloc field.While in there, and since ->async_size is now being used a bit morefrequently and in the issue path, move it to io_issue_defs[].Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

d10f19df - io_uring/uring_cmd: switch to always allocating async data

Tue, 19 Mar 2024 02:41:58 +0000

io_uring/uring_cmd: switch to always allocating async dataBasic conversion ensuring async_data is allocated off the prep path. Addsa basic alloc cache as well, as passthrough IO can be quite high in rate.Tested-by: Anuj Gupta Reviewed-by: Anuj Gupta Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

e2ea5a70 - io_uring/net: move connect to always using async data

Tue, 19 Mar 2024 02:37:22 +0000

io_uring/net: move connect to always using async dataWhile doing that, get rid of io_async_connect and just use the genericio_async_msghdr. Both of them have a struct sockaddr_storage in there,and while io_async_msghdr is bigger, if the same type can be used thenthe netmsg_cache can get reused for connect as well.Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

a9165b83 - io_uring/rw: always setup io_async_rw for read/write requests

Mon, 18 Mar 2024 22:13:01 +0000

io_uring/rw: always setup io_async_rw for read/write requestsread/write requests try to put everything on the stack, and then allocand copy if a retry is needed. This necessitates a bunch of nasty codethat deals with intermediate state.Get rid of this, and have the prep side setup everything that is neededupfront, which greatly simplifies the opcode handlers.This includes adding an alloc cache for io_async_rw, to make it cheapto handle.In terms of cost, this should be basically free and transparent. Forthe worst case of {READ,WRITE}_FIXED which didn't need it before,performance is unaffected in the normal peak workload that is beingused to test that. Still runs at 122M IOPS.Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

50220d6a - io_uring/net: get rid of ->prep_async() for send side

Mon, 18 Mar 2024 14:09:47 +0000

io_uring/net: get rid of ->prep_async() for send sideMove the io_async_msghdr out of the issue path and into prep handling,e it's now done unconditionally and hence does not need to be partof the issue path. This means any usage of io_sendrecv_prep_async() andio_sendmsg_prep_async(), and hence the forced async setup path is nowunified with the normal prep setup.Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

c6f32c7d - io_uring/net: get rid of ->prep_async() for receive side

Mon, 18 Mar 2024 13:36:03 +0000

io_uring/net: get rid of ->prep_async() for receive sideMove the io_async_msghdr out of the issue path and into prep handling,since it's now done unconditionally and hence does not need to be partof the issue path. This reduces the footprint of the multishot fastpath of multiple invocations of ->issue() per prep, and also means thatusing ->prep_async() can be dropped for recvmsg asthis is now done viasetup on the prep side.Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c

4a3223f7 - io_uring/net: switch io_recv() to using io_async_msghdr

Tue, 05 Mar 2024 22:39:16 +0000

io_uring/net: switch io_recv() to using io_async_msghdrNo functional changes in this patch, just in preparation for carryingmore state than what is available now, if necessary.Signed-off-by: Jens Axboe List of files: /linux-6.15/io_uring/opdef.c