Changes in xsk_buff_pool.h

bf20af07 - xsk: Fix offset calculation in unaligned mode

Wed, 16 Apr 2025 11:29:25 +0000

xsk: Fix offset calculation in unaligned modeBring back previous offset calculation behaviourin AF_XDP unaligned umem mode.In unaligned mode, upper 16 bits should containdata offset, lower 48 bits should containonly specific chunk location without offset.Remove pool->headroom duplication into 48bit address.Signed-off-by: Eryk Kubanski Fixes: bea14124bacb ("xsk: Get rid of xdp_buff_xsk::orig_addr")Acked-by: Magnus Karlsson Link: https://patch.msgid.link/20250416112925.7501-1-e.kubanski@partner.samsung.comSigned-off-by: Jakub Kicinski List of files: /linux-6.15/include/net/xsk_buff_pool.h

a1356ac7 - xsk: Fix race condition in AF_XDP generic RX path

Wed, 16 Apr 2025 10:19:08 +0000

xsk: Fix race condition in AF_XDP generic RX pathMove rx_lock from xsk_socket to xsk_buff_pool.Fix synchronization for shared umem mode ingeneric RX path where multiple sockets sharesingle xsk_buff_pool.RX queue is exclusive to xsk_socket, while FILLqueue can be shared between multiple sockets.This could result in race condition where twoCPU cores access RX path of two different socketssharing the same umem.Protect both queues by acquiring spinlock in sharedxsk_buff_pool.Lock contention may be minimized in the future by someper-thread FQ buffering.It's safe and necessary to move spin_lock_bh(rx_lock)after xsk_rcv_check():* xs->pool and spinlock_init is synchronized by xsk_bind() -> xsk_is_bound() memory barriers.* xsk_rcv_check() may return true at the moment of xsk_release() or xsk_unbind_dev(), however this will not cause any data races or race conditions. xsk_unbind_dev() removes xdp socket from all maps and waits for completion of all outstanding rx operations. Packets in RX path will either complete safely or drop.Signed-off-by: Eryk Kubanski Fixes: bf0bdd1343efb ("xdp: fix race on generic receive path")Acked-by: Magnus Karlsson Link: https://patch.msgid.link/20250416101908.10919-1-e.kubanski@partner.samsung.comSigned-off-by: Jakub Kicinski List of files: /linux-6.15/include/net/xsk_buff_pool.h

23d9324a - xsk: add helper to get &xdp_desc's DMA and meta pointer in one go

Thu, 06 Feb 2025 18:26:29 +0000

xsk: add helper to get &xdp_desc's DMA and meta pointer in one goCurrently, when your driver supports XSk Tx metadata and you want tosend an XSk frame, you need to do the following:* call external xsk_buff_raw_get_dma();* call inline xsk_buff_get_metadata(), which calls external xsk_buff_raw_get_data() and then do some inline checks.This effectively means that the following piece:addr = pool->unaligned ? xp_unaligned_add_offset_to_addr(addr) : addr;is done twice per frame, plus you have 2 external calls per frame, plusthis: meta = pool->addrs + addr - pool->tx_metadata_len; if (unlikely(!xsk_buff_valid_tx_metadata(meta)))is always inlined, even if there's no meta or it's invalid.Add xsk_buff_raw_get_ctx() (xp_raw_get_ctx() to be precise) to do thatin one go. It returns a small structure with 2 fields: DMA address,filled unconditionally, and metadata pointer, non-NULL only if it'spresent and valid. The address correction is performed only once andyou also have only 1 external call per XSk frame, which does all thecalculations and checks outside of your hotpath. You only need tocheck `if (ctx.meta)` for the metadata presence.To not copy any existing code, derive address correction and gettingvirtual and DMA address into small helpers. bloat-o-meter reports noobject code changes for the existing functionality.Signed-off-by: Alexander Lobakin Link: https://patch.msgid.link/20250206182630.3914318-5-aleksander.lobakin@intel.comSigned-off-by: Jakub Kicinski List of files: /linux-6.15/include/net/xsk_buff_pool.h

dcf3827c - xdp, xsk: constify read-only arguments of some static inline helpers

Tue, 03 Dec 2024 17:37:26 +0000

xdp, xsk: constify read-only arguments of some static inline helpersLots of read-only helpers for &xdp_buff and &xdp_frame, such as gettingthe frame length, skb_shared_info etc., don't have their argumentsmarked with `const` for no reason. Add the missing annotations to leaveless place for mistakes and more for optimization.Reviewed-by: Toke Høiland-Jørgensen Signed-off-by: Alexander Lobakin Link: https://patch.msgid.link/20241203173733.3181246-4-aleksander.lobakin@intel.comSigned-off-by: Jakub Kicinski List of files: /linux-6.15/include/net/xsk_buff_pool.h

ca5c9494 - xsk: align &xdp_buff_xsk harder

Tue, 03 Dec 2024 17:37:24 +0000

xsk: align &xdp_buff_xsk harderAfter the series "XSk buff on a diet" by Maciej, the greatest pow-2which &xdp_buff_xsk can be divided got reduced from 16 to 8 on x86_64.Also, sizeof(xdp_buff_xsk) now is 120 bytes, which, taking the previoussentence into account, leads to that it leaves 8 bytes at the end ofcacheline, which means an array of buffs will have its elementsmessed between the cachelines chaotically.Use __aligned_largest for this struct. This alignment is usually 16bytes, which makes it fill two full cachelines and align an arraynicely. ___cacheline_aligned may be excessive here, especially onarches with 128-256 byte CLs, as well as 32-bit arches (76 -> 96bytes on MIPS32R2), while not doing better than _largest.Signed-off-by: Alexander Lobakin Reviewed-by: Toke Høiland-Jørgensen Link: https://patch.msgid.link/20241203173733.3181246-2-aleksander.lobakin@intel.comSigned-off-by: Jakub Kicinski List of files: /linux-6.15/include/net/xsk_buff_pool.h

6e126872 - xsk: Carry a copy of xdp_zc_max_segs within xsk_buff_pool

Mon, 07 Oct 2024 12:24:56 +0000

xsk: Carry a copy of xdp_zc_max_segs within xsk_buff_poolThis so we avoid dereferencing struct net_device within hot path.Signed-off-by: Maciej Fijalkowski Signed-off-by: Daniel Borkmann Acked-by: Magnus Karlsson Link: https://lore.kernel.org/bpf/20241007122458.282590-5-maciej.fijalkowski@intel.com List of files: /linux-6.15/include/net/xsk_buff_pool.h

bea14124 - xsk: Get rid of xdp_buff_xsk::orig_addr

Mon, 07 Oct 2024 12:24:55 +0000

xsk: Get rid of xdp_buff_xsk::orig_addrContinue the process of dieting xdp_buff_xsk by removing orig_addrmember. It can be calculated from xdp->data_hard_start where it waspreviously used, so it is not anything that has to be carried around instruct used widely in hot path.This has been used for initializing xdp_buff_xsk::frame_dma during poolsetup and as a shortcut in xp_get_handle() to retrieve address providedto xsk Rx queue.Signed-off-by: Maciej Fijalkowski Signed-off-by: Daniel Borkmann Acked-by: Magnus Karlsson Link: https://lore.kernel.org/bpf/20241007122458.282590-4-maciej.fijalkowski@intel.com List of files: /linux-6.15/include/net/xsk_buff_pool.h

30ec2c1b - xsk: s/free_list_node/list_node/

Mon, 07 Oct 2024 12:24:54 +0000

xsk: s/free_list_node/list_node/Now that free_list_node's purpose is two-folded, make it just a'list_node'.Signed-off-by: Maciej Fijalkowski Signed-off-by: Daniel Borkmann Acked-by: Magnus Karlsson Link: https://lore.kernel.org/bpf/20241007122458.282590-3-maciej.fijalkowski@intel.com List of files: /linux-6.15/include/net/xsk_buff_pool.h

b692bf9a - xsk: Get rid of xdp_buff_xsk::xskb_list_node

Mon, 07 Oct 2024 12:24:53 +0000

xsk: Get rid of xdp_buff_xsk::xskb_list_nodeLet's bring xdp_buff_xsk back to occupying 2 cachelines by removingxskb_list_node - for the purpose of gathering the xskb fragsfree_list_node can be used, head of the list (xsk_buff_pool::xskb_list)stays as-is, just reuse the node ptr.It is safe to do as a single xdp_buff_xsk can never reside in twopool's lists simultaneously.Signed-off-by: Maciej Fijalkowski Signed-off-by: Daniel Borkmann Acked-by: Magnus Karlsson Link: https://lore.kernel.org/bpf/20241007122458.282590-2-maciej.fijalkowski@intel.com List of files: /linux-6.15/include/net/xsk_buff_pool.h

163943ac - xsk: use generic DMA sync shortcut instead of a custom one

Tue, 07 May 2024 11:20:26 +0000

xsk: use generic DMA sync shortcut instead of a custom oneXSk infra's been using its own DMA sync shortcut to try avoidingredundant function calls. Now that there is a generic one, removethe custom implementation and rely on the generic helpers.xsk_buff_dma_sync_for_cpu() doesn't need the second argument anymore,remove it.Signed-off-by: Alexander Lobakin Signed-off-by: Christoph Hellwig List of files: /linux-6.15/include/net/xsk_buff_pool.h

b4e352ff - xsk: add functions to fill control buffer

Tue, 05 Dec 2023 21:08:36 +0000

xsk: add functions to fill control bufferCommit 94ecc5ca4dbf ("xsk: Add cb area to struct xdp_buff_xsk") has addeda buffer for custom data to xdp_buff_xsk. Particularly, this memory is usedfor data, consumed by XDP hints kfuncs. It does not always change ona per-packet basis and some parts can be set for example, at the same timeas RX queue info.Add functions to fill all cbs in xsk_buff_pool with the same metadata.Signed-off-by: Maciej Fijalkowski Signed-off-by: Larysa Zaremba Acked-by: Magnus Karlsson Link: https://lore.kernel.org/r/20231205210847.28460-8-larysa.zaremba@intel.comSigned-off-by: Alexei Starovoitov List of files: /linux-6.15/include/net/xsk_buff_pool.h

11614723 - xsk: Add option to calculate TX checksum in SW

Mon, 27 Nov 2023 19:03:14 +0000

xsk: Add option to calculate TX checksum in SWFor XDP_COPY mode, add a UMEM option XDP_UMEM_TX_SW_CSUMto call skb_checksum_help in transmit path. Might be usefulto debugging issues with real hardware. I also use this modein the selftests.Signed-off-by: Stanislav Fomichev Link: https://lore.kernel.org/r/20231127190319.1190813-9-sdf@google.comSigned-off-by: Alexei Starovoitov List of files: /linux-6.15/include/net/xsk_buff_pool.h

48eb03dd - xsk: Add TX timestamp and TX checksum offload support

Mon, 27 Nov 2023 19:03:08 +0000

xsk: Add TX timestamp and TX checksum offload supportThis change actually defines the (initial) metadata layoutthat should be used by AF_XDP userspace (xsk_tx_metadata).The first field is flags which requests appropriate offloads,followed by the offload-specific fields. The supported per-deviceoffloads are exported via netlink (new xsk-flags).The offloads themselves are still implemented in a bit of aframework-y fashion that's left from my initial kfunc attempt.I'm introducing new xsk_tx_metadata_ops which drivers aresupposed to implement. The drivers are also supposedto call xsk_tx_metadata_request/xsk_tx_metadata_complete inthe right places. Since xsk_tx_metadata_{request,_complete}are static inline, we don't incur any extra overhead doingindirect calls.The benefit of this scheme is as follows:- keeps all metadata layout parsing away from driver code- makes it easy to grep and see which drivers implement what- don't need any extra flags to maintain to keep track of what offloads are implemented; if the callback is implemented - the offload is supported (used by netlink reporting code)Two offloads are defined right now:1. XDP_TXMD_FLAGS_CHECKSUM: skb-style csum_start+csum_offset2. XDP_TXMD_FLAGS_TIMESTAMP: writes TX timestamp back into metadata area upon completion (tx_timestamp field)XDP_TXMD_FLAGS_TIMESTAMP is also implemented for XDP_COPY mode: it writesSW timestamp from the skb destructor (note I'm reusing hwtstamps to passmetadata pointer).The struct is forward-compatible and can be extended in the futureby appending more fields.Reviewed-by: Song Yoong Siang Signed-off-by: Stanislav Fomichev Acked-by: Jakub Kicinski Link: https://lore.kernel.org/r/20231127190319.1190813-3-sdf@google.comSigned-off-by: Alexei Starovoitov List of files: /linux-6.15/include/net/xsk_buff_pool.h

341ac980 - xsk: Support tx_metadata_len

Mon, 27 Nov 2023 19:03:07 +0000

xsk: Support tx_metadata_lenFor zerocopy mode, tx_desc->addr can point to an arbitrary offsetand carry some TX metadata in the headroom. For copy mode, thereis no way currently to populate skb metadata.Introduce new tx_metadata_len umem config option that indicates how manybytes to treat as metadata. Metadata bytes come prior to tx_desc address(same as in RX case).The size of the metadata has mostly the same constraints as XDP:- less than 256 bytes- 8-byte aligned (compared to 4-byte alignment on xdp, due to 8-byte timestamp in the completion)- non-zeroThis data is not interpreted in any way right now.Reviewed-by: Song Yoong Siang Signed-off-by: Stanislav Fomichev Reviewed-by: Jakub Kicinski Link: https://lore.kernel.org/r/20231127190319.1190813-2-sdf@google.comSigned-off-by: Alexei Starovoitov List of files: /linux-6.15/include/net/xsk_buff_pool.h

24ea5012 - xsk: support mbuf on ZC RX

Wed, 19 Jul 2023 13:24:08 +0000

xsk: support mbuf on ZC RXGiven that skb_shared_info relies on skb_frag_t, in order to supportxskb chaining, introduce xdp_buff_xsk::xskb_list_node andxsk_buff_pool::xskb_list.This is needed so ZC drivers can add frags as xskb nodes which will makeit possible to handle it both when producing AF_XDP Rx descriptors aswell as freeing/recycling all the frags that a single frame carries.Speaking of latter, update xsk_buff_free() to take care of list nodes.For the former (adding as frags), introduce xsk_buff_add_frag() for ZCdrivers usage that is going to be used to add a frag to xskb list frompool.xsk_buff_get_frag() will be utilized by XDP_TX and, on contrary, willreturn xdp_buff.One of the previous patches added a wrapper for ZC Rx so implement xskblist walk and production of Rx descriptors there.On bind() path, bail out if socket wants to use ZC multi-buffer butunderlying netdev does not support it.Signed-off-by: Maciej Fijalkowski Link: https://lore.kernel.org/r/20230719132421.584801-12-maciej.fijalkowski@intel.comSigned-off-by: Alexei Starovoitov List of files: /linux-6.15/include/net/xsk_buff_pool.h

1b725b0c - xsk: allow core/drivers to test EOP bit

Wed, 19 Jul 2023 13:24:04 +0000

xsk: allow core/drivers to test EOP bitDrivers are used to check for EOP bit whereas AF_XDP operates oninverted logic - user space indicates that current frag is not the lastone and packet continues. For AF_XDP core needs, add xp_mb_desc() thatwill simply test XDP_PKT_CONTD from xdp_desc::options, but in order topreserve drivers default behavior, introduce an interface for ZC driversthat will negate xp_mb_desc() result and therefore make it easier totest EOP bit from during production of HW Tx descriptors.Signed-off-by: Maciej Fijalkowski Link: https://lore.kernel.org/r/20230719132421.584801-8-maciej.fijalkowski@intel.comSigned-off-by: Alexei Starovoitov List of files: /linux-6.15/include/net/xsk_buff_pool.h

6ec7be9a - xsk: Use pool->dma_pages to check for DMA

Sun, 23 Apr 2023 18:01:56 +0000

xsk: Use pool->dma_pages to check for DMACompare pool->dma_pages instead of pool->dma_pages_cnt to check for anactive DMA mapping. pool->dma_pages needs to be read anyway to accessthe map so this compiles to more efficient code.Signed-off-by: Kal Conley Signed-off-by: Daniel Borkmann Reviewed-by: Xuan Zhuo Acked-by: Magnus Karlsson Link: https://lore.kernel.org/bpf/20230423180157.93559-1-kal.conley@dectris.com List of files: /linux-6.15/include/net/xsk_buff_pool.h

d769ccaf - xsk: Fix unaligned descriptor validation

Wed, 05 Apr 2023 23:59:18 +0000

xsk: Fix unaligned descriptor validationMake sure unaligned descriptors that straddle the end of the UMEM areconsidered invalid. Currently, descriptor validation is broken forzero-copy mode which only checks descriptors at page granularity.For example, descriptors in zero-copy mode that overrun the end of theUMEM but not a page boundary are (incorrectly) considered valid. TheUMEM boundary check needs to happen before the page boundary andcontiguity checks in xp_desc_crosses_non_contig_pg(). Do this check inxp_unaligned_validate_desc() instead like xp_check_unaligned() alreadydoes.Fixes: 2b43470add8c ("xsk: Introduce AF_XDP buffer allocation API")Signed-off-by: Kal Conley Acked-by: Magnus Karlsson Link: https://lore.kernel.org/r/20230405235920.7305-2-kal.conley@dectris.comSigned-off-by: Martin KaFai Lau List of files: /linux-6.15/include/net/xsk_buff_pool.h

94ecc5ca - xsk: Add cb area to struct xdp_buff_xsk

Thu, 19 Jan 2023 22:15:33 +0000

xsk: Add cb area to struct xdp_buff_xskAdd an area after the xdp_buff in struct xdp_buff_xsk that drivers can useto stash extra information to use in metadata kfuncs. The maximum size of24 bytes means the full xdp_buff_xsk structure will take up exactly twocache lines (with the cb field spanning both). Also add a macro drivers canuse to check their own wrapping structs against the available size.Cc: John Fastabend Cc: David Ahern Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.netCc: netdev@vger.kernel.orgSuggested-by: Jakub Kicinski Signed-off-by: Toke Høiland-Jørgensen Signed-off-by: Stanislav Fomichev Link: https://lore.kernel.org/r/20230119221536.3349901-15-sdf@google.comSigned-off-by: Martin KaFai Lau List of files: /linux-6.15/include/net/xsk_buff_pool.h

60240bc2 - xsk: Inherit need_wakeup flag for shared sockets

Wed, 21 Sep 2022 13:57:01 +0000

xsk: Inherit need_wakeup flag for shared socketsThe flag for need_wakeup is not set for xsks with `XDP_SHARED_UMEM`flag and of different queue ids and/or devices. They should inheritthe flag from the first socket buffer pool since no flags can bespecified once `XDP_SHARED_UMEM` is specified.Fixes: b5aea28dca134 ("xsk: Add shared umem support between queue ids")Signed-off-by: Jalal Mostafa Signed-off-by: Daniel Borkmann Acked-by: Magnus Karlsson Link: https://lore.kernel.org/bpf/20220921135701.10199-1-jalal.a.mostapha@gmail.com List of files: /linux-6.15/include/net/xsk_buff_pool.h