Upgrade ENA to v2.4.1

ena: Remove redundant declaration of ena_log_level.

GCC6 raises a -Wredundant-decl error due to duplicate declarations
in ena_fbsd_log.h and ena_plat.h.

Sponsored by: Chelsio

Upgrade ENA to v2.4.1

ena: Remove redundant declaration of ena_log_level.

GCC6 raises a -Wredundant-decl error due to duplicate declarations
in ena_fbsd_log.h and ena_plat.h.

Sponsored by: Chelsio Communications

(cherry picked from commit 8843787aa1bdbd10de6ba47a04489179ec2d2d3c)

ena: Avoid unnecessary mbuf collapses for LLQ condition

In case of Low-latency Queue, one small enough descriptor can be pushed
directly to the ENA hw, thus saving one fragment. Check for this
condition before performing collapse.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit c81f8c26115a64b9a97ecdb2a64e824dd839ee73)

ena: Trigger reset on ena_com_prepare_tx failure

All ena_com_prepare_tx errors other than ENA_COM_NO_MEM are fatal and
require device reset.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 36130d2979d695dd439bc607feb00dcdb9a1937b)

ena: Prevent reset after device destruction

Check for ENA_FLAG_TRIGGER_RESET inside a locked context in order to
avoid potential race conditions with ena_destroy_device. This aligns the
reset task logic with the Linux driver.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 433ab9b6987b42b3e5b25b8b5dc7e5178c7ef9bb)

ena: Add extra log messages

Stay aligned with the Linux driver by adding the following logs:
* inform the user about retrying queue creation
* warn on non-empty ena_tx_buffer.mbuf prior to ena_tx_map_mbuf

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 77160654a162b5faa8ad7a02e18d2bef2589f868)

ena: Add locking assertions

ENA silently assumed that ena_up, ena_down and ena_start_xmit routines
should be called within locked context. Driver's logic heavily assumes
on concurrent access to those routines, so for safety and better
documentation about this assumption, the locking assertions were added
to the above functions.

The assertion was added only for the main steps (skipping the helper
functions) which can be called from multiple places including the kernel
and the driver itself.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit cb98c439d66c303353a9f4abbbe9ddb51559c638)

ena: Move RSS logic into its own source files

Delegate RSS related functionality into separate .c/.h files in
preparation for the full RSS support.

While at it, reorder functions and remove prototypes for ones with
internal linkage.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 986e7b9227668caf9620f207e3c1d708c87b634d)

ena: Disable meta descriptor caching for netmap

If LLQ is being used, `ena_tx_ctx.meta_valid` must stay enabled. This
fixes netmap support on latest generation ENA HW and aligns it with the
core driver behavior.

As netmap doesn't support any csum offloads, the
`adapter->disable_meta_caching` value can be simply passed to the HW.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit a831466830de6ab55fc03170290b313157196e81)

ena: Share ena_global_lock between driver instances

In order to use `ena_global_lock` in sysctl context, it must be kept
outside the driver instance's software context, as sysctls can be called
before attach and after detach, leading to lock use before sx_init and
after sx_destroy otherwise.
Solve this issue by turning `ena_global_lock` into a file scope
variable, shared between all instances of the driver and associated
sysctl context, and in turn initialized/destroyed in dedicated
SYSINIT/SYSUNINIT functions.
As a side effect, this change also fixes existing race in the reset
routine, when simultaneously accessing sysctl exposed properties.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 07aff471c0de2de9a1dc5c7749c46b525bdd0201)

ena: Add missing statistics

Provide the following sysctl statistics in order to stay aligned with
the Linux driver:
* rx_ring.csum_good
* tx_ring.unmask_interrupt_num

Also rename the 'bad_csum' statistic name to 'csum_bad' for alignment.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 223c8cb12e951c63807300a0cbdc4a1569520b4b)

ena: Implement full RSS reconfiguration

Bind RX/TX queues and MSI-X vectors to matching CPUs based on the RSS
bucket entries.

Introduce sysctls for the following RSS functionality:
- rss.indir_table: indirection table mapping
- rss.indir_table_size: indirection table size
- rss.key: RSS hash key (if Toeplitz used)

Said sysctls are only available when compiled without `option RSS`, as
kernel-side RSS support currently doesn't offer RSS reconfiguration.

Migrate the hash algorithm from CRC32 to Toeplitz and change the initial
hash value to 0x0 in order to match the standard Toeplitz implementation.
Provide helpers for hash key inversion required for HW operations.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 6d1ef2abd330fac4057f092abbbdc28a568b4327)

ena: fix building in-kernel driver

When building ENA as compiled into the kernel, the driver would fail to
build. Resolve the problem by introducing the following changes:
1. Add missing `ena_rss.c` entry in `sys/conf/files`.
2. Prevent SYSCTL_ADD_INT from throwing an assert due to an extra
CTLTYPE_INT flag.

Fixes: 986e7b92276 ("ena: Move RSS logic into its own source files")
Fixes: 6d1ef2abd33 ("ena: Implement full RSS reconfiguration")

Obtained from: Semihalf
Sponsored by: Amazon, Inc.
MFC after: 1 week

(cherry picked from commit a3f0d18237bdcf272461d3b4b682de384c572144)

ena: Update driver version to v2.4.1

Some of the changes in this release:
* Hardware RSS hash key reconfiguration and indirection table
reconfiguration support.
* Full kernel RSS support.
* Extra statistic counters.
* Netmap support for ENAv3.
* Locking assertions.
* Extra log messages.
* Reset handling fixes.

Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 42c7760be3ea420668f625f2064ae347aa7e818e)

show more ...

Upgrade ENA to v2.4.0

ena: change ENA C++-style comment into C-style

According to man style(9), only C-style comments should be used.

Submitted by: Michal Krawczyk <[email protected]>
Obtained from:

Upgrade ENA to v2.4.0

ena: change ENA C++-style comment into C-style

According to man style(9), only C-style comments should be used.

Submitted by: Michal Krawczyk <[email protected]>
Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 438c9e3cf89403628ec237cfecdd0538f208087b)

ena: add support for the large LLQ headers in ENA

Default LLQ (Low-latency queue) maximum header size is 96 bytes and can
be too small for some types of packets - like IPv6 packets with multiple
extension. This can be fixed, by using large LLQ headers.

If the device supports larger LLQ headers, the user can activate this
feature by setting sysctl tunable 'hw.ena.force_large_llq_header' to '1'
in the /boot/loader.conf file.

In case the device isn't supporting this feature, the default value (96B)
will be used.

Submitted by: Michal Krawczyk <[email protected]>
Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit beaadec9eaec8e6b266faff3b0880a141728fcef)

ena: remove surplus NULL checks when freeing ENA resources

Calling free on a NULL pointer is valid, as appropriate check is already
done internally:

/* free(NULL, ...) does nothing */
if (addr == NULL)
return;

Submitted by: Artur Rojek <[email protected]>
Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit ddec69e6a796283497ebada0569f34bd41bbdf37)

ena: hide sysctl nodes for unused ENA queues

IO queue related attributes are registered statically at driver attach
with the rest of the ENA specific sysctl nodes. However, the number of
queues can be changed at runtime via the `ena_sysctl_io_queues_nb`
request, leading to a potential exposure of attributes for non-existing
queues.

Introduce a new `ena_sysctl_update_queue_node_nb` function, which
updates the sysctl nodes after the number of queues is altered.
This happens by either registering or unregistering node specific oids,
based on a delta between the previous and current queue count.

NOTE: All unregistered oids must be registered again before the driver
detach, e.g. by another call to this function.

Submitted by: Artur Rojek <[email protected]>
Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 0e7d31f63b9db869c91228d8ed1e984bdee2b931)

Merge tag 'vendor/ena-com/2.4.0'

Update the driver in order not to break its compilation
and make use of the new ENA logging system

Migrate platform code to the new logging system provided by ena_com
layer.

Make ENA_INFO the new default log level.

Remove all explicit use of `device_printf`, all new logs requiring one
of the log macros to be used.

(cherry picked from commit 3fc5d816f8831d6fc2816ac97bd78dc486cd080c)

Update ENA driver man page

Bring the obsolete man page up to date:
* update diagnostic error messages
* add documentation of loader tunables
* document netmap support
* add a driver history section
* update the contact information

Submitted by: Artur Rojek <[email protected]>
Submitted by: Michal Krawczyk <[email protected]>
Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit e34856a2c44a45512463aed0d1794f34258c66ee)

Update ENA version to v2.4.0

Some of the changes in this release:
* Large LLQ headers,
* Bug/stability fixes,
* Change of the README/Documentation.

Submitted by: Michal Krawczyk <[email protected]>
Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

(cherry picked from commit 93f0df457bf1d0e5f71839ab969c94d1f95813fb)

show more ...

Add SPDX license tag to the ENA driver files

Refering to guide: https://wiki.freebsd.org/SPDX the SPDX tag should not
replace the standard license text, however it should be added over the
standard

Add SPDX license tag to the ENA driver files

Refering to guide: https://wiki.freebsd.org/SPDX the SPDX tag should not
replace the standard license text, however it should be added over the
standard license text to make the automation easier.

Because of that, the old license was kept, but the SPDX tag was added
on top of every ENA driver file.

Submited by: Michal Krawczyk <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27117

show more ...

Update ENA driver version to v2.2.0

Driver version upgrade is connected with support for the new device
fetures, like Tx drops reporting or disabling meta caching.

Moreover, the driver configuratio

Update ENA driver version to v2.2.0

Driver version upgrade is connected with support for the new device
fetures, like Tx drops reporting or disabling meta caching.

Moreover, the driver configuration from the sysctl was reworked to
provide safer and better flow for configuring:
* number of IO queues (new feature),
* drbr size on Tx,
* Rx queue size.

Moreover, a lot of minor bug fixes and improvements were added.

Copyright date in the license of the modified files in this release was
updated to 2020.

Submitted by: Michal Krawczyk <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

show more ...

Create ENA IO queues with optional backoff

If requested size of IO queues is not supported try to decrease it until
finding the highest value that can be satisfied.

Submitted by: Maciej Bielski <m

Create ENA IO queues with optional backoff

If requested size of IO queues is not supported try to decrease it until
finding the highest value that can be satisfied.

Submitted by: Maciej Bielski <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

show more ...

Rework ENA Rx queue size configuration

This patch reworks how the Rx queue size is being reconfigured and how
the information from the device is being processed.

Reconfiguration of the queues and r

Rework ENA Rx queue size configuration

This patch reworks how the Rx queue size is being reconfigured and how
the information from the device is being processed.

Reconfiguration of the queues and reset of the device in order to make
the changes alive isn't the best approach. It can be done synchronously
and it will let to pass information if the reconfiguration was
successful to the user. It now is done in the ena_update_queue_size()
function.

To avoid reallocation of the ring buffer, statistic counters and the
reinitialization of the mutexes when only new size has to be assigned,
the io queues initialization function has been split into 2 stages:
basic, which is just copying appropriate fields and the advanced, which
allocates and inits more advanced structures for the IO rings.

Moreover, now the max allowed Rx and Tx ring size is being kept
statically in the adapter and the size of the variables holding those
values has been changed to uint32_t everywhere.

Information about IO queues size is now being logged in the up routine
instead of the attach.

Submitted by: Michal Krawczyk <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

show more ...

Use single global lock in the ENA driver

Currently, the driver had 2 global locks - one was sx lock used for
up/down synchronization and the second one was mutex, which was used
for link configurati

Use single global lock in the ENA driver

Currently, the driver had 2 global locks - one was sx lock used for
up/down synchronization and the second one was mutex, which was used
for link configuration and timer service callout.

It is better to have single lock for that. We cannot use mutex, as it
can sleep and cause witness errors in up/down configuration, so sx lock
seems to be the only choice.

Callout cannot use sx lock, but the timer service is MP safe, so we just
need to avoid race between ena_down() and ena_detach(). It can be
avoided by acquiring sx lock.

Simple macros were added that are encapsulating implementation of the
lock and makes the code cleaner.

Submitted by: Michal Krawczyk <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

show more ...

Add trigger reset function in the ENA driver

As the reset triggering is no longer a simple macro that was just
setting appropriate flag, the new function for triggering reset was
added. It improves

Add trigger reset function in the ENA driver

As the reset triggering is no longer a simple macro that was just
setting appropriate flag, the new function for triggering reset was
added. It improves code readability a lot, as we are avoiding additional
indentation.

Submitted by: Michal Krawczyk <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

show more ...

Add support for ENA NETMAP partial initialization

In NETMAP mode not all queues need to be allocated to NETMAP. Some of
them could be left to the kernel. Configuration is managed by the flags
nr_mod

Add support for ENA NETMAP partial initialization

In NETMAP mode not all queues need to be allocated to NETMAP. Some of
them could be left to the kernel. Configuration is managed by the flags
nr_mode and nr_pending_mode provided per each NETMAP kring.

ENA driver checks those flags and perform proper rings initialization.

Differential Revision: https://reviews.freebsd.org/D21937
Submitted by: Rafal Kozik <[email protected]>
Michal Krawczyk <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

show more ...

Add support for ENA NETMAP Tx

Two new tables are added to ena_tx_buffer structure:
* netmap_map_seg stores DMA mapping structures,
* netmap_buf_idx stores buff indexes taken from the slots.

When Tx

Add support for ENA NETMAP Tx

Two new tables are added to ena_tx_buffer structure:
* netmap_map_seg stores DMA mapping structures,
* netmap_buf_idx stores buff indexes taken from the slots.

When Tx resources are being set, the new mapping structures are created
and netmap Tx rings are being reset.

When Tx resources are being released, used netmap bufs are unmapped from
DMA and then mapping structures are destroyed.

When Tx interrupt occurrs, ena_netmap_tx_irq is called.

ena_netmap_txsync callback signalizes that there are new packets which
should be transmitted.
First, it fills ena_netmap_ctx. Then it performs two actions:
* ena_netmap_tx_frames moves packets from netmap ring to NIC,
* ena_netmap_tx_cleanup restores buffers from NIC and gives them back
to the userspace app.
0 is returned in case of Tx error that could be handled by the driver.

ena_netmap_tx_frames checks if there are packets ready for transmission.
Then, for each of them, ena_netmap_tx_frame is called. If error occurs,
transmitting is stopped, but if the error was cause due to HW ring being
full, information about that is not propagated to the userspace app.
When all packets are ready, doorbell is written to NIC and netmap ring
state is updated.

Parsing of one packet is done by the ena_netmap_tx_frame function.
First, it checks if number of slots does not exceed NIC limit. Invalid
packets are being dropped and the error is propagated to the upper
layer. As each netmap buffer has equal size, which is typically greater
then 2KiB, there shouldn't be any packets which contain too many slots.
Then, the ena_com_tx_ctx structure is being filled. As netmap does not
support any hardware offloads, ena_com_tx_meta structure is set to zero.
After that, ena_netmap_map_slots maps all memory slots for DMA.
If the device works in the LLQ mode, the push header is being determined
by checking if the header fits within the first socket.
If so, the portion of data is being copied directly from the slot.
In other case, the data is copied to the intermediate buffer.
First slots are treated the same as as the others, because DMA mapping
has no impact on LLQ mode. Index of each netmap buffer is taken from
slot and stored in netmap_buf_idx array. In case of mapping error,
memory is unmapped and packets are put back to the netmap ring.

ena_netmap_tx_cleanup performs out of order cleanup of sent buffers.
First, req_id is taken and is validated. As validate_tx_req_id from
ena.c is specific to kernels mbuf, another implementation is provided.
Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers
are being unmaped from DMA and put back to netmap ring. In the end,
state of netmap and NIC rings are being updated.

Differential Revision: https://reviews.freebsd.org/D21936
Submitted by: Rafal Kozik <[email protected]>
Michal Krawczyk <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

show more ...

Add support for ENA NETMAP Rx

Most of code used for Rx ring initialization could be reused in NETMAP.
Reset of NETMAP ring and new alloc method was added. Driver decides if
use kernels mbufs or NETM

Add support for ENA NETMAP Rx

Most of code used for Rx ring initialization could be reused in NETMAP.
Reset of NETMAP ring and new alloc method was added. Driver decides if
use kernels mbufs or NETMAPs slots based on IFCAP_NETMAP flag. It
allows to reuse ena_refill_rx_bufs, which provides proper handling of
Rx out of order completion.

ena_netmap_alloc_rx_slot takes exactly the same arguments as
ena_alloc_rx_mbuf, but instead of allocating one mbuf it takes one slot
from NETMAP ring. Based on queue id proper netmap_ring is found. As
NETMAP provides the "partial opening" feature not all of the rings are
avaiable. Not used points to invalid ring. If there is available slot,
it is taken from the ring. Its buffer is mapped to DMA and its index is
stored in ena_rx_buffer field in ena_rx_buffer structure. Then ena_buf
is filled with addresses and ring state is updated.

Cleanup is handled by ena_netmap_free_rx_slot. It unmaps DMA and returns
buffer to ring. As we could not return more bufs than we have taken and
we should not override occupied slots, buf_index should be 0. It is
being checked by assertion.

ena_netmap_rxsync callback puts received packets back to NETMAP ring and
passes them to user space by updating ring pointers. First it fills
ena_netmap_ctx.
Then it performs two actions:
* ena_netmap_rx_frames moves received frames from NIC to NETMAP ring,
* ena_netmap_rx_cleanup fills NIC ring with slots released by userspace
app.

In case of Rx error that could be handled by NIC driver (for example by
performing reset) rx sync should return 0.

ena_netmap_rx_frames first checks if NETMAP ring is in consistent
state and then in the loop receives new frames. When all available
frames are taken nr_hwtail is updated.

Receiving one frame is handled by ena_netmap_rx_frame. If no error
occurrs, each Descriptor is loaded by ena_netmap_rx_load_desc function.
If packets take more than one segments NS_MOREFRAG flag must be set in
all, but not last slot. In case of wrong req_id packet is removed from
NETMAP ring. If packet is successful received counters are updated.

Refiling of NIC ring is performed by ena_netmap_rx_cleanup function.
It calculates number of available slots and call ena_refill_rx_bufs with
proper number.

Differential Revision: https://reviews.freebsd.org/D21935
Submitted by: Rafal Kozik <[email protected]>
Michal Krawczyk <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

show more ...

Introduce NETMAP support in ENA

Mock implementation of NETMAP routines is located in ena_netmap.c/.h
files. All code is protected under the DEV_NETMAP macro. Makefile was
updated with files and flag

Introduce NETMAP support in ENA

Mock implementation of NETMAP routines is located in ena_netmap.c/.h
files. All code is protected under the DEV_NETMAP macro. Makefile was
updated with files and flag.

As ENA driver provide own implementations of (un)likely it must be
undefined before including NETMAP headers.

ena_netmap_attach function is called on the end of NIC attach. It fills
structure with NIC configuration and callbacks. Then provides it to
netmap_attach. Similarly netmap_detach is called during ena_detach.

Three callbacks are used.
nm_register is implemented by ena_netmap_reg. It is called when user
space application open or close NIC in NETMAP mode. Current action is
recognized based on onoff parameter: true means on and false off. As
NICs rings need to be reconfigured ena_down and ena_up are reused.
When user space application wants to receive new packets from NIC
nm_rxsync is called, and when there are new packets ready for Tx
nm_txsync is called.

Differential Revision: https://reviews.freebsd.org/D21934
Submitted by: Rafal Kozik <[email protected]>
Michal Krawczyk <[email protected]>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

show more ...