cxgbe/t4_tom: Implement uld_stop and uld_restart for ULD_TOM.

This allows the adapter to be suspended or reset even when stateful TOE is
active, in some limited configurations.

The LLD has already

cxgbe/t4_tom: Implement uld_stop and uld_restart for ULD_TOM.

This allows the adapter to be suspended or reset even when stateful TOE is
active, in some limited configurations.

The LLD has already stopped the adapter hardware and all its queues by the time
these ULD routines get called. The general approach in t4_tom is to purge the
lookup tables immediately so that they are ready for operation by the time the
adapter resumes, and park all the resources left hanging by the stopped hardware
into separate "stranded" queues that can be dealt with at leisure.

Outstanding active opens, live connections, and synq entries (for connections in
the middle of the 3-way handshake) are all treated as if the hardware had
reported an abrupt error for the tid. The servers/listeners are a bit different
in that no error is reported. They're just noted as non-functional when the
hardware stops and are recreated by the driver during restart.

Sponsored by: Chelsio Communications

(cherry picked from commit c1c524852f625cf5f420653f7850d1fe3ff6b4ca)

show more ...

cxgbe(4): Stop and restart the atid allocator with the LLD.

atids are used by both filters and TOE and the atid table is in the base
driver (LLD). New atids cannot be allocated when the allocator i

cxgbe(4): Stop and restart the atid allocator with the LLD.

atids are used by both filters and TOE and the atid table is in the base
driver (LLD). New atids cannot be allocated when the allocator is
stopped but existing ones can still be freed. It is expected that the
owners of outstanding atids will release them in their own stop
processing, before the adapter is restarted.

Sponsored by: Chelsio Communications

(cherry picked from commit fd3aca5b41968421f243b760ac5733f39f25cc56)

show more ...

cxgbe(4): Basic infrastructure for ULDs to participate in adapter reset.

The suspend/resume/reset implementation in the base driver (LLD)
currently works when only stateless features are in use. Th

cxgbe(4): Basic infrastructure for ULDs to participate in adapter reset.

The suspend/resume/reset implementation in the base driver (LLD)
currently works when only stateless features are in use. This commit
adds basic infrastructure for stateful upper layer drivers (ULDs) to
participate in suspend/resume/reset.

* Add a uld_restart to indicate that the adapter has been restarted
after a stop and the ULD should resume operations.
* Move the existing functionality in t4_suspend/t4_resume to stop_lld and
restart_lld. Use these and the new uld restart routines everywhere the
adapter has to be stopped abruptly and restarted, namely:
1. PCIE bus suspend/resume/reset methods invoked by the kernel.
2. Manual internal-reset using driver sysctl.
3. Automatic internal-reset on a fatal error.
* Implement an alternate internal-reset for use in VMs and for testing.

Typical reset sequence is:
stop_adapter(sc);
stop_lld(sc);
stop_all_uld(sc);
set_adapter_hwstatus(sc, false);

/* hw reset takes place here. */

restart_adapter(sc);
restart_lld(sc);
set_adapter_hwstatus(sc, true);
restart_all_uld(sc);

Sponsored by: Chelsio Communications

(cherry picked from commit 5241b210a4e1029f3005faf82cd74965645c401b)

show more ...

cxgbe(4): Changes to ULD list management.

* Convert t4_uld_list to an array. There will be at most 3 items in the
list and it's simpler to track them in an array with a fixed slot for
each ULD.

cxgbe(4): Changes to ULD list management.

* Convert t4_uld_list to an array. There will be at most 3 items in the
list and it's simpler to track them in an array with a fixed slot for
each ULD.
* There is no need to refcount ULDs so stop doing that.
* Add uld_ prefix to all members of uld_info.
* Rename async_event to uld_stop to match its actual purpose. Call it
for all ULDs and not just ULD_IWARP.

Reviewed by: jhb
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D46029

(cherry picked from commit cf5e6370f15cffabbbf508083ba7d48ec8abfa79)

show more ...

cxgbe(4): Reword the comment explaining the atid/cookie split.

Avoid a magic constant while here. No functional change intended.

Sponsored by: Chelsio Communications

(cherry picked from commit 43

cxgbe(4): Reword the comment explaining the atid/cookie split.

Avoid a magic constant while here. No functional change intended.

Sponsored by: Chelsio Communications

(cherry picked from commit 43f6f08488046788b0ad66e9a5119f36e5de71ab)

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

cxgbe: Enable TOE TLS RX when an RX key is provided via setsockopt().

Rather than requiring a socket to be created as a TLS socket from the
get go, switch a TOE socket from "plain" TOE to TLS mode w

cxgbe: Enable TOE TLS RX when an RX key is provided via setsockopt().

Rather than requiring a socket to be created as a TLS socket from the
get go, switch a TOE socket from "plain" TOE to TLS mode when a
receive key is added to the socket.

The firmware is only able to switch a "plain" TOE connection to TLS
mode if the head of the pending socket data is the start of a TLS
record, so the connection is migrated to TLS mode as a multi-step
process.

When TOE TLS RX is enabled, the associated connection's receive side
is frozen via a flag in the TCB. The state of the socket buffer is
then examined to determine if the pending data in the socket buffer
ends on a TLS record boundary. If so, the connection is migrated to
TLS mode and unfrozen. Otherwise, the connection is unfrozen
temporarily until more data arrives. Once more data arrives, the
receive queue is frozen again and rechecked. This continues until the
connection is paused at a record boundary. Any records received
before TLS mode is enabled are decrypted as software records.

Note that this removes the 'rx_tls_ports' sysctl. TOE TLS offload for
receive is now enabled automatically on existing TOE connections when
using a KTLS-aware SSL library just as it was previously enabled
automatically for TLS transmit. This also enables TLS offload for TOE
connections which enable TLS after passing initial data in the clear
(e.g. STARTTLS with SMTP).

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37351

show more ...

cxgbei: Support for ISO (iSCSI segmentation offload).

ISO can be disabled before establishing a connection by setting
dev.tNnex.N.toe.iso to 0.

Sponsored by: Chelsio Communications
Differential Rev

cxgbei: Support for ISO (iSCSI segmentation offload).

ISO can be disabled before establishing a connection by setting
dev.tNnex.N.toe.iso to 0.

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D31223

show more ...

cxgbe/t4_tom: Implement tod_pmtu_update.

tod_pmtu_update was added to the kernel in 01d74fe1ffc.

Sponsored by: Chelsio Communications

cxgbe: Add a struct sge_ofld_txq type.

This type mirrors struct sge_ofld_rxq and holds state for TCP offload
transmit queues. Currently it only holds a work queue but will
include additional state

cxgbe: Add a struct sge_ofld_txq type.

This type mirrors struct sge_ofld_rxq and holds state for TCP offload
transmit queues. Currently it only holds a work queue but will
include additional state in future changes.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D29382

show more ...

Clear TLS offload mode if a TLS socket hangs without receiving data.

By default, if a TOE TLS socket stops receiving data for more than 5
seconds, revert the connection back to plain TOE mode. This

Clear TLS offload mode if a TLS socket hangs without receiving data.

By default, if a TOE TLS socket stops receiving data for more than 5
seconds, revert the connection back to plain TOE mode. This provides
a fallback if the userland SSL library does not support KTLS. In
addition, for client TLS 1.3 sockets using connect(), the TOE socket
blocks before the handshake has completed since the socket option is
only invoked for the final handshake.

The timeout defaults to 5 seconds, but can be changed at boot via the
hw.cxgbe.toe.tls_rx_timeout tunable or for an individual interface via
the dev.<nexus>.toe.tls_rx_timeout sysctl.

Reviewed by: np
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27470

show more ...

Store the send tag type in the common send tag header.

Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with
their own identical headers that stored the type that the
type-specific tag

Store the send tag type in the common send tag header.

Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with
their own identical headers that stored the type that the
type-specific tag structures inherited from, so in practice it seems
drivers need this in the tag anyway. This permits removing these
extra header indirections (struct cxgbe_snd_tag and struct
mlx5e_snd_tag).

In addition, this permits driver-independent code to query the type of
a tag, e.g. to know what type of tag is being queried via
if_snd_query.

Reviewed by: gallatin, hselasky, np, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D26689

show more ...

cxgbe/iw_cxgbe: Add an async callback to notify iw_cxgbe in case of a
fatal error.

Submitted by: Krishnamraju Eraparaju @ Chelsio
MFC after: 2 weeks
Sponsored by: Chelsio Communications

NIC KTLS for Chelsio T6 adapters.

This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.

NIC KTLS on T6 is n

NIC KTLS for Chelsio T6 adapters.

This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.

NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:

1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.

2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.

3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.

Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.

TCP packets for TLS requests are broken down into the following
classes (with associated counters):

- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).

- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.

- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.

In addition, other per-txq sysctls are provided:

- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.

- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.

- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.

- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.

- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.

- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.

To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:

hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962

show more ...

Split Chelsio send tags into a generic base tag and a ratelimit tag.

NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a
distinct tag from a ratelimit tag. To support this, refactor
cxg

Split Chelsio send tags into a generic base tag and a ratelimit tag.

NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a
distinct tag from a ratelimit tag. To support this, refactor
cxgbe_snd_tag to be a simple send tag with a type and convert the
existing ratelimit tag to a new cxgbe_rate_tag structure.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D22072

show more ...

cxgbe/t4_tom: Catch up with r344433, which removed tcb_autorcvbuf_inc.

The declaration in tcp_var.h is still around so t4_tom continued to
compile but wouldn't load. A separate commit will fix tcp_

cxgbe/t4_tom: Catch up with r344433, which removed tcb_autorcvbuf_inc.

The declaration in tcp_var.h is still around so t4_tom continued to
compile but wouldn't load. A separate commit will fix tcp_var.h

Reported By: Dustin Marquess (dmarquess at gmail)

Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Use two hashes instead of a table to keep track of
hashfilters. Two because the driver needs to look up a hashfilter by
its 4-tuple or tid.

A couple of fixes while here:
- Reject attempts

cxgbe(4): Use two hashes instead of a table to keep track of
hashfilters. Two because the driver needs to look up a hashfilter by
its 4-tuple or tid.

A couple of fixes while here:
- Reject attempts to add duplicate hashfilters.
- Do not assume that any part of the 4-tuple that isn't specified is 0.
This makes it consistent with all other mandatory parameters that
already require explicit user input.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Add support for high priority filters on T6+. They have their
own region in the TCAM starting with T6, unlike previous chips where
they were in the same region as normal filters.

These fi

cxgbe(4): Add support for high priority filters on T6+. They have their
own region in the TCAM starting with T6, unlike previous chips where
they were in the same region as normal filters.

These filters "hit" before anything else in the LE's lookup. The exact
order is:
a) High priority filters
b) TOE's active region (TCAM and/or hash)
c) Servers (TOE hw listeners)
d) Normal filters

MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Improvements in TID management.

- Ignore any type of TID where the start/end values are not in the
correct order. There are situations where the firmware isn't able to
reserve room fo

cxgbe(4): Improvements in TID management.

- Ignore any type of TID where the start/end values are not in the
correct order. There are situations where the firmware isn't able to
reserve room for the number requested in the config file but doesn't
report a failure during configuration and instead sets end <= start.

- Track start/end in tid_tab and remove some redundant copies from
adapter->params.

- Move all the start/end and other read-only parameters to a quiet part
of tid_tab, away from the tid locks.

MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Data path for rate-limited tx.

This is hardware support for the SO_MAX_PACING_RATE sockopt (see
setsockopt(2)), which is available in kernels built with "options
RATELIMIT".

Relnotes: Yes

cxgbe(4): Data path for rate-limited tx.

This is hardware support for the SO_MAX_PACING_RATE sockopt (see
setsockopt(2)), which is available in kernels built with "options
RATELIMIT".

Relnotes: Yes
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Implement ifnet callbacks that deal with send tags.

An etid (ethoffload tid) is allocated for a send tag and it acquires a
reference on the traffic class that matches the send parameters
a

cxgbe(4): Implement ifnet callbacks that deal with send tags.

An etid (ethoffload tid) is allocated for a send tag and it acquires a
reference on the traffic class that matches the send parameters
associated with the tag.

Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Add support for hash filters.

These filters reside in the card's memory instead of its TCAM and can be
configured via a new "hashfilter" subcommand in cxgbetool. Hash and
normal TCAM filt

cxgbe(4): Add support for hash filters.

These filters reside in the card's memory instead of its TCAM and can be
configured via a new "hashfilter" subcommand in cxgbetool. Hash and
normal TCAM filters can be used together. The hardware does an
exact-match of packet fields for hash filters, unlike the masked match
performed for TCAM filters. Any T5/T6 card with memory can support at
least half a million hash filters. The sample config file with the
driver configures 512K of these, it is possible to double this to 1
million+ in some cases.

The chip does an exact-match of fields of incoming datagrams with hash
filters and performs the action configured for the filter if it matches.
The fields to match are specified in a "filter mask" in the firmware
config file. The filter mask always includes the 5-tuple (sip, dip,
sport, dport, ipproto). It can, optionally, also include any subset of
the filter mode (see filterMode and filterMask in the firmware config
file).

For example:
filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe
filterMask = protocol, port, vlan

Exact values of the 5-tuple, the physical port, and VLAN tag would have
to be provided while setting up a hash filter with the chip
configuration above.

Hash filters support all actions supported by TCAM filters. A packet
that hits a hash filter can be dropped, let through (with optional
steering to a specific queue or RSS region), switched out of another
port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed.
(Support for some of these will show up in the driver in a follow-up
commit very shortly).

Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Convert ACT_OPEN_RPL to a shared CPL.

Reserve 3b in the 14b atid to identify the owner and use it to dispatch
the CPL. This allows all CPLs that use an atid to be used as shared
CPLs, alt

cxgbe(4): Convert ACT_OPEN_RPL to a shared CPL.

Reserve 3b in the 14b atid to identify the owner and use it to dispatch
the CPL. This allows all CPLs that use an atid to be used as shared
CPLs, although ACT_OPEN_RPL is the only one being converted in this
revision.

Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Add support for Connection Offload Policy (aka COP).

COP allows fine-grained control on whether to offload a TCP connection
using t4_tom, and what settings to apply to a connection selecte

cxgbe(4): Add support for Connection Offload Policy (aka COP).

COP allows fine-grained control on whether to offload a TCP connection
using t4_tom, and what settings to apply to a connection selected for
offload. t4_tom must still be loaded and IFCAP_TOE must still be
enabled for full TCP offload to take place on an interface. The
difference is that IFCAP_TOE used to be the only knob and would enable
TOE for all new connections on the inteface, but now the driver will
also consult the COP, if any, before offloading to the hardware TOE.

A policy is a plain text file with any number of rules, one per line.
Each rule has a "match" part consisting of a socket-type (L = listen,
A = active open, P = passive open, D = don't care) and a pcap-filter(7)
expression, and a "settings" part that specifies whether to offload the
connection or not and the parameters to use if so. The general format
of a rule is: [socket-type] expr => settings

Example. See cxgbetool(8) for more information.
[L] ip && port http => offload
[L] port 443 => !offload
[L] port ssh => offload
[P] src net 192.168/16 && dst port ssh => offload !nagle !timestamp cong newreno
[P] dst port ssh => offload !nagle ecn cong tahoe
[P] dst port http => offload
[A] dst port 443 => offload tls
[A] dst net 192.168/16 => offload !timestamp cong highspeed

The driver processes the rules for each new listen, active open, or
passive open and stops at the first match. There is an implicit rule at
the end of every policy that prohibits offload when no rule in the
policy matches:
[D] all => !offload

This is a reworked and expanded version of a patch submitted by
Krishnamraju Eraparaju @ Chelsio.

Sponsored by: Chelsio Communications

show more ...