1.. SPDX-License-Identifier: GPL-2.0
2
3============
4Devlink Trap
5============
6
7Background
8==========
9
10Devices capable of offloading the kernel's datapath and perform functions such
11as bridging and routing must also be able to send specific packets to the
12kernel (i.e., the CPU) for processing.
13
14For example, a device acting as a multicast-aware bridge must be able to send
15IGMP membership reports to the kernel for processing by the bridge module.
16Without processing such packets, the bridge module could never populate its
17MDB.
18
19As another example, consider a device acting as router which has received an IP
20packet with a TTL of 1. Upon routing the packet the device must send it to the
21kernel so that it will route it as well and generate an ICMP Time Exceeded
22error datagram. Without letting the kernel route such packets itself, utilities
23such as ``traceroute`` could never work.
24
25The fundamental ability of sending certain packets to the kernel for processing
26is called "packet trapping".
27
28Overview
29========
30
31The ``devlink-trap`` mechanism allows capable device drivers to register their
32supported packet traps with ``devlink`` and report trapped packets to
33``devlink`` for further analysis.
34
35Upon receiving trapped packets, ``devlink`` will perform a per-trap packets and
36bytes accounting and potentially report the packet to user space via a netlink
37event along with all the provided metadata (e.g., trap reason, timestamp, input
38port). This is especially useful for drop traps (see :ref:`Trap-Types`)
39as it allows users to obtain further visibility into packet drops that would
40otherwise be invisible.
41
42The following diagram provides a general overview of ``devlink-trap``::
43
44                                    Netlink event: Packet w/ metadata
45                                                   Or a summary of recent drops
46                                  ^
47                                  |
48         Userspace                |
49        +---------------------------------------------------+
50         Kernel                   |
51                                  |
52                          +-------+--------+
53                          |                |
54                          |  drop_monitor  |
55                          |                |
56                          +-------^--------+
57                                  |
58                                  |
59                                  |
60                             +----+----+
61                             |         |      Kernel's Rx path
62                             | devlink |      (non-drop traps)
63                             |         |
64                             +----^----+      ^
65                                  |           |
66                                  +-----------+
67                                  |
68                          +-------+-------+
69                          |               |
70                          | Device driver |
71                          |               |
72                          +-------^-------+
73         Kernel                   |
74        +---------------------------------------------------+
75         Hardware                 |
76                                  | Trapped packet
77                                  |
78                               +--+---+
79                               |      |
80                               | ASIC |
81                               |      |
82                               +------+
83
84.. _Trap-Types:
85
86Trap Types
87==========
88
89The ``devlink-trap`` mechanism supports the following packet trap types:
90
91  * ``drop``: Trapped packets were dropped by the underlying device. Packets
92    are only processed by ``devlink`` and not injected to the kernel's Rx path.
93    The trap action (see :ref:`Trap-Actions`) can be changed.
94  * ``exception``: Trapped packets were not forwarded as intended by the
95    underlying device due to an exception (e.g., TTL error, missing neighbour
96    entry) and trapped to the control plane for resolution. Packets are
97    processed by ``devlink`` and injected to the kernel's Rx path. Changing the
98    action of such traps is not allowed, as it can easily break the control
99    plane.
100
101.. _Trap-Actions:
102
103Trap Actions
104============
105
106The ``devlink-trap`` mechanism supports the following packet trap actions:
107
108  * ``trap``: The sole copy of the packet is sent to the CPU.
109  * ``drop``: The packet is dropped by the underlying device and a copy is not
110    sent to the CPU.
111  * ``mirror``: The packet is forwarded by the underlying device and a copy is
112    sent to the CPU.
113
114Generic Packet Traps
115====================
116
117Generic packet traps are used to describe traps that trap well-defined packets
118or packets that are trapped due to well-defined conditions (e.g., TTL error).
119Such traps can be shared by multiple device drivers and their description must
120be added to the following table:
121
122.. list-table:: List of Generic Packet Traps
123   :widths: 5 5 90
124
125   * - Name
126     - Type
127     - Description
128   * - ``source_mac_is_multicast``
129     - ``drop``
130     - Traps incoming packets that the device decided to drop because of a
131       multicast source MAC
132   * - ``vlan_tag_mismatch``
133     - ``drop``
134     - Traps incoming packets that the device decided to drop in case of VLAN
135       tag mismatch: The ingress bridge port is not configured with a PVID and
136       the packet is untagged or prio-tagged
137   * - ``ingress_vlan_filter``
138     - ``drop``
139     - Traps incoming packets that the device decided to drop in case they are
140       tagged with a VLAN that is not configured on the ingress bridge port
141   * - ``ingress_spanning_tree_filter``
142     - ``drop``
143     - Traps incoming packets that the device decided to drop in case the STP
144       state of the ingress bridge port is not "forwarding"
145   * - ``port_list_is_empty``
146     - ``drop``
147     - Traps packets that the device decided to drop in case they need to be
148       flooded (e.g., unknown unicast, unregistered multicast) and there are
149       no ports the packets should be flooded to
150   * - ``port_loopback_filter``
151     - ``drop``
152     - Traps packets that the device decided to drop in case after layer 2
153       forwarding the only port from which they should be transmitted through
154       is the port from which they were received
155   * - ``blackhole_route``
156     - ``drop``
157     - Traps packets that the device decided to drop in case they hit a
158       blackhole route
159   * - ``ttl_value_is_too_small``
160     - ``exception``
161     - Traps unicast packets that should be forwarded by the device whose TTL
162       was decremented to 0 or less
163   * - ``tail_drop``
164     - ``drop``
165     - Traps packets that the device decided to drop because they could not be
166       enqueued to a transmission queue which is full
167   * - ``non_ip``
168     - ``drop``
169     - Traps packets that the device decided to drop because they need to
170       undergo a layer 3 lookup, but are not IP or MPLS packets
171   * - ``uc_dip_over_mc_dmac``
172     - ``drop``
173     - Traps packets that the device decided to drop because they need to be
174       routed and they have a unicast destination IP and a multicast destination
175       MAC
176   * - ``dip_is_loopback_address``
177     - ``drop``
178     - Traps packets that the device decided to drop because they need to be
179       routed and their destination IP is the loopback address (i.e., 127.0.0.0/8
180       and ::1/128)
181   * - ``sip_is_mc``
182     - ``drop``
183     - Traps packets that the device decided to drop because they need to be
184       routed and their source IP is multicast (i.e., 224.0.0.0/8 and ff::/8)
185   * - ``sip_is_loopback_address``
186     - ``drop``
187     - Traps packets that the device decided to drop because they need to be
188       routed and their source IP is the loopback address (i.e., 127.0.0.0/8 and ::1/128)
189   * - ``ip_header_corrupted``
190     - ``drop``
191     - Traps packets that the device decided to drop because they need to be
192       routed and their IP header is corrupted: wrong checksum, wrong IP version
193       or too short Internet Header Length (IHL)
194   * - ``ipv4_sip_is_limited_bc``
195     - ``drop``
196     - Traps packets that the device decided to drop because they need to be
197       routed and their source IP is limited broadcast (i.e., 255.255.255.255/32)
198   * - ``ipv6_mc_dip_reserved_scope``
199     - ``drop``
200     - Traps IPv6 packets that the device decided to drop because they need to
201       be routed and their IPv6 multicast destination IP has a reserved scope
202       (i.e., ffx0::/16)
203   * - ``ipv6_mc_dip_interface_local_scope``
204     - ``drop``
205     - Traps IPv6 packets that the device decided to drop because they need to
206       be routed and their IPv6 multicast destination IP has an interface-local scope
207       (i.e., ffx1::/16)
208   * - ``mtu_value_is_too_small``
209     - ``exception``
210     - Traps packets that should have been routed by the device, but were bigger
211       than the MTU of the egress interface
212   * - ``unresolved_neigh``
213     - ``exception``
214     - Traps packets that did not have a matching IP neighbour after routing
215   * - ``mc_reverse_path_forwarding``
216     - ``exception``
217     - Traps multicast IP packets that failed reverse-path forwarding (RPF)
218       check during multicast routing
219   * - ``reject_route``
220     - ``exception``
221     - Traps packets that hit reject routes (i.e., "unreachable", "prohibit")
222   * - ``ipv4_lpm_miss``
223     - ``exception``
224     - Traps unicast IPv4 packets that did not match any route
225   * - ``ipv6_lpm_miss``
226     - ``exception``
227     - Traps unicast IPv6 packets that did not match any route
228   * - ``non_routable_packet``
229     - ``drop``
230     - Traps packets that the device decided to drop because they are not
231       supposed to be routed. For example, IGMP queries can be flooded by the
232       device in layer 2 and reach the router. Such packets should not be
233       routed and instead dropped
234   * - ``decap_error``
235     - ``exception``
236     - Traps NVE and IPinIP packets that the device decided to drop because of
237       failure during decapsulation (e.g., packet being too short, reserved
238       bits set in VXLAN header)
239   * - ``overlay_smac_is_mc``
240     - ``drop``
241     - Traps NVE packets that the device decided to drop because their overlay
242       source MAC is multicast
243   * - ``ingress_flow_action_drop``
244     - ``drop``
245     - Traps packets dropped during processing of ingress flow action drop
246   * - ``egress_flow_action_drop``
247     - ``drop``
248     - Traps packets dropped during processing of egress flow action drop
249
250Driver-specific Packet Traps
251============================
252
253Device drivers can register driver-specific packet traps, but these must be
254clearly documented. Such traps can correspond to device-specific exceptions and
255help debug packet drops caused by these exceptions. The following list includes
256links to the description of driver-specific traps registered by various device
257drivers:
258
259  * :doc:`netdevsim`
260  * :doc:`mlxsw`
261
262.. _Generic-Packet-Trap-Groups:
263
264Generic Packet Trap Groups
265==========================
266
267Generic packet trap groups are used to aggregate logically related packet
268traps. These groups allow the user to batch operations such as setting the trap
269action of all member traps. In addition, ``devlink-trap`` can report aggregated
270per-group packets and bytes statistics, in case per-trap statistics are too
271narrow. The description of these groups must be added to the following table:
272
273.. list-table:: List of Generic Packet Trap Groups
274   :widths: 10 90
275
276   * - Name
277     - Description
278   * - ``l2_drops``
279     - Contains packet traps for packets that were dropped by the device during
280       layer 2 forwarding (i.e., bridge)
281   * - ``l3_drops``
282     - Contains packet traps for packets that were dropped by the device during
283       layer 3 forwarding
284   * - ``l3_exceptions``
285     - Contains packet traps for packets that hit an exception (e.g., TTL
286       error) during layer 3 forwarding
287   * - ``buffer_drops``
288     - Contains packet traps for packets that were dropped by the device due to
289       an enqueue decision
290   * - ``tunnel_drops``
291     - Contains packet traps for packets that were dropped by the device during
292       tunnel encapsulation / decapsulation
293   * - ``acl_drops``
294     - Contains packet traps for packets that were dropped by the device during
295       ACL processing
296
297Packet Trap Policers
298====================
299
300As previously explained, the underlying device can trap certain packets to the
301CPU for processing. In most cases, the underlying device is capable of handling
302packet rates that are several orders of magnitude higher compared to those that
303can be handled by the CPU.
304
305Therefore, in order to prevent the underlying device from overwhelming the CPU,
306devices usually include packet trap policers that are able to police the
307trapped packets to rates that can be handled by the CPU.
308
309The ``devlink-trap`` mechanism allows capable device drivers to register their
310supported packet trap policers with ``devlink``. The device driver can choose
311to associate these policers with supported packet trap groups (see
312:ref:`Generic-Packet-Trap-Groups`) during its initialization, thereby exposing
313its default control plane policy to user space.
314
315Device drivers should allow user space to change the parameters of the policers
316(e.g., rate, burst size) as well as the association between the policers and
317trap groups by implementing the relevant callbacks.
318
319If possible, device drivers should implement a callback that allows user space
320to retrieve the number of packets that were dropped by the policer because its
321configured policy was violated.
322
323Testing
324=======
325
326See ``tools/testing/selftests/drivers/net/netdevsim/devlink_trap.sh`` for a
327test covering the core infrastructure. Test cases should be added for any new
328functionality.
329
330Device drivers should focus their tests on device-specific functionality, such
331as the triggering of supported packet traps.
332