1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2010-2014 Intel Corporation. 3 4Kernel NIC Interface Sample Application 5======================================= 6 7The Kernel NIC Interface (KNI) is a DPDK control plane solution that 8allows userspace applications to exchange packets with the kernel networking stack. 9To accomplish this, DPDK userspace applications use an IOCTL call 10to request the creation of a KNI virtual device in the Linux* kernel. 11The IOCTL call provides interface information and the DPDK's physical address space, 12which is re-mapped into the kernel address space by the KNI kernel loadable module 13that saves the information to a virtual device context. 14The DPDK creates FIFO queues for packet ingress and egress 15to the kernel module for each device allocated. 16 17The KNI kernel loadable module is a standard net driver, 18which upon receiving the IOCTL call access the DPDK's FIFO queue to 19receive/transmit packets from/to the DPDK userspace application. 20The FIFO queues contain pointers to data packets in the DPDK. This: 21 22* Provides a faster mechanism to interface with the kernel net stack and eliminates system calls 23 24* Facilitates the DPDK using standard Linux* userspace net tools (tshark, rsync, and so on) 25 26* Eliminate the copy_to_user and copy_from_user operations on packets. 27 28The Kernel NIC Interface sample application is a simple example that demonstrates the use 29of the DPDK to create a path for packets to go through the Linux* kernel. 30This is done by creating one or more kernel net devices for each of the DPDK ports. 31The application allows the use of standard Linux tools (ethtool, iproute, tshark) with the DPDK ports and 32also the exchange of packets between the DPDK application and the Linux* kernel. 33 34The Kernel NIC Interface sample application requires that the 35KNI kernel module ``rte_kni`` be loaded into the kernel. See 36:doc:`../prog_guide/kernel_nic_interface` for more information on loading 37the ``rte_kni`` kernel module. 38 39Overview 40-------- 41 42The Kernel NIC Interface sample application ``kni`` allocates one or more 43KNI interfaces for each physical NIC port. For each physical NIC port, 44``kni`` uses two DPDK threads in user space; one thread reads from the port and 45writes to the corresponding KNI interfaces and the other thread reads from 46the KNI interfaces and writes the data unmodified to the physical NIC port. 47 48It is recommended to configure one KNI interface for each physical NIC port. 49The application can be configured with more than one KNI interface for 50each physical NIC port for performance testing or it can work together with 51VMDq support in future. 52 53The packet flow through the Kernel NIC Interface application is as shown 54in the following figure. 55 56.. _figure_kernel_nic: 57 58.. figure:: img/kernel_nic.* 59 60 Kernel NIC Application Packet Flow 61 62If link monitoring is enabled with the ``-m`` command line flag, one 63additional pthread is launched which will check the link status of each 64physical NIC port and will update the carrier status of the corresponding 65KNI interface(s) to match the physical NIC port's state. This means that 66the KNI interface(s) will be disabled automatically when the Ethernet link 67goes down and enabled when the Ethernet link goes up. 68 69If link monitoring is enabled, the ``rte_kni`` kernel module should be loaded 70such that the :ref:`default carrier state <kni_default_carrier_state>` is 71set to *off*. This ensures that the KNI interface is only enabled *after* 72the Ethernet link of the corresponding NIC port has reached the linkup state. 73 74If link monitoring is not enabled, the ``rte_kni`` kernel module should be 75loaded with the :ref:`default carrier state <kni_default_carrier_state>` 76set to *on*. This sets the carrier state of the KNI interfaces to *on* 77when the KNI interfaces are enabled without regard to the actual link state 78of the corresponding NIC port. This is useful for testing in loopback 79mode where the NIC port may not be physically connected to anything. 80 81Compiling the Application 82------------------------- 83 84To compile the sample application see :doc:`compiling`. 85 86The application is located in the ``examples/kni`` sub-directory. 87 88.. note:: 89 90 This application is intended as a linux only. 91 92Running the kni Example Application 93----------------------------------- 94 95The ``kni`` example application requires a number of command line options: 96 97.. code-block:: console 98 99 dpdk-kni [EAL options] -- -p PORTMASK --config="(port,lcore_rx,lcore_tx[,lcore_kthread,...])[,(port,lcore_rx,lcore_tx[,lcore_kthread,...])]" [-P] [-m] 100 101Where: 102 103* ``-p PORTMASK``: 104 105 Hexadecimal bitmask of ports to configure. 106 107* ``--config="(port,lcore_rx,lcore_tx[,lcore_kthread,...])[,(port,lcore_rx,lcore_tx[,lcore_kthread,...])]"``: 108 109 Determines which lcores the Rx and Tx DPDK tasks, and (optionally) 110 the KNI kernel thread(s) are bound to for each physical port. 111 112* ``-P``: 113 114 Optional flag to set all ports to promiscuous mode so that packets are 115 accepted regardless of the packet's Ethernet MAC destination address. 116 Without this option, only packets with the Ethernet MAC destination 117 address set to the Ethernet address of the port are accepted. 118 119* ``-m``: 120 121 Optional flag to enable monitoring and updating of the Ethernet 122 carrier state. With this option set, a thread will be started which 123 will periodically check the Ethernet link status of the physical 124 Ethernet ports and set the carrier state of the corresponding KNI 125 network interface to match it. This means that the KNI interface will 126 be disabled automatically when the Ethernet link goes down and enabled 127 when the Ethernet link goes up. 128 129Refer to *DPDK Getting Started Guide* for general information on running 130applications and the Environment Abstraction Layer (EAL) options. 131 132The ``-c coremask`` or ``-l corelist`` parameter of the EAL options must 133include the lcores specified by ``lcore_rx`` and ``lcore_tx`` for each port, 134but does not need to include lcores specified by ``lcore_kthread`` as those 135cores are used to pin the kernel threads in the ``rte_kni`` kernel module. 136 137The ``--config`` parameter must include a set of 138``(port,lcore_rx,lcore_tx,[lcore_kthread,...])`` values for each physical 139port specified in the ``-p PORTMASK`` parameter. 140 141The optional ``lcore_kthread`` lcore ID parameter in ``--config`` can be 142specified zero, one or more times for each physical port. 143 144If no lcore ID is specified for ``lcore_kthread``, one KNI interface will 145be created for the physical port ``port`` and the KNI kernel thread(s) 146will have no specific core affinity. 147 148If one or more lcore IDs are specified for ``lcore_kthread``, a KNI interface 149will be created for each lcore ID specified, bound to the physical port 150``port``. If the ``rte_kni`` kernel module is loaded in :ref:`multiple 151kernel thread <kni_kernel_thread_mode>` mode, a kernel thread will be created 152for each KNI interface and bound to the specified core. If the ``rte_kni`` 153kernel module is loaded in :ref:`single kernel thread <kni_kernel_thread_mode>` 154mode, only one kernel thread is started for all KNI interfaces. The kernel 155thread will be bound to the first ``lcore_kthread`` lcore ID specified. 156 157Example Configurations 158~~~~~~~~~~~~~~~~~~~~~~~ 159 160The following commands will first load the ``rte_kni`` kernel module in 161:ref:`multiple kernel thread <kni_kernel_thread_mode>` mode. The ``kni`` 162application is then started using two ports; Port 0 uses lcore 4 for the 163Rx task, lcore 6 for the Tx task, and will create a single KNI interface 164``vEth0_0`` with the kernel thread bound to lcore 8. Port 1 uses lcore 1655 for the Rx task, lcore 7 for the Tx task, and will create a single KNI 166interface ``vEth1_0`` with the kernel thread bound to lcore 9. 167 168.. code-block:: console 169 170 # rmmod rte_kni 171 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko kthread_mode=multiple 172 # ./<build-dir>/examples/dpdk-kni -l 4-7 -n 4 -- -P -p 0x3 -m --config="(0,4,6,8),(1,5,7,9)" 173 174The following example is identical, except an additional ``lcore_kthread`` 175core is specified per physical port. In this case, ``kni`` will create 176four KNI interfaces: ``vEth0_0``/``vEth0_1`` bound to physical port 0 and 177``vEth1_0``/``vEth1_1`` bound to physical port 1. 178 179The kernel thread for each interface will be bound as follows: 180 181 * ``vEth0_0`` - bound to lcore 8. 182 * ``vEth0_1`` - bound to lcore 10. 183 * ``vEth1_0`` - bound to lcore 9. 184 * ``vEth1_1`` - bound to lcore 11 185 186.. code-block:: console 187 188 # rmmod rte_kni 189 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko kthread_mode=multiple 190 # ./<build-dir>/examples/dpdk-kni -l 4-7 -n 4 -- -P -p 0x3 -m --config="(0,4,6,8,10),(1,5,7,9,11)" 191 192The following example can be used to test the interface between the ``kni`` 193test application and the ``rte_kni`` kernel module. In this example, 194the ``rte_kni`` kernel module is loaded in :ref:`single kernel thread 195mode <kni_kernel_thread_mode>`, :ref:`loopback mode <kni_loopback_mode>` 196enabled, and the :ref:`default carrier state <kni_default_carrier_state>` 197is set to *on* so that the corresponding physical NIC port does not have 198to be connected in order to use the KNI interface. One KNI interface 199``vEth0_0`` is created for port 0 and one KNI interface ``vEth1_0`` is 200created for port 1. Since ``rte_kni`` is loaded in "single kernel thread" 201mode, the one kernel thread is bound to lcore 8. 202 203Since the physical NIC ports are not being used, link monitoring can be 204disabled by **not** specifying the ``-m`` flag to ``kni``: 205 206.. code-block:: console 207 208 # rmmod rte_kni 209 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko lo_mode=lo_mode_fifo carrier=on 210 # ./<build-dir>/examples/dpdk-kni -l 4-7 -n 4 -- -P -p 0x3 --config="(0,4,6,8),(1,5,7,9)" 211 212KNI Operations 213-------------- 214 215Once the ``kni`` application is started, the user can use the normal 216Linux commands to manage the KNI interfaces as if they were any other 217Linux network interface. 218 219Enable KNI interface and assign an IP address: 220 221.. code-block:: console 222 223 # ip addr add dev vEth0_0 192.168.0.1 224 225Show KNI interface configuration and statistics: 226 227.. code-block:: console 228 229 # ip -s -d addr show vEth0_0 230 231The user can also check and reset the packet statistics inside the ``kni`` 232application by sending the app the USR1 and USR2 signals: 233 234.. code-block:: console 235 236 # Print statistics 237 # pkill -USR1 kni 238 239 # Zero statistics 240 # pkill -USR2 kni 241 242Dump network traffic: 243 244.. code-block:: console 245 246 # tshark -n -i vEth0_0 247 248The normal Linux commands can also be used to change the MAC address and 249MTU size used by the physical NIC which corresponds to the KNI interface. 250However, if more than one KNI interface is configured for a physical port, 251these commands will only work on the first KNI interface for that port. 252 253Change the MAC address: 254 255.. code-block:: console 256 257 # ip link set dev vEth0_0 lladdr 0C:01:02:03:04:08 258 259Change the MTU size: 260 261.. code-block:: console 262 263 # ip link set dev vEth0_0 mtu 1450 264 265Limited ethtool support: 266 267.. code-block:: console 268 269 # ethtool -i vEth0_0 270 271When the ``kni`` application is closed, all the KNI interfaces are deleted 272from the Linux kernel. 273 274Explanation 275----------- 276 277The following sections provide some explanation of code. 278 279Initialization 280~~~~~~~~~~~~~~ 281 282Setup of mbuf pool, driver and queues is similar to the setup done in the :doc:`l2_forward_real_virtual`.. 283In addition, one or more kernel NIC interfaces are allocated for each 284of the configured ports according to the command line parameters. 285 286The code for allocating the kernel NIC interfaces for a specific port is 287in the function ``kni_alloc``. 288 289The other step in the initialization process that is unique to this sample application 290is the association of each port with lcores for RX, TX and kernel threads. 291 292* One lcore to read from the port and write to the associated one or more KNI devices 293 294* Another lcore to read from one or more KNI devices and write to the port 295 296* Other lcores for pinning the kernel threads on one by one 297 298This is done by using the ``kni_port_params_array[]`` array, which is indexed by the port ID. 299The code is in the function ``parse_config``. 300 301Packet Forwarding 302~~~~~~~~~~~~~~~~~ 303 304After the initialization steps are completed, the main_loop() function is run on each lcore. 305This function first checks the lcore_id against the user provided lcore_rx and lcore_tx 306to see if this lcore is reading from or writing to kernel NIC interfaces. 307 308For the case that reads from a NIC port and writes to the kernel NIC interfaces (``kni_ingress``), 309the packet reception is the same as in L2 Forwarding sample application 310(see :ref:`l2_fwd_app_rx_tx_packets`). 311The packet transmission is done by sending mbufs into the kernel NIC interfaces by ``rte_kni_tx_burst()``. 312The KNI library automatically frees the mbufs after the kernel successfully copied the mbufs. 313 314For the other case that reads from kernel NIC interfaces 315and writes to a physical NIC port (``kni_egress``), 316packets are retrieved by reading mbufs from kernel NIC interfaces by ``rte_kni_rx_burst()``. 317The packet transmission is the same as in the L2 Forwarding sample application 318(see :ref:`l2_fwd_app_rx_tx_packets`). 319