1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2019 Marvell International Ltd. 3 4Marvell OCTEON TX2 Platform Guide 5================================= 6 7This document gives an overview of **Marvell OCTEON TX2** RVU H/W block, 8packet flow and procedure to build DPDK on OCTEON TX2 platform. 9 10More information about OCTEON TX2 SoC can be found at `Marvell Official Website 11<https://www.marvell.com/embedded-processors/infrastructure-processors/>`_. 12 13Supported OCTEON TX2 SoCs 14------------------------- 15 16- CN98xx 17- CN96xx 18- CN93xx 19 20OCTEON TX2 Resource Virtualization Unit architecture 21---------------------------------------------------- 22 23The :numref:`figure_octeontx2_resource_virtualization` diagram depicts the 24RVU architecture and a resource provisioning example. 25 26.. _figure_octeontx2_resource_virtualization: 27 28.. figure:: img/octeontx2_resource_virtualization.* 29 30 OCTEON TX2 Resource virtualization architecture and provisioning example 31 32 33Resource Virtualization Unit (RVU) on Marvell's OCTEON TX2 SoC maps HW 34resources belonging to the network, crypto and other functional blocks onto 35PCI-compatible physical and virtual functions. 36 37Each functional block has multiple local functions (LFs) for 38provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV 39physical functions (PFs) and virtual functions (VFs). 40 41The :numref:`table_octeontx2_rvu_dpdk_mapping` shows the various local 42functions (LFs) provided by the RVU and its functional mapping to 43DPDK subsystem. 44 45.. _table_octeontx2_rvu_dpdk_mapping: 46 47.. table:: RVU managed functional blocks and its mapping to DPDK subsystem 48 49 +---+-----+--------------------------------------------------------------+ 50 | # | LF | DPDK subsystem mapping | 51 +===+=====+==============================================================+ 52 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security| 53 +---+-----+--------------------------------------------------------------+ 54 | 2 | NPA | rte_mempool | 55 +---+-----+--------------------------------------------------------------+ 56 | 3 | NPC | rte_flow | 57 +---+-----+--------------------------------------------------------------+ 58 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter | 59 +---+-----+--------------------------------------------------------------+ 60 | 5 | SSO | rte_eventdev | 61 +---+-----+--------------------------------------------------------------+ 62 | 6 | TIM | rte_event_timer_adapter | 63 +---+-----+--------------------------------------------------------------+ 64 | 7 | LBK | rte_ethdev | 65 +---+-----+--------------------------------------------------------------+ 66 | 8 | DPI | rte_rawdev | 67 +---+-----+--------------------------------------------------------------+ 68 | 9 | SDP | rte_ethdev | 69 +---+-----+--------------------------------------------------------------+ 70 | 10| REE | rte_regexdev | 71 +---+-----+--------------------------------------------------------------+ 72 73PF0 is called the administrative / admin function (AF) and has exclusive 74privileges to provision RVU functional block's LFs to each of the PF/VF. 75 76PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving 77requests from PF/VF, AF does resource provisioning and other HW configuration. 78 79AF is always attached to host, but PF/VFs may be used by host kernel itself, 80or attached to VMs or to userspace applications like DPDK, etc. So, AF has to 81handle provisioning/configuration requests sent by any device from any domain. 82 83The AF driver does not receive or process any data. 84It is only a configuration driver used in control path. 85 86The :numref:`figure_octeontx2_resource_virtualization` diagram also shows a 87resource provisioning example where, 88 891. PFx and PFx-VF0 bound to Linux netdev driver. 902. PFx-VF1 ethdev driver bound to the first DPDK application. 913. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application. 92 93LBK HW Access 94------------- 95 96Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX. 97The loopback block has N channels and contains data buffering that is shared across 98all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's 99VFs are exposed as ethdev device and odd-even pairs of VFs are tied together, 100that is, packets sent on odd VF end up received on even VF and vice versa. 101This would enable HW accelerated means of communication between two domains 102where even VF bound to the first domain and odd VF bound to the second domain. 103 104Typical application usage models are, 105 106#. Communication between the Linux kernel and DPDK application. 107#. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement. 108#. Communication between two different DPDK applications. 109 110SDP interface 111------------- 112 113System DPI Packet Interface unit(SDP) provides PCIe endpoint support for remote host 114to DMA packets into and out of OCTEON TX2 SoC. SDP interface comes in to live only when 115OCTEON TX2 SoC is connected in PCIe endpoint mode. It can be used to send/receive 116packets to/from remote host machine using input/output queue pairs exposed to it. 117SDP interface receives input packets from remote host from NIX-RX and sends packets 118to remote host using NIX-TX. Remote host machine need to use corresponding driver 119(kernel/user mode) to communicate with SDP interface on OCTEON TX2 SoC. SDP supports 120single PCIe SRIOV physical function(PF) and multiple virtual functions(VF's). Users 121can bind PF or VF to use SDP interface and it will be enumerated as ethdev ports. 122 123The primary use case for SDP is to enable the smart NIC use case. Typical usage models are, 124 125#. Communication channel between remote host and OCTEON TX2 SoC over PCIe. 126#. Transfer packets received from network interface to remote host over PCIe and 127 vice-versa. 128 129OCTEON TX2 packet flow 130---------------------- 131 132The :numref:`figure_octeontx2_packet_flow_hw_accelerators` diagram depicts 133the packet flow on OCTEON TX2 SoC in conjunction with use of various HW accelerators. 134 135.. _figure_octeontx2_packet_flow_hw_accelerators: 136 137.. figure:: img/octeontx2_packet_flow_hw_accelerators.* 138 139 OCTEON TX2 packet flow in conjunction with use of HW accelerators 140 141HW Offload Drivers 142------------------ 143 144This section lists dataplane H/W block(s) available in OCTEON TX2 SoC. 145 146#. **Ethdev Driver** 147 See :doc:`../nics/octeontx2` for NIX Ethdev driver information. 148 149#. **Mempool Driver** 150 See :doc:`../mempool/octeontx2` for NPA mempool driver information. 151 152#. **Event Device Driver** 153 See :doc:`../eventdevs/octeontx2` for SSO event device driver information. 154 155#. **DMA Rawdev Driver** 156 See :doc:`../rawdevs/octeontx2_dma` for DMA driver information. 157 158#. **Crypto Device Driver** 159 See :doc:`../cryptodevs/octeontx2` for CPT crypto device driver information. 160 161#. **Regex Device Driver** 162 See :doc:`../regexdevs/octeontx2` for REE regex device driver information. 163 164Procedure to Setup Platform 165--------------------------- 166 167There are three main prerequisites for setting up DPDK on OCTEON TX2 168compatible board: 169 1701. **OCTEON TX2 Linux kernel driver** 171 172 The dependent kernel drivers can be obtained from the 173 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_. 174 175 Alternatively, the Marvell SDK also provides the required kernel drivers. 176 177 Linux kernel should be configured with the following features enabled: 178 179.. code-block:: console 180 181 # 64K pages enabled for better performance 182 CONFIG_ARM64_64K_PAGES=y 183 CONFIG_ARM64_VA_BITS_48=y 184 # huge pages support enabled 185 CONFIG_HUGETLBFS=y 186 CONFIG_HUGETLB_PAGE=y 187 # VFIO enabled with TYPE1 IOMMU at minimum 188 CONFIG_VFIO_IOMMU_TYPE1=y 189 CONFIG_VFIO_VIRQFD=y 190 CONFIG_VFIO=y 191 CONFIG_VFIO_NOIOMMU=y 192 CONFIG_VFIO_PCI=y 193 CONFIG_VFIO_PCI_MMAP=y 194 # SMMUv3 driver 195 CONFIG_ARM_SMMU_V3=y 196 # ARMv8.1 LSE atomics 197 CONFIG_ARM64_LSE_ATOMICS=y 198 # OCTEONTX2 drivers 199 CONFIG_OCTEONTX2_MBOX=y 200 CONFIG_OCTEONTX2_AF=y 201 # Enable if netdev PF driver required 202 CONFIG_OCTEONTX2_PF=y 203 # Enable if netdev VF driver required 204 CONFIG_OCTEONTX2_VF=y 205 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y 206 # Enable if OCTEONTX2 DMA PF driver required 207 CONFIG_OCTEONTX2_DPI_PF=n 208 2092. **ARM64 Linux Tool Chain** 210 211 For example, the *aarch64* Linaro Toolchain, which can be obtained from 212 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_. 213 214 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is 215 optimized for OCTEON TX2 CPU. 216 2173. **Rootfile system** 218 219 Any *aarch64* supporting filesystem may be used. For example, 220 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained 221 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_. 222 223 Alternatively, the Marvell SDK provides the buildroot based root filesystem. 224 The SDK includes all the above prerequisites necessary to bring up the OCTEON TX2 board. 225 226- Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment. 227 228 229Debugging Options 230----------------- 231 232.. _table_octeontx2_common_debug_options: 233 234.. table:: OCTEON TX2 common debug options 235 236 +---+------------+-------------------------------------------------------+ 237 | # | Component | EAL log command | 238 +===+============+=======================================================+ 239 | 1 | Common | --log-level='pmd\.octeontx2\.base,8' | 240 +---+------------+-------------------------------------------------------+ 241 | 2 | Mailbox | --log-level='pmd\.octeontx2\.mbox,8' | 242 +---+------------+-------------------------------------------------------+ 243 244Debugfs support 245~~~~~~~~~~~~~~~ 246 247The **OCTEON TX2 Linux kernel driver** provides support to dump RVU blocks 248context or stats using debugfs. 249 250Enable ``debugfs`` by: 251 2521. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``. 2532. Boot OCTEON TX2 with debugfs supported kernel. 2543. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using. 255 256.. code-block:: console 257 258 # mount -t debugfs none /sys/kernel/debug 259 260Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC, 261SSO & CGX. 262 263The file structure under ``/sys/kernel/debug`` is as follows 264 265.. code-block:: console 266 267 octeontx2/ 268 |-- cgx 269 | |-- cgx0 270 | | '-- lmac0 271 | | '-- stats 272 | |-- cgx1 273 | | |-- lmac0 274 | | | '-- stats 275 | | '-- lmac1 276 | | '-- stats 277 | '-- cgx2 278 | '-- lmac0 279 | '-- stats 280 |-- cpt 281 | |-- cpt_engines_info 282 | |-- cpt_engines_sts 283 | |-- cpt_err_info 284 | |-- cpt_lfs_info 285 | '-- cpt_pc 286 |---- nix 287 | |-- cq_ctx 288 | |-- ndc_rx_cache 289 | |-- ndc_rx_hits_miss 290 | |-- ndc_tx_cache 291 | |-- ndc_tx_hits_miss 292 | |-- qsize 293 | |-- rq_ctx 294 | |-- sq_ctx 295 | '-- tx_stall_hwissue 296 |-- npa 297 | |-- aura_ctx 298 | |-- ndc_cache 299 | |-- ndc_hits_miss 300 | |-- pool_ctx 301 | '-- qsize 302 |-- npc 303 | |-- mcam_info 304 | '-- rx_miss_act_stats 305 |-- rsrc_alloc 306 '-- sso 307 |-- hws 308 | '-- sso_hws_info 309 '-- hwgrp 310 |-- sso_hwgrp_aq_thresh 311 |-- sso_hwgrp_iaq_walk 312 |-- sso_hwgrp_pc 313 |-- sso_hwgrp_free_list_walk 314 |-- sso_hwgrp_ient_walk 315 '-- sso_hwgrp_taq_walk 316 317RVU block LF allocation: 318 319.. code-block:: console 320 321 cat /sys/kernel/debug/octeontx2/rsrc_alloc 322 323 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT 324 PF1 0 0 325 PF4 1 326 PF13 0, 1 0, 1 0 327 328CGX example usage: 329 330.. code-block:: console 331 332 cat /sys/kernel/debug/octeontx2/cgx/cgx2/lmac0/stats 333 334 =======Link Status====== 335 Link is UP 40000 Mbps 336 =======RX_STATS====== 337 Received packets: 0 338 Octets of received packets: 0 339 Received PAUSE packets: 0 340 Received PAUSE and control packets: 0 341 Filtered DMAC0 (NIX-bound) packets: 0 342 Filtered DMAC0 (NIX-bound) octets: 0 343 Packets dropped due to RX FIFO full: 0 344 Octets dropped due to RX FIFO full: 0 345 Error packets: 0 346 Filtered DMAC1 (NCSI-bound) packets: 0 347 Filtered DMAC1 (NCSI-bound) octets: 0 348 NCSI-bound packets dropped: 0 349 NCSI-bound octets dropped: 0 350 =======TX_STATS====== 351 Packets dropped due to excessive collisions: 0 352 Packets dropped due to excessive deferral: 0 353 Multiple collisions before successful transmission: 0 354 Single collisions before successful transmission: 0 355 Total octets sent on the interface: 0 356 Total frames sent on the interface: 0 357 Packets sent with an octet count < 64: 0 358 Packets sent with an octet count == 64: 0 359 Packets sent with an octet count of 65127: 0 360 Packets sent with an octet count of 128-255: 0 361 Packets sent with an octet count of 256-511: 0 362 Packets sent with an octet count of 512-1023: 0 363 Packets sent with an octet count of 1024-1518: 0 364 Packets sent with an octet count of > 1518: 0 365 Packets sent to a broadcast DMAC: 0 366 Packets sent to the multicast DMAC: 0 367 Transmit underflow and were truncated: 0 368 Control/PAUSE packets sent: 0 369 370CPT example usage: 371 372.. code-block:: console 373 374 cat /sys/kernel/debug/octeontx2/cpt/cpt_pc 375 376 CPT instruction requests 0 377 CPT instruction latency 0 378 CPT NCB read requests 0 379 CPT NCB read latency 0 380 CPT read requests caused by UC fills 0 381 CPT active cycles pc 1395642 382 CPT clock count pc 5579867595493 383 384NIX example usage: 385 386.. code-block:: console 387 388 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/octeontx2/nix/cq_ctx 389 cat /sys/kernel/debug/octeontx2/nix/cq_ctx 390 echo 0 0 > /sys/kernel/debug/octeontx2/nix/cq_ctx 391 cat /sys/kernel/debug/octeontx2/nix/cq_ctx 392 393 =====cq_ctx for nixlf:0 and qidx:0 is===== 394 W0: base 158ef1a00 395 396 W1: wrptr 0 397 W1: avg_con 0 398 W1: cint_idx 0 399 W1: cq_err 0 400 W1: qint_idx 0 401 W1: bpid 0 402 W1: bp_ena 0 403 404 W2: update_time 31043 405 W2:avg_level 255 406 W2: head 0 407 W2:tail 0 408 409 W3: cq_err_int_ena 5 410 W3:cq_err_int 0 411 W3: qsize 4 412 W3:caching 1 413 W3: substream 0x000 414 W3: ena 1 415 W3: drop_ena 1 416 W3: drop 64 417 W3: bp 0 418 419NPA example usage: 420 421.. code-block:: console 422 423 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/octeontx2/npa/pool_ctx 424 cat /sys/kernel/debug/octeontx2/npa/pool_ctx 425 echo 0 0 > /sys/kernel/debug/octeontx2/npa/pool_ctx 426 cat /sys/kernel/debug/octeontx2/npa/pool_ctx 427 428 ======POOL : 0======= 429 W0: Stack base 1375bff00 430 W1: ena 1 431 W1: nat_align 1 432 W1: stack_caching 1 433 W1: stack_way_mask 0 434 W1: buf_offset 1 435 W1: buf_size 19 436 W2: stack_max_pages 24315 437 W2: stack_pages 24314 438 W3: op_pc 267456 439 W4: stack_offset 2 440 W4: shift 5 441 W4: avg_level 255 442 W4: avg_con 0 443 W4: fc_ena 0 444 W4: fc_stype 0 445 W4: fc_hyst_bits 0 446 W4: fc_up_crossing 0 447 W4: update_time 62993 448 W5: fc_addr 0 449 W6: ptr_start 1593adf00 450 W7: ptr_end 180000000 451 W8: err_int 0 452 W8: err_int_ena 7 453 W8: thresh_int 0 454 W8: thresh_int_ena 0 455 W8: thresh_up 0 456 W8: thresh_qint_idx 0 457 W8: err_qint_idx 0 458 459NPC example usage: 460 461.. code-block:: console 462 463 cat /sys/kernel/debug/octeontx2/npc/mcam_info 464 465 NPC MCAM info: 466 RX keywidth : 224bits 467 TX keywidth : 224bits 468 469 MCAM entries : 2048 470 Reserved : 158 471 Available : 1890 472 473 MCAM counters : 512 474 Reserved : 1 475 Available : 511 476 477SSO example usage: 478 479.. code-block:: console 480 481 Usage: echo [<hws>/all] > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info 482 echo 0 > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info 483 484 ================================================== 485 SSOW HWS[0] Arbitration State 0x0 486 SSOW HWS[0] Guest Machine Control 0x0 487 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff 488 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff 489 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff 490 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff 491 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff 492 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff 493 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff 494 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff 495 ================================================== 496 497Compile DPDK 498------------ 499 500DPDK may be compiled either natively on OCTEON TX2 platform or cross-compiled on 501an x86 based platform. 502 503Native Compilation 504~~~~~~~~~~~~~~~~~~ 505 506.. code-block:: console 507 508 meson build 509 ninja -C build 510 511Cross Compilation 512~~~~~~~~~~~~~~~~~ 513 514Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details. 515 516.. code-block:: console 517 518 meson build --cross-file config/arm/arm64_octeontx2_linux_gcc 519 ninja -C build 520 521.. note:: 522 523 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain, 524 if Marvell toolchain is available then it can be used by overriding the 525 c, cpp, ar, strip ``binaries`` attributes to respective Marvell 526 toolchain binaries in ``config/arm/arm64_octeontx2_linux_gcc`` file. 527