1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(C) 2021 Marvell. 3 4Marvell cnxk platform guide 5=========================== 6 7This document gives an overview of **Marvell OCTEON CN9K and CN10K** RVU H/W block, 8packet flow and procedure to build DPDK on OCTEON cnxk platform. 9 10More information about CN9K and CN10K SoC can be found at `Marvell Official Website 11<https://www.marvell.com/embedded-processors/infrastructure-processors/>`_. 12 13Supported OCTEON cnxk SoCs 14-------------------------- 15 16- CN93xx 17- CN96xx 18- CN98xx 19- CN106xx 20- CNF105xx 21 22Resource Virtualization Unit architecture 23----------------------------------------- 24 25The :numref:`figure_cnxk_resource_virtualization` diagram depicts the 26RVU architecture and a resource provisioning example. 27 28.. _figure_cnxk_resource_virtualization: 29 30.. figure:: img/cnxk_resource_virtualization.* 31 32 cnxk Resource virtualization architecture and provisioning example 33 34 35Resource Virtualization Unit (RVU) on Marvell's OCTEON CN9K/CN10K SoC maps HW 36resources belonging to the network, crypto and other functional blocks onto 37PCI-compatible physical and virtual functions. 38 39Each functional block has multiple local functions (LFs) for 40provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV 41physical functions (PFs) and virtual functions (VFs). 42 43The :numref:`table_cnxk_rvu_dpdk_mapping` shows the various local 44functions (LFs) provided by the RVU and its functional mapping to 45DPDK subsystem. 46 47.. _table_cnxk_rvu_dpdk_mapping: 48 49.. table:: RVU managed functional blocks and its mapping to DPDK subsystem 50 51 +---+-----+--------------------------------------------------------------+ 52 | # | LF | DPDK subsystem mapping | 53 +===+=====+==============================================================+ 54 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security| 55 +---+-----+--------------------------------------------------------------+ 56 | 2 | NPA | rte_mempool | 57 +---+-----+--------------------------------------------------------------+ 58 | 3 | NPC | rte_flow | 59 +---+-----+--------------------------------------------------------------+ 60 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter | 61 +---+-----+--------------------------------------------------------------+ 62 | 5 | SSO | rte_eventdev | 63 +---+-----+--------------------------------------------------------------+ 64 | 6 | TIM | rte_event_timer_adapter | 65 +---+-----+--------------------------------------------------------------+ 66 | 7 | LBK | rte_ethdev | 67 +---+-----+--------------------------------------------------------------+ 68 | 8 | DPI | rte_dmadev | 69 +---+-----+--------------------------------------------------------------+ 70 | 9 | SDP | rte_ethdev | 71 +---+-----+--------------------------------------------------------------+ 72 | 10| REE | rte_regexdev | 73 +---+-----+--------------------------------------------------------------+ 74 | 11| BPHY| rte_rawdev | 75 +---+-----+--------------------------------------------------------------+ 76 | 12| GPIO| rte_rawdev | 77 +---+-----+--------------------------------------------------------------+ 78 79PF0 is called the administrative / admin function (AF) and has exclusive 80privileges to provision RVU functional block's LFs to each of the PF/VF. 81 82PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving 83requests from PF/VF, AF does resource provisioning and other HW configuration. 84 85AF is always attached to host, but PF/VFs may be used by host kernel itself, 86or attached to VMs or to userspace applications like DPDK, etc. So, AF has to 87handle provisioning/configuration requests sent by any device from any domain. 88 89The AF driver does not receive or process any data. 90It is only a configuration driver used in control path. 91 92The :numref:`figure_cnxk_resource_virtualization` diagram also shows a 93resource provisioning example where, 94 951. PFx and PFx-VF0 bound to Linux netdev driver. 962. PFx-VF1 ethdev driver bound to the first DPDK application. 973. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application. 98 99LBK HW Access 100------------- 101 102Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX. 103The loopback block has N channels and contains data buffering that is shared across 104all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's 105VFs are exposed as ethdev device and odd-even pairs of VFs are tied together, 106that is, packets sent on odd VF end up received on even VF and vice versa. 107This would enable HW accelerated means of communication between two domains 108where even VF bound to the first domain and odd VF bound to the second domain. 109 110Typical application usage models are, 111 112#. Communication between the Linux kernel and DPDK application. 113#. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement. 114#. Communication between two different DPDK applications. 115 116SDP interface 117------------- 118 119System DPI Packet Interface unit(SDP) provides PCIe endpoint support for remote host 120to DMA packets into and out of cnxk SoC. SDP interface comes in to live only when 121cnxk SoC is connected in PCIe endpoint mode. It can be used to send/receive 122packets to/from remote host machine using input/output queue pairs exposed to it. 123SDP interface receives input packets from remote host from NIX-RX and sends packets 124to remote host using NIX-TX. Remote host machine need to use corresponding driver 125(kernel/user mode) to communicate with SDP interface on cnxk SoC. SDP supports 126single PCIe SRIOV physical function(PF) and multiple virtual functions(VF's). Users 127can bind PF or VF to use SDP interface and it will be enumerated as ethdev ports. 128 129The primary use case for SDP is to enable the smart NIC use case. Typical usage models are, 130 131#. Communication channel between remote host and cnxk SoC over PCIe. 132#. Transfer packets received from network interface to remote host over PCIe and 133 vice-versa. 134 135cnxk packet flow 136---------------------- 137 138The :numref:`figure_cnxk_packet_flow_hw_accelerators` diagram depicts 139the packet flow on cnxk SoC in conjunction with use of various HW accelerators. 140 141.. _figure_cnxk_packet_flow_hw_accelerators: 142 143.. figure:: img/cnxk_packet_flow_hw_accelerators.* 144 145 cnxk packet flow in conjunction with use of HW accelerators 146 147HW Offload Drivers 148------------------ 149 150This section lists dataplane H/W block(s) available in cnxk SoC. 151 152#. **Ethdev Driver** 153 See :doc:`../nics/cnxk` for NIX Ethdev driver information. 154 155#. **Mempool Driver** 156 See :doc:`../mempool/cnxk` for NPA mempool driver information. 157 158#. **Baseband PHY Driver** 159 See :doc:`../rawdevs/cnxk_bphy` for Baseband PHY driver information. 160 161#. **Dmadev Driver** 162 See :doc:`../dmadevs/cnxk` for DPI Dmadev driver information. 163 164#. **Regex Device Driver** 165 See :doc:`../regexdevs/cn9k` for REE Regex device driver information. 166 167Procedure to Setup Platform 168--------------------------- 169 170There are three main prerequisites for setting up DPDK on cnxk 171compatible board: 172 1731. **RVU AF Linux kernel driver** 174 175 The dependent kernel drivers can be obtained from the 176 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_. 177 178 Alternatively, the Marvell SDK also provides the required kernel drivers. 179 180 Linux kernel should be configured with the following features enabled: 181 182.. code-block:: console 183 184 # 64K pages enabled for better performance 185 CONFIG_ARM64_64K_PAGES=y 186 CONFIG_ARM64_VA_BITS_48=y 187 # huge pages support enabled 188 CONFIG_HUGETLBFS=y 189 CONFIG_HUGETLB_PAGE=y 190 # VFIO enabled with TYPE1 IOMMU at minimum 191 CONFIG_VFIO_IOMMU_TYPE1=y 192 CONFIG_VFIO_VIRQFD=y 193 CONFIG_VFIO=y 194 CONFIG_VFIO_NOIOMMU=y 195 CONFIG_VFIO_PCI=y 196 CONFIG_VFIO_PCI_MMAP=y 197 # SMMUv3 driver 198 CONFIG_ARM_SMMU_V3=y 199 # ARMv8.1 LSE atomics 200 CONFIG_ARM64_LSE_ATOMICS=y 201 # OCTEONTX2 drivers 202 CONFIG_OCTEONTX2_MBOX=y 203 CONFIG_OCTEONTX2_AF=y 204 # Enable if netdev PF driver required 205 CONFIG_OCTEONTX2_PF=y 206 # Enable if netdev VF driver required 207 CONFIG_OCTEONTX2_VF=y 208 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y 209 # Enable if OCTEONTX2 DMA PF driver required 210 CONFIG_OCTEONTX2_DPI_PF=n 211 2122. **ARM64 Linux Tool Chain** 213 214 For example, the *aarch64* Linaro Toolchain, which can be obtained from 215 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_. 216 217 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is 218 optimized for cnxk CPU. 219 2203. **Rootfile system** 221 222 Any *aarch64* supporting filesystem may be used. For example, 223 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained 224 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_. 225 226 Alternatively, the Marvell SDK provides the buildroot based root filesystem. 227 The SDK includes all the above prerequisites necessary to bring up the cnxk board. 228 229- Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment. 230 231 232Debugging Options 233----------------- 234 235.. _table_cnxk_common_debug_options: 236 237.. table:: cnxk common debug options 238 239 +---+------------+-------------------------------------------------------+ 240 | # | Component | EAL log command | 241 +===+============+=======================================================+ 242 | 1 | Common | --log-level='pmd\.cnxk\.base,8' | 243 +---+------------+-------------------------------------------------------+ 244 | 2 | Mailbox | --log-level='pmd\.cnxk\.mbox,8' | 245 +---+------------+-------------------------------------------------------+ 246 247Debugfs support 248~~~~~~~~~~~~~~~ 249 250The **RVU AF Linux kernel driver** provides support to dump RVU blocks 251context or stats using debugfs. 252 253Enable ``debugfs`` by: 254 2551. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``. 2562. Boot OCTEON CN9K/CN10K with debugfs supported kernel. 2573. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using. 258 259.. code-block:: console 260 261 # mount -t debugfs none /sys/kernel/debug 262 263Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC, 264SSO & RPM. 265 266The file structure under ``/sys/kernel/debug`` is as follows 267 268.. code-block:: console 269 270 octeontx2/ 271 | 272 cn10k/ 273 |-- rpm 274 | |-- rpm0 275 | | '-- lmac0 276 | | '-- stats 277 | |-- rpm1 278 | | |-- lmac0 279 | | | '-- stats 280 | | '-- lmac1 281 | | '-- stats 282 | '-- rpm2 283 | '-- lmac0 284 | '-- stats 285 |-- cpt 286 | |-- cpt_engines_info 287 | |-- cpt_engines_sts 288 | |-- cpt_err_info 289 | |-- cpt_lfs_info 290 | '-- cpt_pc 291 |---- nix 292 | |-- cq_ctx 293 | |-- ndc_rx_cache 294 | |-- ndc_rx_hits_miss 295 | |-- ndc_tx_cache 296 | |-- ndc_tx_hits_miss 297 | |-- qsize 298 | |-- rq_ctx 299 | '-- sq_ctx 300 |-- npa 301 | |-- aura_ctx 302 | |-- ndc_cache 303 | |-- ndc_hits_miss 304 | |-- pool_ctx 305 | '-- qsize 306 |-- npc 307 | |-- mcam_info 308 | |-- mcam_rules 309 | '-- rx_miss_act_stats 310 |-- rsrc_alloc 311 '-- sso 312 |-- hws 313 | '-- sso_hws_info 314 '-- hwgrp 315 |-- sso_hwgrp_aq_thresh 316 |-- sso_hwgrp_iaq_walk 317 |-- sso_hwgrp_pc 318 |-- sso_hwgrp_free_list_walk 319 |-- sso_hwgrp_ient_walk 320 '-- sso_hwgrp_taq_walk 321 322RVU block LF allocation: 323 324.. code-block:: console 325 326 cat /sys/kernel/debug/cn10k/rsrc_alloc 327 328 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT 329 PF1 0 0 330 PF4 1 331 PF13 0, 1 0, 1 0 332 333RPM example usage: 334 335.. code-block:: console 336 337 cat /sys/kernel/debug/cn10k/rpm/rpm0/lmac0/stats 338 339 =======Link Status====== 340 341 Link is UP 25000 Mbps 342 343 =======NIX RX_STATS(rpm port level)====== 344 345 rx_ucast_frames: 0 346 rx_mcast_frames: 0 347 rx_bcast_frames: 0 348 rx_frames: 0 349 rx_bytes: 0 350 rx_drops: 0 351 rx_errors: 0 352 353 =======NIX TX_STATS(rpm port level)====== 354 355 tx_ucast_frames: 0 356 tx_mcast_frames: 0 357 tx_bcast_frames: 0 358 tx_frames: 0 359 tx_bytes: 0 360 tx_drops: 0 361 362 =======rpm RX_STATS====== 363 364 Octets of received packets: 0 365 Octets of received packets with out error: 0 366 Received packets with alignment errors: 0 367 Control/PAUSE packets received: 0 368 Packets received with Frame too long Errors: 0 369 Packets received with a1nrange length Errors: 0 370 Received packets: 0 371 Packets received with FrameCheckSequenceErrors: 0 372 Packets received with VLAN header: 0 373 Error packets: 0 374 Packets received with unicast DMAC: 0 375 Packets received with multicast DMAC: 0 376 Packets received with broadcast DMAC: 0 377 Dropped packets: 0 378 Total frames received on interface: 0 379 Packets received with an octet count < 64: 0 380 Packets received with an octet count == 64: 0 381 Packets received with an octet count of 65–127: 0 382 Packets received with an octet count of 128-255: 0 383 Packets received with an octet count of 256-511: 0 384 Packets received with an octet count of 512-1023: 0 385 Packets received with an octet count of 1024-1518: 0 386 Packets received with an octet count of > 1518: 0 387 Oversized Packets: 0 388 Jabber Packets: 0 389 Fragmented Packets: 0 390 CBFC(class based flow control) pause frames received for class 0: 0 391 CBFC pause frames received for class 1: 0 392 CBFC pause frames received for class 2: 0 393 CBFC pause frames received for class 3: 0 394 CBFC pause frames received for class 4: 0 395 CBFC pause frames received for class 5: 0 396 CBFC pause frames received for class 6: 0 397 CBFC pause frames received for class 7: 0 398 CBFC pause frames received for class 8: 0 399 CBFC pause frames received for class 9: 0 400 CBFC pause frames received for class 10: 0 401 CBFC pause frames received for class 11: 0 402 CBFC pause frames received for class 12: 0 403 CBFC pause frames received for class 13: 0 404 CBFC pause frames received for class 14: 0 405 CBFC pause frames received for class 15: 0 406 MAC control packets received: 0 407 408 =======rpm TX_STATS====== 409 410 Total octets sent on the interface: 0 411 Total octets transmitted OK: 0 412 Control/Pause frames sent: 0 413 Total frames transmitted OK: 0 414 Total frames sent with VLAN header: 0 415 Error Packets: 0 416 Packets sent to unicast DMAC: 0 417 Packets sent to the multicast DMAC: 0 418 Packets sent to a broadcast DMAC: 0 419 Packets sent with an octet count == 64: 0 420 Packets sent with an octet count of 65–127: 0 421 Packets sent with an octet count of 128-255: 0 422 Packets sent with an octet count of 256-511: 0 423 Packets sent with an octet count of 512-1023: 0 424 Packets sent with an octet count of 1024-1518: 0 425 Packets sent with an octet count of > 1518: 0 426 CBFC(class based flow control) pause frames transmitted for class 0: 0 427 CBFC pause frames transmitted for class 1: 0 428 CBFC pause frames transmitted for class 2: 0 429 CBFC pause frames transmitted for class 3: 0 430 CBFC pause frames transmitted for class 4: 0 431 CBFC pause frames transmitted for class 5: 0 432 CBFC pause frames transmitted for class 6: 0 433 CBFC pause frames transmitted for class 7: 0 434 CBFC pause frames transmitted for class 8: 0 435 CBFC pause frames transmitted for class 9: 0 436 CBFC pause frames transmitted for class 10: 0 437 CBFC pause frames transmitted for class 11: 0 438 CBFC pause frames transmitted for class 12: 0 439 CBFC pause frames transmitted for class 13: 0 440 CBFC pause frames transmitted for class 14: 0 441 CBFC pause frames transmitted for class 15: 0 442 MAC control packets sent: 0 443 Total frames sent on the interface: 0 444 445CPT example usage: 446 447.. code-block:: console 448 449 cat /sys/kernel/debug/cn10k/cpt/cpt_pc 450 451 CPT instruction requests 0 452 CPT instruction latency 0 453 CPT NCB read requests 0 454 CPT NCB read latency 0 455 CPT read requests caused by UC fills 0 456 CPT active cycles pc 1395642 457 CPT clock count pc 5579867595493 458 459NIX example usage: 460 461.. code-block:: console 462 463 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/cn10k/nix/cq_ctx 464 cat /sys/kernel/debug/cn10k/nix/cq_ctx 465 echo 0 0 > /sys/kernel/debug/cn10k/nix/cq_ctx 466 cat /sys/kernel/debug/cn10k/nix/cq_ctx 467 468 =====cq_ctx for nixlf:0 and qidx:0 is===== 469 W0: base 158ef1a00 470 471 W1: wrptr 0 472 W1: avg_con 0 473 W1: cint_idx 0 474 W1: cq_err 0 475 W1: qint_idx 0 476 W1: bpid 0 477 W1: bp_ena 0 478 479 W2: update_time 31043 480 W2:avg_level 255 481 W2: head 0 482 W2:tail 0 483 484 W3: cq_err_int_ena 5 485 W3:cq_err_int 0 486 W3: qsize 4 487 W3:caching 1 488 W3: substream 0x000 489 W3: ena 1 490 W3: drop_ena 1 491 W3: drop 64 492 W3: bp 0 493 494NPA example usage: 495 496.. code-block:: console 497 498 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/cn10k/npa/pool_ctx 499 cat /sys/kernel/debug/cn10k/npa/pool_ctx 500 echo 0 0 > /sys/kernel/debug/cn10k/npa/pool_ctx 501 cat /sys/kernel/debug/cn10k/npa/pool_ctx 502 503 ======POOL : 0======= 504 W0: Stack base 1375bff00 505 W1: ena 1 506 W1: nat_align 1 507 W1: stack_caching 1 508 W1: stack_way_mask 0 509 W1: buf_offset 1 510 W1: buf_size 19 511 W2: stack_max_pages 24315 512 W2: stack_pages 24314 513 W3: op_pc 267456 514 W4: stack_offset 2 515 W4: shift 5 516 W4: avg_level 255 517 W4: avg_con 0 518 W4: fc_ena 0 519 W4: fc_stype 0 520 W4: fc_hyst_bits 0 521 W4: fc_up_crossing 0 522 W4: update_time 62993 523 W5: fc_addr 0 524 W6: ptr_start 1593adf00 525 W7: ptr_end 180000000 526 W8: err_int 0 527 W8: err_int_ena 7 528 W8: thresh_int 0 529 W8: thresh_int_ena 0 530 W8: thresh_up 0 531 W8: thresh_qint_idx 0 532 W8: err_qint_idx 0 533 534NPC example usage: 535 536.. code-block:: console 537 538 cat /sys/kernel/debug/cn10k/npc/mcam_info 539 540 NPC MCAM info: 541 RX keywidth : 224bits 542 TX keywidth : 224bits 543 544 MCAM entries : 2048 545 Reserved : 158 546 Available : 1890 547 548 MCAM counters : 512 549 Reserved : 1 550 Available : 511 551 552SSO example usage: 553 554.. code-block:: console 555 556 Usage: echo [<hws>/all] > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info 557 echo 0 > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info 558 559 ================================================== 560 SSOW HWS[0] Arbitration State 0x0 561 SSOW HWS[0] Guest Machine Control 0x0 562 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff 563 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff 564 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff 565 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff 566 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff 567 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff 568 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff 569 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff 570 ================================================== 571 572Compile DPDK 573------------ 574 575DPDK may be compiled either natively on OCTEON CN9K/CN10K platform or cross-compiled on 576an x86 based platform. 577 578Native Compilation 579~~~~~~~~~~~~~~~~~~ 580 581.. code-block:: console 582 583 meson build 584 ninja -C build 585 586Cross Compilation 587~~~~~~~~~~~~~~~~~ 588 589Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details. 590 591CN9K: 592 593.. code-block:: console 594 595 meson build --cross-file config/arm/arm64_cn9k_linux_gcc 596 ninja -C build 597 598CN10K: 599 600.. code-block:: console 601 602 meson build --cross-file config/arm/arm64_cn10k_linux_gcc 603 ninja -C build 604 605.. note:: 606 607 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain, 608 if Marvell toolchain is available then it can be used by overriding the 609 c, cpp, ar, strip ``binaries`` attributes to respective Marvell 610 toolchain binaries in ``config/arm/arm64_cn10k_linux_gcc`` file. 611