1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 42to specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as: 73 74 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 75 loader on Linux. See *AMD ROCm Platform Release Notes* 76 [AMD-ROCm-Release-Notes]_ for supported hardware and 77 software. 78 - AMD's PAL runtime using the *pal-amdhsa* loader on 79 Windows. 80 81 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 82 runtime using the *pal-amdpal* loader on Windows and Linux 83 Pro. 84 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 85 3D runtime using the *mesa-mesa3d* loader on Linux. 86 ============== ============================================================ 87 88 .. table:: AMDGPU Environments 89 :name: amdgpu-environment-table 90 91 ============ ============================================================== 92 Environment Description 93 ============ ============================================================== 94 *<empty>* Default. 95 ============ ============================================================== 96 97.. _amdgpu-processors: 98 99Processors 100---------- 101 102Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 103specify the AMDGPU processor together with optional target features. See 104:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 105specific information. 106 107Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 108 109* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 110 111 112 .. table:: AMDGPU Processors 113 :name: amdgpu-processor-table 114 115 =========== =============== ============ ===== ================= =============== =============== ====================== 116 Processor Alternative Target dGPU/ Target Target OS Support Example 117 Processor Triple APU Features Properties *(see* Products 118 Architecture Supported `amdgpu-os`_ 119 *and 120 corresponding 121 runtime release 122 notes for 123 current 124 information and 125 level of 126 support)* 127 =========== =============== ============ ===== ================= =============== =============== ====================== 128 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 129 ----------------------------------------------------------------------------------------------------------------------- 130 ``r600`` ``r600`` dGPU - Does not 131 support 132 generic 133 address 134 space 135 ``r630`` ``r600`` dGPU - Does not 136 support 137 generic 138 address 139 space 140 ``rs880`` ``r600`` dGPU - Does not 141 support 142 generic 143 address 144 space 145 ``rv670`` ``r600`` dGPU - Does not 146 support 147 generic 148 address 149 space 150 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 151 ----------------------------------------------------------------------------------------------------------------------- 152 ``rv710`` ``r600`` dGPU - Does not 153 support 154 generic 155 address 156 space 157 ``rv730`` ``r600`` dGPU - Does not 158 support 159 generic 160 address 161 space 162 ``rv770`` ``r600`` dGPU - Does not 163 support 164 generic 165 address 166 space 167 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 168 ----------------------------------------------------------------------------------------------------------------------- 169 ``cedar`` ``r600`` dGPU - Does not 170 support 171 generic 172 address 173 space 174 ``cypress`` ``r600`` dGPU - Does not 175 support 176 generic 177 address 178 space 179 ``juniper`` ``r600`` dGPU - Does not 180 support 181 generic 182 address 183 space 184 ``redwood`` ``r600`` dGPU - Does not 185 support 186 generic 187 address 188 space 189 ``sumo`` ``r600`` dGPU - Does not 190 support 191 generic 192 address 193 space 194 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 195 ----------------------------------------------------------------------------------------------------------------------- 196 ``barts`` ``r600`` dGPU - Does not 197 support 198 generic 199 address 200 space 201 ``caicos`` ``r600`` dGPU - Does not 202 support 203 generic 204 address 205 space 206 ``cayman`` ``r600`` dGPU - Does not 207 support 208 generic 209 address 210 space 211 ``turks`` ``r600`` dGPU - Does not 212 support 213 generic 214 address 215 space 216 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 217 ----------------------------------------------------------------------------------------------------------------------- 218 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 219 support 220 generic 221 address 222 space 223 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 224 - ``verde`` support 225 generic 226 address 227 space 228 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 229 - ``oland`` support 230 generic 231 address 232 space 233 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 234 ----------------------------------------------------------------------------------------------------------------------- 235 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 236 flat - *pal-amdhsa* - A6 Pro-7050B 237 scratch - *pal-amdpal* - A8-7100 238 - A8 Pro-7150B 239 - A10-7300 240 - A10 Pro-7350B 241 - FX-7500 242 - A8-7200P 243 - A10-7400P 244 - FX-7600P 245 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 246 flat - *pal-amdhsa* - FirePro W9100 247 scratch - *pal-amdpal* - FirePro S9150 248 - FirePro S9170 249 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 250 flat - *pal-amdhsa* - Radeon R9 290x 251 scratch - *pal-amdpal* - Radeon R390 252 - Radeon R390x 253 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 254 - ``mullins`` flat - *pal-amdpal* - E1-2200 255 scratch - E1-2500 256 - E2-3000 257 - E2-3800 258 - A4-5000 259 - A4-5100 260 - A6-5200 261 - A4 Pro-3340B 262 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 263 flat - *pal-amdpal* - Radeon HD 8770 264 scratch - R7 260 265 - R7 260X 266 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 267 flat - *pal-amdpal* 268 scratch .. TODO:: 269 270 Add product 271 names. 272 273 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 274 ----------------------------------------------------------------------------------------------------------------------- 275 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 276 flat - *pal-amdhsa* - Pro A6-8500B 277 scratch - *pal-amdpal* - A8-8600P 278 - Pro A8-8600B 279 - FX-8800P 280 - Pro A12-8800B 281 - A10-8700P 282 - Pro A10-8700B 283 - A10-8780P 284 - A10-9600P 285 - A10-9630P 286 - A12-9700P 287 - A12-9730P 288 - FX-9800P 289 - FX-9830P 290 - E2-9010 291 - A6-9210 292 - A9-9410 293 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 294 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 295 scratch - *pal-amdpal* - Radeon R9 385 296 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 297 - *pal-amdhsa* - Radeon R9 Fury 298 - *pal-amdpal* - Radeon R9 FuryX 299 - Radeon Pro Duo 300 - FirePro S9300x2 301 - Radeon Instinct MI8 302 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 303 flat - *pal-amdhsa* - Radeon RX 480 304 scratch - *pal-amdpal* - Radeon Instinct MI6 305 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 306 flat - *pal-amdhsa* 307 scratch - *pal-amdpal* 308 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 309 flat - *pal-amdhsa* - FirePro S7100 310 scratch - *pal-amdpal* - FirePro W7100 311 - Mobile FirePro 312 M7170 313 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 314 flat - *pal-amdhsa* 315 scratch - *pal-amdpal* .. TODO:: 316 317 Add product 318 names. 319 320 **GCN GFX9 (Vega)** [AMD-GCN-GFX9]_ 321 ----------------------------------------------------------------------------------------------------------------------- 322 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 323 flat - *pal-amdhsa* Frontier Edition 324 scratch - *pal-amdpal* - Radeon RX Vega 56 325 - Radeon RX Vega 64 326 - Radeon RX Vega 64 327 Liquid 328 - Radeon Instinct MI25 329 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 330 flat - *pal-amdhsa* - Ryzen 5 2400G 331 scratch - *pal-amdpal* 332 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 333 - *pal-amdhsa* 334 - *pal-amdpal* .. TODO:: 335 336 Add product 337 names. 338 339 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 340 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 341 scratch - *pal-amdpal* - Radeon VII 342 - Radeon Pro VII 343 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - Radeon Instinct MI100 Accelerator 344 - xnack - Absolute 345 flat 346 scratch 347 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 348 flat 349 scratch .. TODO:: 350 351 Add product 352 names. 353 354 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 355 - tgsplit flat 356 - xnack scratch .. TODO:: 357 - Packed 358 work-item Add product 359 IDs names. 360 361 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 362 flat - Ryzen 7 4700GE 363 scratch - Ryzen 5 4600G 364 - Ryzen 5 4600GE 365 - Ryzen 3 4300G 366 - Ryzen 3 4300GE 367 - Ryzen Pro 4000G 368 - Ryzen 7 Pro 4700G 369 - Ryzen 7 Pro 4750GE 370 - Ryzen 5 Pro 4650G 371 - Ryzen 5 Pro 4650GE 372 - Ryzen 3 Pro 4350G 373 - Ryzen 3 Pro 4350GE 374 375 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 376 ----------------------------------------------------------------------------------------------------------------------- 377 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 378 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 379 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 380 - Radeon Pro 5600M 381 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 382 - wavefrontsize64 - Absolute - *pal-amdhsa* 383 - xnack flat - *pal-amdpal* 384 scratch 385 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 386 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 387 - xnack scratch - *pal-amdpal* 388 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 389 ----------------------------------------------------------------------------------------------------------------------- 390 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 391 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 392 scratch - *pal-amdpal* - Radeon RX 6900 XT 393 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 394 - wavefrontsize64 flat - *pal-amdhsa* 395 scratch - *pal-amdpal* 396 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 397 - wavefrontsize64 flat - *pal-amdhsa* 398 scratch - *pal-amdpal* .. TODO:: 399 400 Add product 401 names. 402 403 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 404 - wavefrontsize64 flat 405 scratch .. TODO:: 406 407 Add product 408 names. 409 410 =========== =============== ============ ===== ================= =============== =============== ====================== 411 412.. _amdgpu-target-features: 413 414Target Features 415--------------- 416 417Target features control how code is generated to support certain 418processor specific features. Not all target features are supported by 419all processors. The runtime must ensure that the features supported by 420the device used to execute the code match the features enabled when 421generating the code. A mismatch of features may result in incorrect 422execution, or a reduction in performance. 423 424The target features supported by each processor is listed in 425:ref:`amdgpu-processor-table`. 426 427Target features are controlled by exactly one of the following Clang 428options: 429 430``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 431 432 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 433 optional components of the target ID. If omitted, the target feature has the 434 ``any`` value. See :ref:`amdgpu-target-id`. 435 436``-m[no-]<target-feature>`` 437 438 Target features not specified by the target ID are specified using a 439 separate option. These target features can have an ``on`` or ``off`` 440 value. ``on`` is specified by omitting the ``no-`` prefix, and 441 ``off`` is specified by including the ``no-`` prefix. The default 442 if not specified is ``off``. 443 444For example: 445 446``-mcpu=gfx908:xnack+`` 447 Enable the ``xnack`` feature. 448``-mcpu=gfx908:xnack-`` 449 Disable the ``xnack`` feature. 450``-mcumode`` 451 Enable the ``cumode`` feature. 452``-mno-cumode`` 453 Disable the ``cumode`` feature. 454 455 .. table:: AMDGPU Target Features 456 :name: amdgpu-target-features-table 457 458 =============== ============================ ================================================== 459 Target Feature Clang Option to Control Description 460 Name 461 =============== ============================ ================================================== 462 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 463 when generating code for kernels. When disabled 464 native WGP wavefront execution mode is used, 465 when enabled CU wavefront execution mode is used 466 (see :ref:`amdgpu-amdhsa-memory-model`). 467 468 sramecc - ``-mcpu`` If specified, generate code that can only be 469 - ``--offload-arch`` loaded and executed in a process that has a 470 matching setting for SRAMECC. 471 472 If not specified for code object V2 to V3, generate 473 code that can be loaded and executed in a process 474 with SRAMECC enabled. 475 476 If not specified for code object V4, generate 477 code that can be loaded and executed in a process 478 with either setting of SRAMECC. 479 480 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 481 work-groups are launched in threadgroup split mode. 482 When enabled the waves of a work-group may be 483 launched in different CUs. 484 485 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 486 generating code for kernels. When disabled 487 native wavefront size 32 is used, when enabled 488 wavefront size 64 is used. 489 490 xnack - ``-mcpu`` If specified, generate code that can only be 491 - ``--offload-arch`` loaded and executed in a process that has a 492 matching setting for XNACK replay. 493 494 If not specified for code object V2 to V3, generate 495 code that can be loaded and executed in a process 496 with XNACK replay enabled. 497 498 If not specified for code object V4, generate 499 code that can be loaded and executed in a process 500 with either setting of XNACK replay. 501 502 XNACK replay can be used for demand paging and 503 page migration. If enabled in the device, then if 504 a page fault occurs the code may execute 505 incorrectly unless generated with XNACK replay 506 enabled, or generated for code object V4 without 507 specifying XNACK replay. Executing code that was 508 generated with XNACK replay enabled, or generated 509 for code object V4 without specifying XNACK replay, 510 on a device that does not have XNACK replay 511 enabled will execute correctly but may be less 512 performant than code generated for XNACK replay 513 disabled. 514 =============== ============================ ================================================== 515 516.. _amdgpu-target-id: 517 518Target ID 519--------- 520 521AMDGPU supports target IDs. See `Clang Offload Bundler 522<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 523description. The AMDGPU target specific information is: 524 525**processor** 526 Is an AMDGPU processor or alternative processor name specified in 527 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 528 the primary processor and alternative processor names. The canonical form 529 target ID only allow the primary processor name. 530 531**target-feature** 532 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 533 is supported by the processor. The target features supported by each processor 534 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 535 a target ID are marked as being controlled by ``-mcpu`` and 536 ``--offload-arch``. Each target feature must appear at most once in a target 537 ID. The non-canonical form target ID allows the target features to be 538 specified in any order. The canonical form target ID requires the target 539 features to be specified in alphabetic order. 540 541.. _amdgpu-target-id-v2-v3: 542 543Code Object V2 to V3 Target ID 544~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 545 546The target ID syntax for code object V2 to V3 is the same as defined in `Clang 547Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 548when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 549directive and the bundle entry ID. In those cases it has the following BNF 550syntax: 551 552.. code:: 553 554 <target-id> ::== <processor> ( "+" <target-feature> )* 555 556Where a target feature is omitted if *Off* and present if *On* or *Any*. 557 558.. note:: 559 560 The code object V2 to V3 cannot represent *Any* and treats it the same as 561 *On*. 562 563.. _amdgpu-embedding-bundled-objects: 564 565Embedding Bundled Code Objects 566------------------------------ 567 568AMDGPU supports the HIP and OpenMP languages that perform code object embedding 569as described in `Clang Offload Bundler 570<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 571 572.. note:: 573 574 The target ID syntax used for code object V2 to V3 for a bundle entry ID 575 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 576 577.. _amdgpu-address-spaces: 578 579Address Spaces 580-------------- 581 582The AMDGPU architecture supports a number of memory address spaces. The address 583space names use the OpenCL standard names, with some additions. 584 585The AMDGPU address spaces correspond to target architecture specific LLVM 586address space numbers used in LLVM IR. 587 588The AMDGPU address spaces are described in 589:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 590supported for the ``amdgcn`` target. 591 592 .. table:: AMDGPU Address Spaces 593 :name: amdgpu-address-spaces-table 594 595 ================================= =============== =========== ================ ======= ============================ 596 .. 64-Bit Process Address Space 597 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 598 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 599 Space Number Name Name Size 600 ================================= =============== =========== ================ ======= ============================ 601 Generic 0 flat flat 64 0x0000000000000000 602 Global 1 global global 64 0x0000000000000000 603 Region 2 N/A GDS 32 *not implemented for AMDHSA* 604 Local 3 group LDS 32 0xFFFFFFFF 605 Constant 4 constant *same as global* 64 0x0000000000000000 606 Private 5 private scratch 32 0xFFFFFFFF 607 Constant 32-bit 6 *TODO* 0x00000000 608 Buffer Fat Pointer (experimental) 7 *TODO* 609 ================================= =============== =========== ================ ======= ============================ 610 611**Generic** 612 The generic address space is supported unless the *Target Properties* column 613 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 614 space*. 615 616 The generic address space uses the hardware flat address support for two fixed 617 ranges of virtual addresses (the private and local apertures), that are 618 outside the range of addressable global memory, to map from a flat address to 619 a private or local address. This uses FLAT instructions that can take a flat 620 address and access global, private (scratch), and group (LDS) memory depending 621 on if the address is within one of the aperture ranges. 622 623 Flat access to scratch requires hardware aperture setup and setup in the 624 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 625 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 626 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 627 628 To convert between a private or group address space address (termed a segment 629 address) and a flat address the base address of the corresponding aperture 630 can be used. For GFX7-GFX8 these are available in the 631 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 632 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 633 GFX9-GFX10 the aperture base addresses are directly available as inline 634 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 635 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 636 aligned to 2^32 which makes it easier to convert from flat to segment or 637 segment to flat. 638 639 A global address space address has the same value when used as a flat address 640 so no conversion is needed. 641 642**Global and Constant** 643 The global and constant address spaces both use global virtual addresses, 644 which are the same virtual address space used by the CPU. However, some 645 virtual addresses may only be accessible to the CPU, some only accessible 646 by the GPU, and some by both. 647 648 Using the constant address space indicates that the data will not change 649 during the execution of the kernel. This allows scalar read instructions to 650 be used. As the constant address space could only be modified on the host 651 side, a generic pointer loaded from the constant address space is safe to be 652 assumed as a global pointer since only the device global memory is visible 653 and managed on the host side. The vector and scalar L1 caches are invalidated 654 of volatile data before each kernel dispatch execution to allow constant 655 memory to change values between kernel dispatches. 656 657**Region** 658 The region address space uses the hardware Global Data Store (GDS). All 659 wavefronts executing on the same device will access the same memory for any 660 given region address. However, the same region address accessed by wavefronts 661 executing on different devices will access different memory. It is higher 662 performance than global memory. It is allocated by the runtime. The data 663 store (DS) instructions can be used to access it. 664 665**Local** 666 The local address space uses the hardware Local Data Store (LDS) which is 667 automatically allocated when the hardware creates the wavefronts of a 668 work-group, and freed when all the wavefronts of a work-group have 669 terminated. All wavefronts belonging to the same work-group will access the 670 same memory for any given local address. However, the same local address 671 accessed by wavefronts belonging to different work-groups will access 672 different memory. It is higher performance than global memory. The data store 673 (DS) instructions can be used to access it. 674 675**Private** 676 The private address space uses the hardware scratch memory support which 677 automatically allocates memory when it creates a wavefront and frees it when 678 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 679 given private address will be different to the memory accessed by another lane 680 of the same or different wavefront for the same private address. 681 682 If a kernel dispatch uses scratch, then the hardware allocates memory from a 683 pool of backing memory allocated by the runtime for each wavefront. The lanes 684 of the wavefront access this using dword (4 byte) interleaving. The mapping 685 used from private address to backing memory address is: 686 687 ``wavefront-scratch-base + 688 ((private-address / 4) * wavefront-size * 4) + 689 (wavefront-lane-id * 4) + (private-address % 4)`` 690 691 If each lane of a wavefront accesses the same private address, the 692 interleaving results in adjacent dwords being accessed and hence requires 693 fewer cache lines to be fetched. 694 695 There are different ways that the wavefront scratch base address is 696 determined by a wavefront (see 697 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 698 699 Scratch memory can be accessed in an interleaved manner using buffer 700 instructions with the scratch buffer descriptor and per wavefront scratch 701 offset, by the scratch instructions, or by flat instructions. Multi-dword 702 access is not supported except by flat and scratch instructions in 703 GFX9-GFX10. 704 705**Constant 32-bit** 706 *TODO* 707 708**Buffer Fat Pointer** 709 The buffer fat pointer is an experimental address space that is currently 710 unsupported in the backend. It exposes a non-integral pointer that is in 711 the future intended to support the modelling of 128-bit buffer descriptors 712 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 713 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 714 model the buffer descriptors used heavily in graphics workloads targeting 715 the backend. 716 717.. _amdgpu-memory-scopes: 718 719Memory Scopes 720------------- 721 722This section provides LLVM memory synchronization scopes supported by the AMDGPU 723backend memory model when the target triple OS is ``amdhsa`` (see 724:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 725 726The memory model supported is based on the HSA memory model [HSA]_ which is 727based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 728relation is transitive over the synchronizes-with relation independent of scope 729and synchronizes-with allows the memory scope instances to be inclusive (see 730table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 731 732This is different to the OpenCL [OpenCL]_ memory model which does not have scope 733inclusion and requires the memory scopes to exactly match. However, this 734is conservatively correct for OpenCL. 735 736 .. table:: AMDHSA LLVM Sync Scopes 737 :name: amdgpu-amdhsa-llvm-sync-scopes-table 738 739 ======================= =================================================== 740 LLVM Sync Scope Description 741 ======================= =================================================== 742 *none* The default: ``system``. 743 744 Synchronizes with, and participates in modification 745 and seq_cst total orderings with, other operations 746 (except image operations) for all address spaces 747 (except private, or generic that accesses private) 748 provided the other operation's sync scope is: 749 750 - ``system``. 751 - ``agent`` and executed by a thread on the same 752 agent. 753 - ``workgroup`` and executed by a thread in the 754 same work-group. 755 - ``wavefront`` and executed by a thread in the 756 same wavefront. 757 758 ``agent`` Synchronizes with, and participates in modification 759 and seq_cst total orderings with, other operations 760 (except image operations) for all address spaces 761 (except private, or generic that accesses private) 762 provided the other operation's sync scope is: 763 764 - ``system`` or ``agent`` and executed by a thread 765 on the same agent. 766 - ``workgroup`` and executed by a thread in the 767 same work-group. 768 - ``wavefront`` and executed by a thread in the 769 same wavefront. 770 771 ``workgroup`` Synchronizes with, and participates in modification 772 and seq_cst total orderings with, other operations 773 (except image operations) for all address spaces 774 (except private, or generic that accesses private) 775 provided the other operation's sync scope is: 776 777 - ``system``, ``agent`` or ``workgroup`` and 778 executed by a thread in the same work-group. 779 - ``wavefront`` and executed by a thread in the 780 same wavefront. 781 782 ``wavefront`` Synchronizes with, and participates in modification 783 and seq_cst total orderings with, other operations 784 (except image operations) for all address spaces 785 (except private, or generic that accesses private) 786 provided the other operation's sync scope is: 787 788 - ``system``, ``agent``, ``workgroup`` or 789 ``wavefront`` and executed by a thread in the 790 same wavefront. 791 792 ``singlethread`` Only synchronizes with and participates in 793 modification and seq_cst total orderings with, 794 other operations (except image operations) running 795 in the same thread for all address spaces (for 796 example, in signal handlers). 797 798 ``one-as`` Same as ``system`` but only synchronizes with other 799 operations within the same address space. 800 801 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 802 operations within the same address space. 803 804 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 805 other operations within the same address space. 806 807 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 808 other operations within the same address space. 809 810 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 811 other operations within the same address space. 812 ======================= =================================================== 813 814LLVM IR Intrinsics 815------------------ 816 817The AMDGPU backend implements the following LLVM IR intrinsics. 818 819*This section is WIP.* 820 821.. TODO:: 822 823 List AMDGPU intrinsics. 824 825LLVM IR Attributes 826------------------ 827 828The AMDGPU backend supports the following LLVM IR attributes. 829 830 .. table:: AMDGPU LLVM IR Attributes 831 :name: amdgpu-llvm-ir-attributes-table 832 833 ======================================= ========================================================== 834 LLVM Attribute Description 835 ======================================= ========================================================== 836 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 837 will be specified when the kernel is dispatched. Generated 838 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 839 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 840 argument block size for the implicit arguments. This 841 varies by OS and language (for OpenCL see 842 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 843 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 844 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 845 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 846 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 847 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 848 execution unit. Generated by the ``amdgpu_waves_per_eu`` 849 CLANG attribute [CLANG-ATTR]_. 850 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 851 mode register to be set on entry. Overrides the default for 852 the calling convention. 853 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 854 the mode register to be set on entry. Overrides the default 855 for the calling convention. 856 ======================================= ========================================================== 857 858.. _amdgpu-elf-code-object: 859 860ELF Code Object 861=============== 862 863The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 864can be linked by ``lld`` to produce a standard ELF shared code object which can 865be loaded and executed on an AMDGPU target. 866 867.. _amdgpu-elf-header: 868 869Header 870------ 871 872The AMDGPU backend uses the following ELF header: 873 874 .. table:: AMDGPU ELF Header 875 :name: amdgpu-elf-header-table 876 877 ========================== =============================== 878 Field Value 879 ========================== =============================== 880 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 881 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 882 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 883 - ``ELFOSABI_AMDGPU_HSA`` 884 - ``ELFOSABI_AMDGPU_PAL`` 885 - ``ELFOSABI_AMDGPU_MESA3D`` 886 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 887 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 888 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 889 - ``ELFABIVERSION_AMDGPU_PAL`` 890 - ``ELFABIVERSION_AMDGPU_MESA3D`` 891 ``e_type`` - ``ET_REL`` 892 - ``ET_DYN`` 893 ``e_machine`` ``EM_AMDGPU`` 894 ``e_entry`` 0 895 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 896 :ref:`amdgpu-elf-header-e_flags-table-v3`, 897 and :ref:`amdgpu-elf-header-e_flags-table-v4` 898 ========================== =============================== 899 900.. 901 902 .. table:: AMDGPU ELF Header Enumeration Values 903 :name: amdgpu-elf-header-enumeration-values-table 904 905 =============================== ===== 906 Name Value 907 =============================== ===== 908 ``EM_AMDGPU`` 224 909 ``ELFOSABI_NONE`` 0 910 ``ELFOSABI_AMDGPU_HSA`` 64 911 ``ELFOSABI_AMDGPU_PAL`` 65 912 ``ELFOSABI_AMDGPU_MESA3D`` 66 913 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 914 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 915 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 916 ``ELFABIVERSION_AMDGPU_PAL`` 0 917 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 918 =============================== ===== 919 920``e_ident[EI_CLASS]`` 921 The ELF class is: 922 923 * ``ELFCLASS32`` for ``r600`` architecture. 924 925 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 926 process address space applications. 927 928``e_ident[EI_DATA]`` 929 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 930 931``e_ident[EI_OSABI]`` 932 One of the following AMDGPU target architecture specific OS ABIs 933 (see :ref:`amdgpu-os`): 934 935 * ``ELFOSABI_NONE`` for *unknown* OS. 936 937 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 938 939 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 940 941 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 942 943``e_ident[EI_ABIVERSION]`` 944 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 945 object conforms: 946 947 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 948 runtime ABI for code object V2. Specify using the Clang option 949 ``-mcode-object-version=2``. 950 951 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 952 runtime ABI for code object V3. Specify using the Clang option 953 ``-mcode-object-version=3``. This is the default code object 954 version if not specified. 955 956 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 957 runtime ABI for code object V4. Specify using the Clang option 958 ``-mcode-object-version=4``. 959 960 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 961 runtime ABI. 962 963 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 964 3D runtime ABI. 965 966``e_type`` 967 Can be one of the following values: 968 969 970 ``ET_REL`` 971 The type produced by the AMDGPU backend compiler as it is relocatable code 972 object. 973 974 ``ET_DYN`` 975 The type produced by the linker as it is a shared code object. 976 977 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 978 979``e_machine`` 980 The value ``EM_AMDGPU`` is used for the machine for all processors supported 981 by the ``r600`` and ``amdgcn`` architectures (see 982 :ref:`amdgpu-processor-table`). The specific processor is specified in the 983 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 984 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 985 ``e_flags`` for code object V3 to V4 (see 986 :ref:`amdgpu-elf-header-e_flags-table-v3` and 987 :ref:`amdgpu-elf-header-e_flags-table-v4`). 988 989``e_entry`` 990 The entry point is 0 as the entry points for individual kernels must be 991 selected in order to invoke them through AQL packets. 992 993``e_flags`` 994 The AMDGPU backend uses the following ELF header flags: 995 996 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 997 :name: amdgpu-elf-header-e_flags-v2-table 998 999 ===================================== ===== ============================= 1000 Name Value Description 1001 ===================================== ===== ============================= 1002 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1003 target feature is 1004 enabled for all code 1005 contained in the code object. 1006 If the processor 1007 does not support the 1008 ``xnack`` target 1009 feature then must 1010 be 0. 1011 See 1012 :ref:`amdgpu-target-features`. 1013 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1014 handler is enabled for all 1015 code contained in the code 1016 object. If the processor 1017 does not support a trap 1018 handler then must be 0. 1019 See 1020 :ref:`amdgpu-target-features`. 1021 ===================================== ===== ============================= 1022 1023 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1024 :name: amdgpu-elf-header-e_flags-table-v3 1025 1026 ================================= ===== ============================= 1027 Name Value Description 1028 ================================= ===== ============================= 1029 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1030 mask for 1031 ``EF_AMDGPU_MACH_xxx`` values 1032 defined in 1033 :ref:`amdgpu-ef-amdgpu-mach-table`. 1034 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1035 target feature is 1036 enabled for all code 1037 contained in the code object. 1038 If the processor 1039 does not support the 1040 ``xnack`` target 1041 feature then must 1042 be 0. 1043 See 1044 :ref:`amdgpu-target-features`. 1045 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1046 target feature is 1047 enabled for all code 1048 contained in the code object. 1049 If the processor 1050 does not support the 1051 ``sramecc`` target 1052 feature then must 1053 be 0. 1054 See 1055 :ref:`amdgpu-target-features`. 1056 ================================= ===== ============================= 1057 1058 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1059 :name: amdgpu-elf-header-e_flags-table-v4 1060 1061 ============================================ ===== =================================== 1062 Name Value Description 1063 ============================================ ===== =================================== 1064 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1065 mask for 1066 ``EF_AMDGPU_MACH_xxx`` values 1067 defined in 1068 :ref:`amdgpu-ef-amdgpu-mach-table`. 1069 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1070 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1071 values. 1072 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1073 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1074 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1075 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1076 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1077 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1078 values. 1079 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1080 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1081 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1082 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1083 ============================================ ===== =================================== 1084 1085 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1086 :name: amdgpu-ef-amdgpu-mach-table 1087 1088 ==================================== ========== ============================= 1089 Name Value Description (see 1090 :ref:`amdgpu-processor-table`) 1091 ==================================== ========== ============================= 1092 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1093 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1094 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1095 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1096 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1097 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1098 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1099 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1100 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1101 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1102 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1103 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1104 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1105 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1106 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1107 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1108 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1109 *reserved* 0x011 - Reserved for ``r600`` 1110 0x01f architecture processors. 1111 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1112 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1113 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1114 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1115 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1116 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1117 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1118 *reserved* 0x027 Reserved. 1119 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1120 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1121 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1122 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1123 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1124 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1125 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1126 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1127 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1128 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1129 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1130 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1131 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1132 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1133 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1134 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1135 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1136 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1137 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1138 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1139 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1140 *reserved* 0x03d Reserved. 1141 *reserved* 0x03e Reserved. 1142 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1143 *reserved* 0x040 Reserved. 1144 *reserved* 0x041 Reserved. 1145 ==================================== ========== ============================= 1146 1147Sections 1148-------- 1149 1150An AMDGPU target ELF code object has the standard ELF sections which include: 1151 1152 .. table:: AMDGPU ELF Sections 1153 :name: amdgpu-elf-sections-table 1154 1155 ================== ================ ================================= 1156 Name Type Attributes 1157 ================== ================ ================================= 1158 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1159 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1160 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1161 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1162 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1163 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1164 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1165 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1166 ``.note`` ``SHT_NOTE`` *none* 1167 ``.rela``\ *name* ``SHT_RELA`` *none* 1168 ``.rela.dyn`` ``SHT_RELA`` *none* 1169 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1170 ``.shstrtab`` ``SHT_STRTAB`` *none* 1171 ``.strtab`` ``SHT_STRTAB`` *none* 1172 ``.symtab`` ``SHT_SYMTAB`` *none* 1173 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1174 ================== ================ ================================= 1175 1176These sections have their standard meanings (see [ELF]_) and are only generated 1177if needed. 1178 1179``.debug``\ *\** 1180 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1181 information on the DWARF produced by the AMDGPU backend. 1182 1183``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1184 The standard sections used by a dynamic loader. 1185 1186``.note`` 1187 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1188 backend. 1189 1190``.rela``\ *name*, ``.rela.dyn`` 1191 For relocatable code objects, *name* is the name of the section that the 1192 relocation records apply. For example, ``.rela.text`` is the section name for 1193 relocation records associated with the ``.text`` section. 1194 1195 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1196 records from each of the relocatable code object's ``.rela``\ *name* sections. 1197 1198 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1199 the AMDGPU backend. 1200 1201``.text`` 1202 The executable machine code for the kernels and functions they call. Generated 1203 as position independent code. See :ref:`amdgpu-code-conventions` for 1204 information on conventions used in the isa generation. 1205 1206.. _amdgpu-note-records: 1207 1208Note Records 1209------------ 1210 1211The AMDGPU backend code object contains ELF note records in the ``.note`` 1212section. The set of generated notes and their semantics depend on the code 1213object version; see :ref:`amdgpu-note-records-v2` and 1214:ref:`amdgpu-note-records-v3-v4`. 1215 1216As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1217must be generated after the ``name`` field to ensure the ``desc`` field is 4 1218byte aligned. In addition, minimal zero-byte padding must be generated to 1219ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1220field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1221alignment. 1222 1223.. _amdgpu-note-records-v2: 1224 1225Code Object V2 Note Records 1226~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1227 1228.. warning:: 1229 Code object V2 is not the default code object version emitted by 1230 this version of LLVM. 1231 1232The AMDGPU backend code object uses the following ELF note record in the 1233``.note`` section when compiling for code object V2. 1234 1235The note record vendor field is "AMD". 1236 1237Additional note records may be present, but any which are not documented here 1238are deprecated and should not be used. 1239 1240 .. table:: AMDGPU Code Object V2 ELF Note Records 1241 :name: amdgpu-elf-note-records-v2-table 1242 1243 ===== ===================================== ====================================== 1244 Name Type Description 1245 ===== ===================================== ====================================== 1246 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1247 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1248 Finalizer and not the LLVM compiler. 1249 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1250 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1251 YAML [YAML]_ textual format. 1252 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1253 ===== ===================================== ====================================== 1254 1255.. 1256 1257 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1258 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1259 1260 ===================================== ===== 1261 Name Value 1262 ===================================== ===== 1263 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1264 ``NT_AMD_HSA_HSAIL`` 2 1265 ``NT_AMD_HSA_ISA_VERSION`` 3 1266 *reserved* 4-9 1267 ``NT_AMD_HSA_METADATA`` 10 1268 ``NT_AMD_HSA_ISA_NAME`` 11 1269 ===================================== ===== 1270 1271``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1272 Specifies the code object version number. The description field has the 1273 following layout: 1274 1275 .. code:: 1276 1277 struct amdgpu_hsa_note_code_object_version_s { 1278 uint32_t major_version; 1279 uint32_t minor_version; 1280 }; 1281 1282 The ``major_version`` has a value less than or equal to 2. 1283 1284``NT_AMD_HSA_HSAIL`` 1285 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1286 field has the following layout: 1287 1288 .. code:: 1289 1290 struct amdgpu_hsa_note_hsail_s { 1291 uint32_t hsail_major_version; 1292 uint32_t hsail_minor_version; 1293 uint8_t profile; 1294 uint8_t machine_model; 1295 uint8_t default_float_round; 1296 }; 1297 1298``NT_AMD_HSA_ISA_VERSION`` 1299 Specifies the target ISA version. The description field has the following layout: 1300 1301 .. code:: 1302 1303 struct amdgpu_hsa_note_isa_s { 1304 uint16_t vendor_name_size; 1305 uint16_t architecture_name_size; 1306 uint32_t major; 1307 uint32_t minor; 1308 uint32_t stepping; 1309 char vendor_and_architecture_name[1]; 1310 }; 1311 1312 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1313 vendor and architecture names respectively, including the NUL character. 1314 1315 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1316 vendor, immediately followed by the NUL terminated string for the 1317 architecture. 1318 1319 This note record is used by the HSA runtime loader. 1320 1321 Code object V2 only supports a limited number of processors and has fixed 1322 settings for target features. See 1323 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1324 processors and the corresponding target ID. In the table the note record ISA 1325 name is a concatenation of the vendor name, architecture name, major, minor, 1326 and stepping separated by a ":". 1327 1328 The target ID column shows the processor name and fixed target features used 1329 by the LLVM compiler. The LLVM compiler does not generate a 1330 ``NT_AMD_HSA_HSAIL`` note record. 1331 1332 A code object generated by the Finalizer also uses code object V2 and always 1333 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1334 ``sramecc`` target feature is as shown in 1335 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1336 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1337 bit. 1338 1339``NT_AMD_HSA_ISA_NAME`` 1340 Specifies the target ISA name as a non-NUL terminated string. 1341 1342 This note record is not used by the HSA runtime loader. 1343 1344 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1345 V2's limited support of processors and fixed settings for target features. 1346 1347 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1348 from the string to the corresponding target ID. If the ``xnack`` target 1349 feature is supported and enabled, the string produced by the LLVM compiler 1350 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1351 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1352 1353``NT_AMD_HSA_METADATA`` 1354 Specifies extensible metadata associated with the code objects executed on HSA 1355 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1356 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1357 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1358 metadata string. 1359 1360 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1361 :name: amdgpu-elf-note-record-supported_processors-v2-table 1362 1363 ===================== ========================== 1364 Note Record ISA Name Target ID 1365 ===================== ========================== 1366 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1367 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1368 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1369 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1370 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1371 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1372 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1373 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1374 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1375 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1376 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1377 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1378 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1379 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1380 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1381 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1382 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1383 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1384 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1385 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1386 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1387 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1388 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1389 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1390 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1391 ===================== ========================== 1392 1393.. _amdgpu-note-records-v3-v4: 1394 1395Code Object V3 to V4 Note Records 1396~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1397 1398The AMDGPU backend code object uses the following ELF note record in the 1399``.note`` section when compiling for code object V3 to V4. 1400 1401The note record vendor field is "AMDGPU". 1402 1403Additional note records may be present, but any which are not documented here 1404are deprecated and should not be used. 1405 1406 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1407 :name: amdgpu-elf-note-records-table-v3-v4 1408 1409 ======== ============================== ====================================== 1410 Name Type Description 1411 ======== ============================== ====================================== 1412 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1413 binary format. 1414 ======== ============================== ====================================== 1415 1416.. 1417 1418 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1419 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1420 1421 ============================== ===== 1422 Name Value 1423 ============================== ===== 1424 *reserved* 0-31 1425 ``NT_AMDGPU_METADATA`` 32 1426 ============================== ===== 1427 1428``NT_AMDGPU_METADATA`` 1429 Specifies extensible metadata associated with an AMDGPU code object. It is 1430 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1431 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1432 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1433 ``amdhsa`` OS. 1434 1435.. _amdgpu-symbols: 1436 1437Symbols 1438------- 1439 1440Symbols include the following: 1441 1442 .. table:: AMDGPU ELF Symbols 1443 :name: amdgpu-elf-symbols-table 1444 1445 ===================== ================== ================ ================== 1446 Name Type Section Description 1447 ===================== ================== ================ ================== 1448 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1449 - ``.rodata`` 1450 - ``.bss`` 1451 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1452 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1453 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1454 ===================== ================== ================ ================== 1455 1456Global variable 1457 Global variables both used and defined by the compilation unit. 1458 1459 If the symbol is defined in the compilation unit then it is allocated in the 1460 appropriate section according to if it has initialized data or is readonly. 1461 1462 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1463 will resolve relocations using the definition provided by another code object 1464 or explicitly defined by the runtime. 1465 1466 If the symbol resides in local/group memory (LDS) then its section is the 1467 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1468 ``st_value`` field describes alignment requirements as it does for common 1469 symbols. 1470 1471 .. TODO:: 1472 1473 Add description of linked shared object symbols. Seems undefined symbols 1474 are marked as STT_NOTYPE. 1475 1476Kernel descriptor 1477 Every HSA kernel has an associated kernel descriptor. It is the address of the 1478 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1479 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1480 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1481 1482Kernel entry point 1483 Every HSA kernel also has a symbol for its machine code entry point. 1484 1485.. _amdgpu-relocation-records: 1486 1487Relocation Records 1488------------------ 1489 1490AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1491relocatable fields are: 1492 1493``word32`` 1494 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1495 alignment. These values use the same byte order as other word values in the 1496 AMDGPU architecture. 1497 1498``word64`` 1499 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1500 alignment. These values use the same byte order as other word values in the 1501 AMDGPU architecture. 1502 1503Following notations are used for specifying relocation calculations: 1504 1505**A** 1506 Represents the addend used to compute the value of the relocatable field. 1507 1508**G** 1509 Represents the offset into the global offset table at which the relocation 1510 entry's symbol will reside during execution. 1511 1512**GOT** 1513 Represents the address of the global offset table. 1514 1515**P** 1516 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1517 of the storage unit being relocated (computed using ``r_offset``). 1518 1519**S** 1520 Represents the value of the symbol whose index resides in the relocation 1521 entry. Relocations not using this must specify a symbol index of 1522 ``STN_UNDEF``. 1523 1524**B** 1525 Represents the base address of a loaded executable or shared object which is 1526 the difference between the ELF address and the actual load address. 1527 Relocations using this are only valid in executable or shared objects. 1528 1529The following relocation types are supported: 1530 1531 .. table:: AMDGPU ELF Relocation Records 1532 :name: amdgpu-elf-relocation-records-table 1533 1534 ========================== ======= ===== ========== ============================== 1535 Relocation Type Kind Value Field Calculation 1536 ========================== ======= ===== ========== ============================== 1537 ``R_AMDGPU_NONE`` 0 *none* *none* 1538 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1539 Dynamic 1540 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1541 Dynamic 1542 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1543 Dynamic 1544 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1545 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1546 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1547 Dynamic 1548 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1549 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1550 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1551 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1552 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1553 *reserved* 12 1554 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1555 ========================== ======= ===== ========== ============================== 1556 1557``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1558the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1559 1560There is no current OS loader support for 32-bit programs and so 1561``R_AMDGPU_ABS32`` is not used. 1562 1563.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1564 1565Loaded Code Object Path Uniform Resource Identifier (URI) 1566--------------------------------------------------------- 1567 1568The AMD GPU code object loader represents the path of the ELF shared object from 1569which the code object was loaded as a textual Unifom Resource Identifier (URI). 1570Note that the code object is the in memory loaded relocated form of the ELF 1571shared object. Multiple code objects may be loaded at different memory 1572addresses in the same process from the same ELF shared object. 1573 1574The loaded code object path URI syntax is defined by the following BNF syntax: 1575 1576.. code:: 1577 1578 code_object_uri ::== file_uri | memory_uri 1579 file_uri ::== "file://" file_path [ range_specifier ] 1580 memory_uri ::== "memory://" process_id range_specifier 1581 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1582 file_path ::== URI_ENCODED_OS_FILE_PATH 1583 process_id ::== DECIMAL_NUMBER 1584 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1585 1586**number** 1587 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1588 and octal values by "0". 1589 1590**file_path** 1591 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1592 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1593 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1594 the path are separated by "/". 1595 1596**offset** 1597 Is a 0-based byte offset to the start of the code object. For a file URI, it 1598 is from the start of the file specified by the ``file_path``, and if omitted 1599 defaults to 0. For a memory URI, it is the memory address and is required. 1600 1601**size** 1602 Is the number of bytes in the code object. For a file URI, if omitted it 1603 defaults to the size of the file. It is required for a memory URI. 1604 1605**process_id** 1606 Is the identity of the process owning the memory. For Linux it is the C 1607 unsigned integral decimal literal for the process ID (PID). 1608 1609For example: 1610 1611.. code:: 1612 1613 file:///dir1/dir2/file1 1614 file:///dir3/dir4/file2#offset=0x2000&size=3000 1615 memory://1234#offset=0x20000&size=3000 1616 1617.. _amdgpu-dwarf-debug-information: 1618 1619DWARF Debug Information 1620======================= 1621 1622.. warning:: 1623 1624 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1625 is not currently fully implemented and is subject to change. 1626 1627AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1628:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1629object executable code and data to the source language constructs. It can be 1630used by tools such as debuggers and profilers. It uses features defined in 1631:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1632DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1633 1634This section defines the AMDGPU target architecture specific DWARF mappings. 1635 1636.. _amdgpu-dwarf-register-identifier: 1637 1638Register Identifier 1639------------------- 1640 1641This section defines the AMDGPU target architecture register numbers used in 1642DWARF operation expressions (see DWARF Version 5 section 2.5 and 1643:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1644instructions (see DWARF Version 5 section 6.4 and 1645:ref:`amdgpu-dwarf-call-frame-information`). 1646 1647A single code object can contain code for kernels that have different wavefront 1648sizes. The vector registers and some scalar registers are based on the wavefront 1649size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1650simplifies the consumer of the DWARF so that each register has a fixed size, 1651rather than being dynamic according to the wavefront size mode. Similarly, 1652distinct DWARF registers are defined for those registers that vary in size 1653according to the process address size. This allows a consumer to treat a 1654specific AMDGPU processor as a single architecture regardless of how it is 1655configured at run time. The compiler explicitly specifies the DWARF registers 1656that match the mode in which the code it is generating will be executed. 1657 1658DWARF registers are encoded as numbers, which are mapped to architecture 1659registers. The mapping for AMDGPU is defined in 1660:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1661mapping. 1662 1663.. table:: AMDGPU DWARF Register Mapping 1664 :name: amdgpu-dwarf-register-mapping-table 1665 1666 ============== ================= ======== ================================== 1667 DWARF Register AMDGPU Register Bit Size Description 1668 ============== ================= ======== ================================== 1669 0 PC_32 32 Program Counter (PC) when 1670 executing in a 32-bit process 1671 address space. Used in the CFI to 1672 describe the PC of the calling 1673 frame. 1674 1 EXEC_MASK_32 32 Execution Mask Register when 1675 executing in wavefront 32 mode. 1676 2-15 *Reserved* *Reserved for highly accessed 1677 registers using DWARF shortcut.* 1678 16 PC_64 64 Program Counter (PC) when 1679 executing in a 64-bit process 1680 address space. Used in the CFI to 1681 describe the PC of the calling 1682 frame. 1683 17 EXEC_MASK_64 64 Execution Mask Register when 1684 executing in wavefront 64 mode. 1685 18-31 *Reserved* *Reserved for highly accessed 1686 registers using DWARF shortcut.* 1687 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1688 Registers. 1689 96-127 *Reserved* *Reserved for frequently accessed 1690 registers using DWARF 1-byte ULEB.* 1691 128 STATUS 32 Status Register. 1692 129-511 *Reserved* *Reserved for future Scalar 1693 Architectural Registers.* 1694 512 VCC_32 32 Vector Condition Code Register 1695 when executing in wavefront 32 1696 mode. 1697 513-1023 *Reserved* *Reserved for future Vector 1698 Architectural Registers when 1699 executing in wavefront 32 mode.* 1700 768 VCC_64 64 Vector Condition Code Register 1701 when executing in wavefront 64 1702 mode. 1703 769-1023 *Reserved* *Reserved for future Vector 1704 Architectural Registers when 1705 executing in wavefront 64 mode.* 1706 1024-1087 *Reserved* *Reserved for padding.* 1707 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1708 1130-1535 *Reserved* *Reserved for future Scalar 1709 General Purpose Registers.* 1710 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1711 when executing in wavefront 32 1712 mode. 1713 1792-2047 *Reserved* *Reserved for future Vector 1714 General Purpose Registers when 1715 executing in wavefront 32 mode.* 1716 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1717 when executing in wavefront 32 1718 mode. 1719 2304-2559 *Reserved* *Reserved for future Vector 1720 Accumulation Registers when 1721 executing in wavefront 32 mode.* 1722 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1723 when executing in wavefront 64 1724 mode. 1725 2816-3071 *Reserved* *Reserved for future Vector 1726 General Purpose Registers when 1727 executing in wavefront 64 mode.* 1728 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1729 when executing in wavefront 64 1730 mode. 1731 3328-3583 *Reserved* *Reserved for future Vector 1732 Accumulation Registers when 1733 executing in wavefront 64 mode.* 1734 ============== ================= ======== ================================== 1735 1736The vector registers are represented as the full size for the wavefront. They 1737are organized as consecutive dwords (32-bits), one per lane, with the dword at 1738the least significant bit position corresponding to lane 0 and so forth. DWARF 1739location expressions involving the ``DW_OP_LLVM_offset`` and 1740``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1741register corresponding to the lane that is executing the current thread of 1742execution in languages that are implemented using a SIMD or SIMT execution 1743model. 1744 1745If the wavefront size is 32 lanes then the wavefront 32 mode register 1746definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1747mode register definitions are used. Some AMDGPU targets support executing in 1748both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1749to the wavefront mode of the generated code will be used. 1750 1751If code is generated to execute in a 32-bit process address space, then the 175232-bit process address space register definitions are used. If code is generated 1753to execute in a 64-bit process address space, then the 64-bit process address 1754space register definitions are used. The ``amdgcn`` target only supports the 175564-bit process address space. 1756 1757.. _amdgpu-dwarf-address-class-identifier: 1758 1759Address Class Identifier 1760------------------------ 1761 1762The DWARF address class represents the source language memory space. See DWARF 1763Version 5 section 2.12 which is updated by the *DWARF Extensions For 1764Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1765 1766The DWARF address class mapping used for AMDGPU is defined in 1767:ref:`amdgpu-dwarf-address-class-mapping-table`. 1768 1769.. table:: AMDGPU DWARF Address Class Mapping 1770 :name: amdgpu-dwarf-address-class-mapping-table 1771 1772 ========================= ====== ================= 1773 DWARF AMDGPU 1774 -------------------------------- ----------------- 1775 Address Class Name Value Address Space 1776 ========================= ====== ================= 1777 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1778 ``DW_ADDR_LLVM_global`` 0x0001 Global 1779 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1780 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1781 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1782 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1783 ========================= ====== ================= 1784 1785The DWARF address class values defined in the *DWARF Extensions For 1786Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1787 1788In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1789available for use for the AMD extension for access to the hardware GDS memory 1790which is scratchpad memory allocated per device. 1791 1792For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1793address class of ``DW_ADDR_none`` is used. 1794 1795See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1796mapping of DWARF address classes to DWARF address spaces, including address size 1797and NULL value. 1798 1799.. _amdgpu-dwarf-address-space-identifier: 1800 1801Address Space Identifier 1802------------------------ 1803 1804DWARF address spaces correspond to target architecture specific linear 1805addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1806For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1807 1808The DWARF address space mapping used for AMDGPU is defined in 1809:ref:`amdgpu-dwarf-address-space-mapping-table`. 1810 1811.. table:: AMDGPU DWARF Address Space Mapping 1812 :name: amdgpu-dwarf-address-space-mapping-table 1813 1814 ======================================= ===== ======= ======== ================= ======================= 1815 DWARF AMDGPU Notes 1816 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1817 Address Space Name Value Address Bit Size Address Space 1818 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1819 .. 64-bit 32-bit 1820 process process 1821 address address 1822 space space 1823 ======================================= ===== ======= ======== ================= ======================= 1824 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1825 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1826 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1827 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1828 *Reserved* 0x04 1829 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1830 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1831 ======================================= ===== ======= ======== ================= ======================= 1832 1833See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1834including address size and NULL value. 1835 1836The ``DW_ASPACE_none`` address space is the default target architecture address 1837space used in DWARF operations that do not specify an address space. It 1838therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1839related operations can refer to addresses in the program code. 1840 1841The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1842specify the flat address space. If the address corresponds to an address in the 1843local address space, then it corresponds to the wavefront that is executing the 1844focused thread of execution. If the address corresponds to an address in the 1845private address space, then it corresponds to the lane that is executing the 1846focused thread of execution for languages that are implemented using a SIMD or 1847SIMT execution model. 1848 1849.. note:: 1850 1851 CUDA-like languages such as HIP that do not have address spaces in the 1852 language type system, but do allow variables to be allocated in different 1853 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1854 address space in the DWARF expression operations as the default address space 1855 is the global address space. 1856 1857The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1858specify the local address space corresponding to the wavefront that is executing 1859the focused thread of execution. 1860 1861The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1862to specify the private address space corresponding to the lane that is executing 1863the focused thread of execution for languages that are implemented using a SIMD 1864or SIMT execution model. 1865 1866The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1867to specify the unswizzled private address space corresponding to the wavefront 1868that is executing the focused thread of execution. The wavefront view of private 1869memory is the per wavefront unswizzled backing memory layout defined in 1870:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1871location for the backing memory of the wavefront (namely the address is not 1872offset by ``wavefront-scratch-base``). The following formula can be used to 1873convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1874``DW_ASPACE_AMDGPU_private_wave`` address: 1875 1876:: 1877 1878 private-address-wavefront = 1879 ((private-address-lane / 4) * wavefront-size * 4) + 1880 (wavefront-lane-id * 4) + (private-address-lane % 4) 1881 1882If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1883of the dwords for each lane starting with lane 0 is required, then this 1884simplifies to: 1885 1886:: 1887 1888 private-address-wavefront = 1889 private-address-lane * wavefront-size 1890 1891A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1892complete spilled vector register back into a complete vector register in the 1893CFI. The frame pointer can be a private lane address which is dword aligned, 1894which can be shifted to multiply by the wavefront size, and then used to form a 1895private wavefront address that gives a location for a contiguous set of dwords, 1896one per lane, where the vector register dwords are spilled. The compiler knows 1897the wavefront size since it generates the code. Note that the type of the 1898address may have to be converted as the size of a 1899``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1900``DW_ASPACE_AMDGPU_private_wave`` address. 1901 1902.. _amdgpu-dwarf-lane-identifier: 1903 1904Lane identifier 1905--------------- 1906 1907DWARF lane identifies specify a target architecture lane position for hardware 1908that executes in a SIMD or SIMT manner, and on which a source language maps its 1909threads of execution onto those lanes. The DWARF lane identifier is pushed by 1910the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1911section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1912section :ref:`amdgpu-dwarf-operation-expressions`. 1913 1914For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1915wavefront. It is numbered from 0 to the wavefront size minus 1. 1916 1917Operation Expressions 1918--------------------- 1919 1920DWARF expressions are used to compute program values and the locations of 1921program objects. See DWARF Version 5 section 2.5 and 1922:ref:`amdgpu-dwarf-operation-expressions`. 1923 1924DWARF location descriptions describe how to access storage which includes memory 1925and registers. When accessing storage on AMDGPU, bytes are ordered with least 1926significant bytes first, and bits are ordered within bytes with least 1927significant bits first. 1928 1929For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1930unwinding vector registers that are spilled under the execution mask to memory: 1931the zero-single location description is the vector register, and the one-single 1932location description is the spilled memory location description. The 1933``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1934memory location description. 1935 1936In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1937``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1938controlled by the execution mask. An undefined location description together 1939with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1940to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1941 1942Debugger Information Entry Attributes 1943------------------------------------- 1944 1945This section describes how certain debugger information entry attributes are 1946used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1947by *DWARF Extensions For Heterogeneous Debugging* section 1948:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1949 1950.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1951 1952``DW_AT_LLVM_lane_pc`` 1953~~~~~~~~~~~~~~~~~~~~~~ 1954 1955For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1956location of the separate lanes of a SIMT thread. 1957 1958If the lane is an active lane then this will be the same as the current program 1959location. 1960 1961If the lane is inactive, but was active on entry to the subprogram, then this is 1962the program location in the subprogram at which execution of the lane is 1963conceptual positioned. 1964 1965If the lane was not active on entry to the subprogram, then this will be the 1966undefined location. A client debugger can check if the lane is part of a valid 1967work-group by checking that the lane is in the range of the associated 1968work-group within the grid, accounting for partial work-groups. If it is not, 1969then the debugger can omit any information for the lane. Otherwise, the debugger 1970may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1971calling subprogram until it finds a non-undefined location. Conceptually the 1972lane only has the call frames that it has a non-undefined 1973``DW_AT_LLVM_lane_pc``. 1974 1975The following example illustrates how the AMDGPU backend can generate a DWARF 1976location list expression for the nested ``IF/THEN/ELSE`` structures of the 1977following subprogram pseudo code for a target with 64 lanes per wavefront. 1978 1979.. code:: 1980 :number-lines: 1981 1982 SUBPROGRAM X 1983 BEGIN 1984 a; 1985 IF (c1) THEN 1986 b; 1987 IF (c2) THEN 1988 c; 1989 ELSE 1990 d; 1991 ENDIF 1992 e; 1993 ELSE 1994 f; 1995 ENDIF 1996 g; 1997 END 1998 1999The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2000execution mask (``EXEC``) to linearize the control flow. The condition is 2001evaluated to make a mask of the lanes for which the condition evaluates to true. 2002First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2003logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2004``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2005the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2006region the ``EXEC`` mask is restored to the value it had at the beginning of the 2007region. This is shown below. Other approaches are possible, but the basic 2008concept is the same. 2009 2010.. code:: 2011 :number-lines: 2012 2013 $lex_start: 2014 a; 2015 %1 = EXEC 2016 %2 = c1 2017 $lex_1_start: 2018 EXEC = %1 & %2 2019 $if_1_then: 2020 b; 2021 %3 = EXEC 2022 %4 = c2 2023 $lex_1_1_start: 2024 EXEC = %3 & %4 2025 $lex_1_1_then: 2026 c; 2027 EXEC = ~EXEC & %3 2028 $lex_1_1_else: 2029 d; 2030 EXEC = %3 2031 $lex_1_1_end: 2032 e; 2033 EXEC = ~EXEC & %1 2034 $lex_1_else: 2035 f; 2036 EXEC = %1 2037 $lex_1_end: 2038 g; 2039 $lex_end: 2040 2041To create the DWARF location list expression that defines the location 2042description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2043pseudo instruction can be used to annotate the linearized control flow. This can 2044be done by defining an artificial variable for the lane PC. The DWARF location 2045list expression created for it is used as the value of the 2046``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2047 2048A DWARF procedure is defined for each well nested structured control flow region 2049which provides the conceptual lane program location for a lane if it is not 2050active (namely it is divergent). The DWARF operation expression for each region 2051conceptually inherits the value of the immediately enclosing region and modifies 2052it according to the semantics of the region. 2053 2054For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2055the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2056region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2057region since the ``THEN`` region has completed. 2058 2059The lane PC artificial variable is assigned at each region transition. It uses 2060the immediately enclosing region's DWARF procedure to compute the program 2061location for each lane assuming they are divergent, and then modifies the result 2062by inserting the current program location for each lane that the ``EXEC`` mask 2063indicates is active. 2064 2065By having separate DWARF procedures for each region, they can be reused to 2066define the value for any nested region. This reduces the total size of the DWARF 2067operation expressions. 2068 2069The following provides an example using pseudo LLVM MIR. 2070 2071.. code:: 2072 :number-lines: 2073 2074 $lex_start: 2075 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2076 DW_AT_name = "__uint64"; 2077 DW_AT_byte_size = 8; 2078 DW_AT_encoding = DW_ATE_unsigned; 2079 ]; 2080 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2081 DW_AT_name = "__active_lane_pc"; 2082 DW_AT_location = [ 2083 DW_OP_regx PC; 2084 DW_OP_LLVM_extend 64, 64; 2085 DW_OP_regval_type EXEC, %uint_64; 2086 DW_OP_LLVM_select_bit_piece 64, 64; 2087 ]; 2088 ]; 2089 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2090 DW_AT_name = "__divergent_lane_pc"; 2091 DW_AT_location = [ 2092 DW_OP_LLVM_undefined; 2093 DW_OP_LLVM_extend 64, 64; 2094 ]; 2095 ]; 2096 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2097 DW_OP_call_ref %__divergent_lane_pc; 2098 DW_OP_call_ref %__active_lane_pc; 2099 ]; 2100 a; 2101 %1 = EXEC; 2102 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2103 %2 = c1; 2104 $lex_1_start: 2105 EXEC = %1 & %2; 2106 $lex_1_then: 2107 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2108 DW_AT_name = "__divergent_lane_pc_1_then"; 2109 DW_AT_location = DIExpression[ 2110 DW_OP_call_ref %__divergent_lane_pc; 2111 DW_OP_addrx &lex_1_start; 2112 DW_OP_stack_value; 2113 DW_OP_LLVM_extend 64, 64; 2114 DW_OP_call_ref %__lex_1_save_exec; 2115 DW_OP_deref_type 64, %__uint_64; 2116 DW_OP_LLVM_select_bit_piece 64, 64; 2117 ]; 2118 ]; 2119 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2120 DW_OP_call_ref %__divergent_lane_pc_1_then; 2121 DW_OP_call_ref %__active_lane_pc; 2122 ]; 2123 b; 2124 %3 = EXEC; 2125 DBG_VALUE %3, %__lex_1_1_save_exec; 2126 %4 = c2; 2127 $lex_1_1_start: 2128 EXEC = %3 & %4; 2129 $lex_1_1_then: 2130 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2131 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2132 DW_AT_location = DIExpression[ 2133 DW_OP_call_ref %__divergent_lane_pc_1_then; 2134 DW_OP_addrx &lex_1_1_start; 2135 DW_OP_stack_value; 2136 DW_OP_LLVM_extend 64, 64; 2137 DW_OP_call_ref %__lex_1_1_save_exec; 2138 DW_OP_deref_type 64, %__uint_64; 2139 DW_OP_LLVM_select_bit_piece 64, 64; 2140 ]; 2141 ]; 2142 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2143 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2144 DW_OP_call_ref %__active_lane_pc; 2145 ]; 2146 c; 2147 EXEC = ~EXEC & %3; 2148 $lex_1_1_else: 2149 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2150 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2151 DW_AT_location = DIExpression[ 2152 DW_OP_call_ref %__divergent_lane_pc_1_then; 2153 DW_OP_addrx &lex_1_1_end; 2154 DW_OP_stack_value; 2155 DW_OP_LLVM_extend 64, 64; 2156 DW_OP_call_ref %__lex_1_1_save_exec; 2157 DW_OP_deref_type 64, %__uint_64; 2158 DW_OP_LLVM_select_bit_piece 64, 64; 2159 ]; 2160 ]; 2161 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2162 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2163 DW_OP_call_ref %__active_lane_pc; 2164 ]; 2165 d; 2166 EXEC = %3; 2167 $lex_1_1_end: 2168 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2169 DW_OP_call_ref %__divergent_lane_pc; 2170 DW_OP_call_ref %__active_lane_pc; 2171 ]; 2172 e; 2173 EXEC = ~EXEC & %1; 2174 $lex_1_else: 2175 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2176 DW_AT_name = "__divergent_lane_pc_1_else"; 2177 DW_AT_location = DIExpression[ 2178 DW_OP_call_ref %__divergent_lane_pc; 2179 DW_OP_addrx &lex_1_end; 2180 DW_OP_stack_value; 2181 DW_OP_LLVM_extend 64, 64; 2182 DW_OP_call_ref %__lex_1_save_exec; 2183 DW_OP_deref_type 64, %__uint_64; 2184 DW_OP_LLVM_select_bit_piece 64, 64; 2185 ]; 2186 ]; 2187 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2188 DW_OP_call_ref %__divergent_lane_pc_1_else; 2189 DW_OP_call_ref %__active_lane_pc; 2190 ]; 2191 f; 2192 EXEC = %1; 2193 $lex_1_end: 2194 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2195 DW_OP_call_ref %__divergent_lane_pc; 2196 DW_OP_call_ref %__active_lane_pc; 2197 ]; 2198 g; 2199 $lex_end: 2200 2201The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2202that are active, with the current program location. 2203 2204Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2205the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2206instruction, location list entries will be created that describe where the 2207artificial variables are allocated at any given program location. The compiler 2208may allocate them to registers or spill them to memory. 2209 2210The DWARF procedures for each region use the values of the saved execution mask 2211artificial variables to only update the lanes that are active on entry to the 2212region. All other lanes retain the value of the enclosing region where they were 2213last active. If they were not active on entry to the subprogram, then will have 2214the undefined location description. 2215 2216Other structured control flow regions can be handled similarly. For example, 2217loops would set the divergent program location for the region at the end of the 2218loop. Any lanes active will be in the loop, and any lanes not active must have 2219exited the loop. 2220 2221An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2222``IF/THEN/ELSE`` regions. 2223 2224The DWARF procedures can use the active lane artificial variable described in 2225:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2226``EXEC`` mask in order to support whole or quad wavefront mode. 2227 2228.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2229 2230``DW_AT_LLVM_active_lane`` 2231~~~~~~~~~~~~~~~~~~~~~~~~~~ 2232 2233The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2234entry is used to specify the lanes that are conceptually active for a SIMT 2235thread. 2236 2237The execution mask may be modified to implement whole or quad wavefront mode 2238operations. For example, all lanes may need to temporarily be made active to 2239execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2240update it to enable the necessary lanes, perform the operations, and then 2241restore the ``EXEC`` mask from the saved value. While executing the whole 2242wavefront region, the conceptual execution mask is the saved value, not the 2243``EXEC`` value. 2244 2245This is handled by defining an artificial variable for the active lane mask. The 2246active lane mask artificial variable would be the actual ``EXEC`` mask for 2247normal regions, and the saved execution mask for regions where the mask is 2248temporarily updated. The location list expression created for this artificial 2249variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2250attribute. 2251 2252``DW_AT_LLVM_augmentation`` 2253~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2254 2255For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2256debugger information entry has the following value for the augmentation string: 2257 2258:: 2259 2260 [amdgpu:v0.0] 2261 2262The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2263extensions used in the DWARF of the compilation unit. The version number 2264conforms to [SEMVER]_. 2265 2266Call Frame Information 2267---------------------- 2268 2269DWARF Call Frame Information (CFI) describes how a consumer can virtually 2270*unwind* call frames in a running process or core dump. See DWARF Version 5 2271section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2272 2273For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2274 22751. ``augmentation`` string contains the following null-terminated UTF-8 string: 2276 2277 :: 2278 2279 [amd:v0.0] 2280 2281 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2282 extensions used in this CIE or to the FDEs that use it. The version number 2283 conforms to [SEMVER]_. 2284 22852. ``address_size`` for the ``Global`` address space is defined in 2286 :ref:`amdgpu-dwarf-address-space-identifier`. 2287 22883. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2289 22904. ``code_alignment_factor`` is 4 bytes. 2291 2292 .. TODO:: 2293 2294 Add to :ref:`amdgpu-processor-table` table. 2295 22965. ``data_alignment_factor`` is 4 bytes. 2297 2298 .. TODO:: 2299 2300 Add to :ref:`amdgpu-processor-table` table. 2301 23026. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2303 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2304 23057. ``initial_instructions`` Since a subprogram X with fewer registers can be 2306 called from subprogram Y that has more allocated, X will not change any of 2307 the extra registers as it cannot access them. Therefore, the default rule 2308 for all columns is ``same value``. 2309 2310For AMDGPU the register number follows the numbering defined in 2311:ref:`amdgpu-dwarf-register-identifier`. 2312 2313For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2314the return address to get the address of a byte within the call site 2315instructions. See DWARF Version 5 section 6.4.4. 2316 2317Accelerated Access 2318------------------ 2319 2320See DWARF Version 5 section 6.1. 2321 2322Lookup By Name Section Header 2323~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2324 2325See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2326 2327For AMDGPU the lookup by name section header table: 2328 2329``augmentation_string_size`` (uword) 2330 2331 Set to the length of the ``augmentation_string`` value which is always a 2332 multiple of 4. 2333 2334``augmentation_string`` (sequence of UTF-8 characters) 2335 2336 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2337 2338 :: 2339 2340 [amdgpu:v0.0] 2341 2342 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2343 extensions used in the DWARF of this index. The version number conforms to 2344 [SEMVER]_. 2345 2346 .. note:: 2347 2348 This is different to the DWARF Version 5 definition that requires the first 2349 4 characters to be the vendor ID. But this is consistent with the other 2350 augmentation strings and does allow multiple vendor contributions. However, 2351 backwards compatibility may be more desirable. 2352 2353Lookup By Address Section Header 2354~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2355 2356See DWARF Version 5 section 6.1.2. 2357 2358For AMDGPU the lookup by address section header table: 2359 2360``address_size`` (ubyte) 2361 2362 Match the address size for the ``Global`` address space defined in 2363 :ref:`amdgpu-dwarf-address-space-identifier`. 2364 2365``segment_selector_size`` (ubyte) 2366 2367 AMDGPU does not use a segment selector so this is 0. The entries in the 2368 ``.debug_aranges`` do not have a segment selector. 2369 2370Line Number Information 2371----------------------- 2372 2373See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2374 2375AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2376The instruction set must be obtained from the ELF file header ``e_flags`` field 2377in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2378<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2379 2380.. TODO:: 2381 2382 Should the ``isa`` state machine register be used to indicate if the code is 2383 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2384 2385For AMDGPU the line number program header fields have the following values (see 2386DWARF Version 5 section 6.2.4): 2387 2388``address_size`` (ubyte) 2389 Matches the address size for the ``Global`` address space defined in 2390 :ref:`amdgpu-dwarf-address-space-identifier`. 2391 2392``segment_selector_size`` (ubyte) 2393 AMDGPU does not use a segment selector so this is 0. 2394 2395``minimum_instruction_length`` (ubyte) 2396 For GFX9-GFX10 this is 4. 2397 2398``maximum_operations_per_instruction`` (ubyte) 2399 For GFX9-GFX10 this is 1. 2400 2401Source text for online-compiled programs (for example, those compiled by the 2402OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2403See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2404Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2405<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2406 2407The Clang option used to control source embedding in AMDGPU is defined in 2408:ref:`amdgpu-clang-debug-options-table`. 2409 2410 .. table:: AMDGPU Clang Debug Options 2411 :name: amdgpu-clang-debug-options-table 2412 2413 ==================== ================================================== 2414 Debug Flag Description 2415 ==================== ================================================== 2416 -g[no-]embed-source Enable/disable embedding source text in DWARF 2417 debug sections. Useful for environments where 2418 source cannot be written to disk, such as 2419 when performing online compilation. 2420 ==================== ================================================== 2421 2422For example: 2423 2424``-gembed-source`` 2425 Enable the embedded source. 2426 2427``-gno-embed-source`` 2428 Disable the embedded source. 2429 243032-Bit and 64-Bit DWARF Formats 2431------------------------------- 2432 2433See DWARF Version 5 section 7.4 and 2434:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2435 2436For AMDGPU: 2437 2438* For the ``amdgcn`` target architecture only the 64-bit process address space 2439 is supported. 2440 2441* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2442 the 32-bit DWARF format. 2443 2444Unit Headers 2445------------ 2446 2447For AMDGPU the following values apply for each of the unit headers described in 2448DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2449 2450``address_size`` (ubyte) 2451 Matches the address size for the ``Global`` address space defined in 2452 :ref:`amdgpu-dwarf-address-space-identifier`. 2453 2454.. _amdgpu-code-conventions: 2455 2456Code Conventions 2457================ 2458 2459This section provides code conventions used for each supported target triple OS 2460(see :ref:`amdgpu-target-triples`). 2461 2462AMDHSA 2463------ 2464 2465This section provides code conventions used when the target triple OS is 2466``amdhsa`` (see :ref:`amdgpu-target-triples`). 2467 2468.. _amdgpu-amdhsa-code-object-metadata: 2469 2470Code Object Metadata 2471~~~~~~~~~~~~~~~~~~~~ 2472 2473The code object metadata specifies extensible metadata associated with the code 2474objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2475encoding and semantics of this metadata depends on the code object version; see 2476:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2477:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2478:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2479 2480Code object metadata is specified in a note record (see 2481:ref:`amdgpu-note-records`) and is required when the target triple OS is 2482``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2483information necessary to support the HSA compatible runtime kernel queries. For 2484example, the segment sizes needed in a dispatch packet. In addition, a 2485high-level language runtime may require other information to be included. For 2486example, the AMD OpenCL runtime records kernel argument information. 2487 2488.. _amdgpu-amdhsa-code-object-metadata-v2: 2489 2490Code Object V2 Metadata 2491+++++++++++++++++++++++ 2492 2493.. warning:: 2494 Code object V2 is not the default code object version emitted by this version 2495 of LLVM. 2496 2497Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2498(see :ref:`amdgpu-note-records-v2`). 2499 2500The metadata is specified as a YAML formatted string (see [YAML]_ and 2501:doc:`YamlIO`). 2502 2503.. TODO:: 2504 2505 Is the string null terminated? It probably should not if YAML allows it to 2506 contain null characters, otherwise it should be. 2507 2508The metadata is represented as a single YAML document comprised of the mapping 2509defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2510referenced tables. 2511 2512For boolean values, the string values of ``false`` and ``true`` are used for 2513false and true respectively. 2514 2515Additional information can be added to the mappings. To avoid conflicts, any 2516non-AMD key names should be prefixed by "*vendor-name*.". 2517 2518 .. table:: AMDHSA Code Object V2 Metadata Map 2519 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2520 2521 ========== ============== ========= ======================================= 2522 String Key Value Type Required? Description 2523 ========== ============== ========= ======================================= 2524 "Version" sequence of Required - The first integer is the major 2525 2 integers version. Currently 1. 2526 - The second integer is the minor 2527 version. Currently 0. 2528 "Printf" sequence of Each string is encoded information 2529 strings about a printf function call. The 2530 encoded information is organized as 2531 fields separated by colon (':'): 2532 2533 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2534 2535 where: 2536 2537 ``ID`` 2538 A 32-bit integer as a unique id for 2539 each printf function call 2540 2541 ``N`` 2542 A 32-bit integer equal to the number 2543 of arguments of printf function call 2544 minus 1 2545 2546 ``S[i]`` (where i = 0, 1, ... , N-1) 2547 32-bit integers for the size in bytes 2548 of the i-th FormatString argument of 2549 the printf function call 2550 2551 FormatString 2552 The format string passed to the 2553 printf function call. 2554 "Kernels" sequence of Required Sequence of the mappings for each 2555 mapping kernel in the code object. See 2556 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2557 for the definition of the mapping. 2558 ========== ============== ========= ======================================= 2559 2560.. 2561 2562 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2563 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2564 2565 ================= ============== ========= ================================ 2566 String Key Value Type Required? Description 2567 ================= ============== ========= ================================ 2568 "Name" string Required Source name of the kernel. 2569 "SymbolName" string Required Name of the kernel 2570 descriptor ELF symbol. 2571 "Language" string Source language of the kernel. 2572 Values include: 2573 2574 - "OpenCL C" 2575 - "OpenCL C++" 2576 - "HCC" 2577 - "OpenMP" 2578 2579 "LanguageVersion" sequence of - The first integer is the major 2580 2 integers version. 2581 - The second integer is the 2582 minor version. 2583 "Attrs" mapping Mapping of kernel attributes. 2584 See 2585 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2586 for the mapping definition. 2587 "Args" sequence of Sequence of mappings of the 2588 mapping kernel arguments. See 2589 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2590 for the definition of the mapping. 2591 "CodeProps" mapping Mapping of properties related to 2592 the kernel code. See 2593 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2594 for the mapping definition. 2595 ================= ============== ========= ================================ 2596 2597.. 2598 2599 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2600 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2601 2602 =================== ============== ========= ============================== 2603 String Key Value Type Required? Description 2604 =================== ============== ========= ============================== 2605 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2606 3 integers must be >=1 and the dispatch 2607 work-group size X, Y, Z must 2608 correspond to the specified 2609 values. Defaults to 0, 0, 0. 2610 2611 Corresponds to the OpenCL 2612 ``reqd_work_group_size`` 2613 attribute. 2614 "WorkGroupSizeHint" sequence of The dispatch work-group size 2615 3 integers X, Y, Z is likely to be the 2616 specified values. 2617 2618 Corresponds to the OpenCL 2619 ``work_group_size_hint`` 2620 attribute. 2621 "VecTypeHint" string The name of a scalar or vector 2622 type. 2623 2624 Corresponds to the OpenCL 2625 ``vec_type_hint`` attribute. 2626 2627 "RuntimeHandle" string The external symbol name 2628 associated with a kernel. 2629 OpenCL runtime allocates a 2630 global buffer for the symbol 2631 and saves the kernel's address 2632 to it, which is used for 2633 device side enqueueing. Only 2634 available for device side 2635 enqueued kernels. 2636 =================== ============== ========= ============================== 2637 2638.. 2639 2640 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2641 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2642 2643 ================= ============== ========= ================================ 2644 String Key Value Type Required? Description 2645 ================= ============== ========= ================================ 2646 "Name" string Kernel argument name. 2647 "TypeName" string Kernel argument type name. 2648 "Size" integer Required Kernel argument size in bytes. 2649 "Align" integer Required Kernel argument alignment in 2650 bytes. Must be a power of two. 2651 "ValueKind" string Required Kernel argument kind that 2652 specifies how to set up the 2653 corresponding argument. 2654 Values include: 2655 2656 "ByValue" 2657 The argument is copied 2658 directly into the kernarg. 2659 2660 "GlobalBuffer" 2661 A global address space pointer 2662 to the buffer data is passed 2663 in the kernarg. 2664 2665 "DynamicSharedPointer" 2666 A group address space pointer 2667 to dynamically allocated LDS 2668 is passed in the kernarg. 2669 2670 "Sampler" 2671 A global address space 2672 pointer to a S# is passed in 2673 the kernarg. 2674 2675 "Image" 2676 A global address space 2677 pointer to a T# is passed in 2678 the kernarg. 2679 2680 "Pipe" 2681 A global address space pointer 2682 to an OpenCL pipe is passed in 2683 the kernarg. 2684 2685 "Queue" 2686 A global address space pointer 2687 to an OpenCL device enqueue 2688 queue is passed in the 2689 kernarg. 2690 2691 "HiddenGlobalOffsetX" 2692 The OpenCL grid dispatch 2693 global offset for the X 2694 dimension is passed in the 2695 kernarg. 2696 2697 "HiddenGlobalOffsetY" 2698 The OpenCL grid dispatch 2699 global offset for the Y 2700 dimension is passed in the 2701 kernarg. 2702 2703 "HiddenGlobalOffsetZ" 2704 The OpenCL grid dispatch 2705 global offset for the Z 2706 dimension is passed in the 2707 kernarg. 2708 2709 "HiddenNone" 2710 An argument that is not used 2711 by the kernel. Space needs to 2712 be left for it, but it does 2713 not need to be set up. 2714 2715 "HiddenPrintfBuffer" 2716 A global address space pointer 2717 to the runtime printf buffer 2718 is passed in kernarg. 2719 2720 "HiddenHostcallBuffer" 2721 A global address space pointer 2722 to the runtime hostcall buffer 2723 is passed in kernarg. 2724 2725 "HiddenDefaultQueue" 2726 A global address space pointer 2727 to the OpenCL device enqueue 2728 queue that should be used by 2729 the kernel by default is 2730 passed in the kernarg. 2731 2732 "HiddenCompletionAction" 2733 A global address space pointer 2734 to help link enqueued kernels into 2735 the ancestor tree for determining 2736 when the parent kernel has finished. 2737 2738 "HiddenMultiGridSyncArg" 2739 A global address space pointer for 2740 multi-grid synchronization is 2741 passed in the kernarg. 2742 2743 "ValueType" string Unused and deprecated. This should no longer 2744 be emitted, but is accepted for compatibility. 2745 2746 2747 "PointeeAlign" integer Alignment in bytes of pointee 2748 type for pointer type kernel 2749 argument. Must be a power 2750 of 2. Only present if 2751 "ValueKind" is 2752 "DynamicSharedPointer". 2753 "AddrSpaceQual" string Kernel argument address space 2754 qualifier. Only present if 2755 "ValueKind" is "GlobalBuffer" or 2756 "DynamicSharedPointer". Values 2757 are: 2758 2759 - "Private" 2760 - "Global" 2761 - "Constant" 2762 - "Local" 2763 - "Generic" 2764 - "Region" 2765 2766 .. TODO:: 2767 2768 Is GlobalBuffer only Global 2769 or Constant? Is 2770 DynamicSharedPointer always 2771 Local? Can HCC allow Generic? 2772 How can Private or Region 2773 ever happen? 2774 2775 "AccQual" string Kernel argument access 2776 qualifier. Only present if 2777 "ValueKind" is "Image" or 2778 "Pipe". Values 2779 are: 2780 2781 - "ReadOnly" 2782 - "WriteOnly" 2783 - "ReadWrite" 2784 2785 .. TODO:: 2786 2787 Does this apply to 2788 GlobalBuffer? 2789 2790 "ActualAccQual" string The actual memory accesses 2791 performed by the kernel on the 2792 kernel argument. Only present if 2793 "ValueKind" is "GlobalBuffer", 2794 "Image", or "Pipe". This may be 2795 more restrictive than indicated 2796 by "AccQual" to reflect what the 2797 kernel actual does. If not 2798 present then the runtime must 2799 assume what is implied by 2800 "AccQual" and "IsConst". Values 2801 are: 2802 2803 - "ReadOnly" 2804 - "WriteOnly" 2805 - "ReadWrite" 2806 2807 "IsConst" boolean Indicates if the kernel argument 2808 is const qualified. Only present 2809 if "ValueKind" is 2810 "GlobalBuffer". 2811 2812 "IsRestrict" boolean Indicates if the kernel argument 2813 is restrict qualified. Only 2814 present if "ValueKind" is 2815 "GlobalBuffer". 2816 2817 "IsVolatile" boolean Indicates if the kernel argument 2818 is volatile qualified. Only 2819 present if "ValueKind" is 2820 "GlobalBuffer". 2821 2822 "IsPipe" boolean Indicates if the kernel argument 2823 is pipe qualified. Only present 2824 if "ValueKind" is "Pipe". 2825 2826 .. TODO:: 2827 2828 Can GlobalBuffer be pipe 2829 qualified? 2830 2831 ================= ============== ========= ================================ 2832 2833.. 2834 2835 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2836 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2837 2838 ============================ ============== ========= ===================== 2839 String Key Value Type Required? Description 2840 ============================ ============== ========= ===================== 2841 "KernargSegmentSize" integer Required The size in bytes of 2842 the kernarg segment 2843 that holds the values 2844 of the arguments to 2845 the kernel. 2846 "GroupSegmentFixedSize" integer Required The amount of group 2847 segment memory 2848 required by a 2849 work-group in 2850 bytes. This does not 2851 include any 2852 dynamically allocated 2853 group segment memory 2854 that may be added 2855 when the kernel is 2856 dispatched. 2857 "PrivateSegmentFixedSize" integer Required The amount of fixed 2858 private address space 2859 memory required for a 2860 work-item in 2861 bytes. If the kernel 2862 uses a dynamic call 2863 stack then additional 2864 space must be added 2865 to this value for the 2866 call stack. 2867 "KernargSegmentAlign" integer Required The maximum byte 2868 alignment of 2869 arguments in the 2870 kernarg segment. Must 2871 be a power of 2. 2872 "WavefrontSize" integer Required Wavefront size. Must 2873 be a power of 2. 2874 "NumSGPRs" integer Required Number of scalar 2875 registers used by a 2876 wavefront for 2877 GFX6-GFX10. This 2878 includes the special 2879 SGPRs for VCC, Flat 2880 Scratch (GFX7-GFX10) 2881 and XNACK (for 2882 GFX8-GFX10). It does 2883 not include the 16 2884 SGPR added if a trap 2885 handler is 2886 enabled. It is not 2887 rounded up to the 2888 allocation 2889 granularity. 2890 "NumVGPRs" integer Required Number of vector 2891 registers used by 2892 each work-item for 2893 GFX6-GFX10 2894 "MaxFlatWorkGroupSize" integer Required Maximum flat 2895 work-group size 2896 supported by the 2897 kernel in work-items. 2898 Must be >=1 and 2899 consistent with 2900 ReqdWorkGroupSize if 2901 not 0, 0, 0. 2902 "NumSpilledSGPRs" integer Number of stores from 2903 a scalar register to 2904 a register allocator 2905 created spill 2906 location. 2907 "NumSpilledVGPRs" integer Number of stores from 2908 a vector register to 2909 a register allocator 2910 created spill 2911 location. 2912 ============================ ============== ========= ===================== 2913 2914.. _amdgpu-amdhsa-code-object-metadata-v3: 2915 2916Code Object V3 Metadata 2917+++++++++++++++++++++++ 2918 2919Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2920record (see :ref:`amdgpu-note-records-v3-v4`). 2921 2922The metadata is represented as Message Pack formatted binary data (see 2923[MsgPack]_). The top level is a Message Pack map that includes the 2924keys defined in table 2925:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2926tables. 2927 2928Additional information can be added to the maps. To avoid conflicts, 2929any key names should be prefixed by "*vendor-name*." where 2930``vendor-name`` can be the name of the vendor and specific vendor 2931tool that generates the information. The prefix is abbreviated to 2932simply "." when it appears within a map that has been added by the 2933same *vendor-name*. 2934 2935 .. table:: AMDHSA Code Object V3 Metadata Map 2936 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2937 2938 ================= ============== ========= ======================================= 2939 String Key Value Type Required? Description 2940 ================= ============== ========= ======================================= 2941 "amdhsa.version" sequence of Required - The first integer is the major 2942 2 integers version. Currently 1. 2943 - The second integer is the minor 2944 version. Currently 0. 2945 "amdhsa.printf" sequence of Each string is encoded information 2946 strings about a printf function call. The 2947 encoded information is organized as 2948 fields separated by colon (':'): 2949 2950 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2951 2952 where: 2953 2954 ``ID`` 2955 A 32-bit integer as a unique id for 2956 each printf function call 2957 2958 ``N`` 2959 A 32-bit integer equal to the number 2960 of arguments of printf function call 2961 minus 1 2962 2963 ``S[i]`` (where i = 0, 1, ... , N-1) 2964 32-bit integers for the size in bytes 2965 of the i-th FormatString argument of 2966 the printf function call 2967 2968 FormatString 2969 The format string passed to the 2970 printf function call. 2971 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2972 map kernel in the code object. See 2973 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2974 for the definition of the keys included 2975 in that map. 2976 ================= ============== ========= ======================================= 2977 2978.. 2979 2980 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2981 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2982 2983 =================================== ============== ========= ================================ 2984 String Key Value Type Required? Description 2985 =================================== ============== ========= ================================ 2986 ".name" string Required Source name of the kernel. 2987 ".symbol" string Required Name of the kernel 2988 descriptor ELF symbol. 2989 ".language" string Source language of the kernel. 2990 Values include: 2991 2992 - "OpenCL C" 2993 - "OpenCL C++" 2994 - "HCC" 2995 - "HIP" 2996 - "OpenMP" 2997 - "Assembler" 2998 2999 ".language_version" sequence of - The first integer is the major 3000 2 integers version. 3001 - The second integer is the 3002 minor version. 3003 ".args" sequence of Sequence of maps of the 3004 map kernel arguments. See 3005 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3006 for the definition of the keys 3007 included in that map. 3008 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3009 3 integers must be >=1 and the dispatch 3010 work-group size X, Y, Z must 3011 correspond to the specified 3012 values. Defaults to 0, 0, 0. 3013 3014 Corresponds to the OpenCL 3015 ``reqd_work_group_size`` 3016 attribute. 3017 ".workgroup_size_hint" sequence of The dispatch work-group size 3018 3 integers X, Y, Z is likely to be the 3019 specified values. 3020 3021 Corresponds to the OpenCL 3022 ``work_group_size_hint`` 3023 attribute. 3024 ".vec_type_hint" string The name of a scalar or vector 3025 type. 3026 3027 Corresponds to the OpenCL 3028 ``vec_type_hint`` attribute. 3029 3030 ".device_enqueue_symbol" string The external symbol name 3031 associated with a kernel. 3032 OpenCL runtime allocates a 3033 global buffer for the symbol 3034 and saves the kernel's address 3035 to it, which is used for 3036 device side enqueueing. Only 3037 available for device side 3038 enqueued kernels. 3039 ".kernarg_segment_size" integer Required The size in bytes of 3040 the kernarg segment 3041 that holds the values 3042 of the arguments to 3043 the kernel. 3044 ".group_segment_fixed_size" integer Required The amount of group 3045 segment memory 3046 required by a 3047 work-group in 3048 bytes. This does not 3049 include any 3050 dynamically allocated 3051 group segment memory 3052 that may be added 3053 when the kernel is 3054 dispatched. 3055 ".private_segment_fixed_size" integer Required The amount of fixed 3056 private address space 3057 memory required for a 3058 work-item in 3059 bytes. If the kernel 3060 uses a dynamic call 3061 stack then additional 3062 space must be added 3063 to this value for the 3064 call stack. 3065 ".kernarg_segment_align" integer Required The maximum byte 3066 alignment of 3067 arguments in the 3068 kernarg segment. Must 3069 be a power of 2. 3070 ".wavefront_size" integer Required Wavefront size. Must 3071 be a power of 2. 3072 ".sgpr_count" integer Required Number of scalar 3073 registers required by a 3074 wavefront for 3075 GFX6-GFX9. A register 3076 is required if it is 3077 used explicitly, or 3078 if a higher numbered 3079 register is used 3080 explicitly. This 3081 includes the special 3082 SGPRs for VCC, Flat 3083 Scratch (GFX7-GFX9) 3084 and XNACK (for 3085 GFX8-GFX9). It does 3086 not include the 16 3087 SGPR added if a trap 3088 handler is 3089 enabled. It is not 3090 rounded up to the 3091 allocation 3092 granularity. 3093 ".vgpr_count" integer Required Number of vector 3094 registers required by 3095 each work-item for 3096 GFX6-GFX9. A register 3097 is required if it is 3098 used explicitly, or 3099 if a higher numbered 3100 register is used 3101 explicitly. 3102 ".max_flat_workgroup_size" integer Required Maximum flat 3103 work-group size 3104 supported by the 3105 kernel in work-items. 3106 Must be >=1 and 3107 consistent with 3108 ReqdWorkGroupSize if 3109 not 0, 0, 0. 3110 ".sgpr_spill_count" integer Number of stores from 3111 a scalar register to 3112 a register allocator 3113 created spill 3114 location. 3115 ".vgpr_spill_count" integer Number of stores from 3116 a vector register to 3117 a register allocator 3118 created spill 3119 location. 3120 =================================== ============== ========= ================================ 3121 3122.. 3123 3124 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3125 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3126 3127 ====================== ============== ========= ================================ 3128 String Key Value Type Required? Description 3129 ====================== ============== ========= ================================ 3130 ".name" string Kernel argument name. 3131 ".type_name" string Kernel argument type name. 3132 ".size" integer Required Kernel argument size in bytes. 3133 ".offset" integer Required Kernel argument offset in 3134 bytes. The offset must be a 3135 multiple of the alignment 3136 required by the argument. 3137 ".value_kind" string Required Kernel argument kind that 3138 specifies how to set up the 3139 corresponding argument. 3140 Values include: 3141 3142 "by_value" 3143 The argument is copied 3144 directly into the kernarg. 3145 3146 "global_buffer" 3147 A global address space pointer 3148 to the buffer data is passed 3149 in the kernarg. 3150 3151 "dynamic_shared_pointer" 3152 A group address space pointer 3153 to dynamically allocated LDS 3154 is passed in the kernarg. 3155 3156 "sampler" 3157 A global address space 3158 pointer to a S# is passed in 3159 the kernarg. 3160 3161 "image" 3162 A global address space 3163 pointer to a T# is passed in 3164 the kernarg. 3165 3166 "pipe" 3167 A global address space pointer 3168 to an OpenCL pipe is passed in 3169 the kernarg. 3170 3171 "queue" 3172 A global address space pointer 3173 to an OpenCL device enqueue 3174 queue is passed in the 3175 kernarg. 3176 3177 "hidden_global_offset_x" 3178 The OpenCL grid dispatch 3179 global offset for the X 3180 dimension is passed in the 3181 kernarg. 3182 3183 "hidden_global_offset_y" 3184 The OpenCL grid dispatch 3185 global offset for the Y 3186 dimension is passed in the 3187 kernarg. 3188 3189 "hidden_global_offset_z" 3190 The OpenCL grid dispatch 3191 global offset for the Z 3192 dimension is passed in the 3193 kernarg. 3194 3195 "hidden_none" 3196 An argument that is not used 3197 by the kernel. Space needs to 3198 be left for it, but it does 3199 not need to be set up. 3200 3201 "hidden_printf_buffer" 3202 A global address space pointer 3203 to the runtime printf buffer 3204 is passed in kernarg. 3205 3206 "hidden_hostcall_buffer" 3207 A global address space pointer 3208 to the runtime hostcall buffer 3209 is passed in kernarg. 3210 3211 "hidden_default_queue" 3212 A global address space pointer 3213 to the OpenCL device enqueue 3214 queue that should be used by 3215 the kernel by default is 3216 passed in the kernarg. 3217 3218 "hidden_completion_action" 3219 A global address space pointer 3220 to help link enqueued kernels into 3221 the ancestor tree for determining 3222 when the parent kernel has finished. 3223 3224 "hidden_multigrid_sync_arg" 3225 A global address space pointer for 3226 multi-grid synchronization is 3227 passed in the kernarg. 3228 3229 ".value_type" string Unused and deprecated. This should no longer 3230 be emitted, but is accepted for compatibility. 3231 3232 ".pointee_align" integer Alignment in bytes of pointee 3233 type for pointer type kernel 3234 argument. Must be a power 3235 of 2. Only present if 3236 ".value_kind" is 3237 "dynamic_shared_pointer". 3238 ".address_space" string Kernel argument address space 3239 qualifier. Only present if 3240 ".value_kind" is "global_buffer" or 3241 "dynamic_shared_pointer". Values 3242 are: 3243 3244 - "private" 3245 - "global" 3246 - "constant" 3247 - "local" 3248 - "generic" 3249 - "region" 3250 3251 .. TODO:: 3252 3253 Is "global_buffer" only "global" 3254 or "constant"? Is 3255 "dynamic_shared_pointer" always 3256 "local"? Can HCC allow "generic"? 3257 How can "private" or "region" 3258 ever happen? 3259 3260 ".access" string Kernel argument access 3261 qualifier. Only present if 3262 ".value_kind" is "image" or 3263 "pipe". Values 3264 are: 3265 3266 - "read_only" 3267 - "write_only" 3268 - "read_write" 3269 3270 .. TODO:: 3271 3272 Does this apply to 3273 "global_buffer"? 3274 3275 ".actual_access" string The actual memory accesses 3276 performed by the kernel on the 3277 kernel argument. Only present if 3278 ".value_kind" is "global_buffer", 3279 "image", or "pipe". This may be 3280 more restrictive than indicated 3281 by ".access" to reflect what the 3282 kernel actual does. If not 3283 present then the runtime must 3284 assume what is implied by 3285 ".access" and ".is_const" . Values 3286 are: 3287 3288 - "read_only" 3289 - "write_only" 3290 - "read_write" 3291 3292 ".is_const" boolean Indicates if the kernel argument 3293 is const qualified. Only present 3294 if ".value_kind" is 3295 "global_buffer". 3296 3297 ".is_restrict" boolean Indicates if the kernel argument 3298 is restrict qualified. Only 3299 present if ".value_kind" is 3300 "global_buffer". 3301 3302 ".is_volatile" boolean Indicates if the kernel argument 3303 is volatile qualified. Only 3304 present if ".value_kind" is 3305 "global_buffer". 3306 3307 ".is_pipe" boolean Indicates if the kernel argument 3308 is pipe qualified. Only present 3309 if ".value_kind" is "pipe". 3310 3311 .. TODO:: 3312 3313 Can "global_buffer" be pipe 3314 qualified? 3315 3316 ====================== ============== ========= ================================ 3317 3318.. _amdgpu-amdhsa-code-object-metadata-v4: 3319 3320Code Object V4 Metadata 3321+++++++++++++++++++++++ 3322 3323.. warning:: 3324 Code object V4 is not the default code object version emitted by this version 3325 of LLVM. 3326 3327Code object V4 metadata is the same as 3328:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3329defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3330 3331 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3332 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3333 3334 ================= ============== ========= ======================================= 3335 String Key Value Type Required? Description 3336 ================= ============== ========= ======================================= 3337 "amdhsa.version" sequence of Required - The first integer is the major 3338 2 integers version. Currently 1. 3339 - The second integer is the minor 3340 version. Currently 1. 3341 "amdhsa.target" string Required The target name of the code using the syntax: 3342 3343 .. code:: 3344 3345 <target-triple> [ "-" <target-id> ] 3346 3347 A canonical target ID must be 3348 used. See :ref:`amdgpu-target-triples` 3349 and :ref:`amdgpu-target-id`. 3350 ================= ============== ========= ======================================= 3351 3352.. 3353 3354Kernel Dispatch 3355~~~~~~~~~~~~~~~ 3356 3357The HSA architected queuing language (AQL) defines a user space memory interface 3358that can be used to control the dispatch of kernels, in an agent independent 3359way. An agent can have zero or more AQL queues created for it using an HSA 3360compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3361are 64 bytes) can be placed. See the *HSA Platform System Architecture 3362Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3363 3364The packet processor of a kernel agent is responsible for detecting and 3365dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3366packet processor is implemented by the hardware command processor (CP), 3367asynchronous dispatch controller (ADC) and shader processor input controller 3368(SPI). 3369 3370An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3371the kernel mode driver to initialize and register the AQL queue with CP. 3372 3373To dispatch a kernel the following actions are performed. This can occur in the 3374CPU host program, or from an HSA kernel executing on a GPU. 3375 33761. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3377 executed is obtained. 33782. A pointer to the kernel descriptor (see 3379 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3380 It must be for a kernel that is contained in a code object that that was 3381 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3382 queue is associated. 33833. Space is allocated for the kernel arguments using the HSA compatible runtime 3384 allocator for a memory region with the kernarg property for the kernel agent 3385 that will execute the kernel. It must be at least 16-byte aligned. 33864. Kernel argument values are assigned to the kernel argument memory 3387 allocation. The layout is defined in the *HSA Programmer's Language 3388 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3389 kernel argument memory in the same way constant memory is accessed. (Note 3390 that the HSA specification allows an implementation to copy the kernel 3391 argument contents to another location that is accessed by the kernel.) 33925. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3393 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3394 for the packet. The packet must be set up, and the final write must use an 3395 atomic store release to set the packet kind to ensure the packet contents are 3396 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3397 notify the kernel agent that the AQL queue has been updated. These rules, and 3398 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3399 System Architecture Specification* [HSA]_. 34006. A kernel dispatch packet includes information about the actual dispatch, 3401 such as grid and work-group size, together with information from the code 3402 object about the kernel, such as segment sizes. The HSA compatible runtime 3403 queries on the kernel symbol can be used to obtain the code object values 3404 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 34057. CP executes micro-code and is responsible for detecting and setting up the 3406 GPU to execute the wavefronts of a kernel dispatch. 34078. CP ensures that when the a wavefront starts executing the kernel machine 3408 code, the scalar general purpose registers (SGPR) and vector general purpose 3409 registers (VGPR) are set up as required by the machine code. The required 3410 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3411 register state is defined in 3412 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 34139. The prolog of the kernel machine code (see 3414 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3415 before continuing executing the machine code that corresponds to the kernel. 341610. When the kernel dispatch has completed execution, CP signals the completion 3417 signal specified in the kernel dispatch packet if not 0. 3418 3419.. _amdgpu-amdhsa-memory-spaces: 3420 3421Memory Spaces 3422~~~~~~~~~~~~~ 3423 3424The memory space properties are: 3425 3426 .. table:: AMDHSA Memory Spaces 3427 :name: amdgpu-amdhsa-memory-spaces-table 3428 3429 ================= =========== ======== ======= ================== 3430 Memory Space Name HSA Segment Hardware Address NULL Value 3431 Name Name Size 3432 ================= =========== ======== ======= ================== 3433 Private private scratch 32 0x00000000 3434 Local group LDS 32 0xFFFFFFFF 3435 Global global global 64 0x0000000000000000 3436 Constant constant *same as 64 0x0000000000000000 3437 global* 3438 Generic flat flat 64 0x0000000000000000 3439 Region N/A GDS 32 *not implemented 3440 for AMDHSA* 3441 ================= =========== ======== ======= ================== 3442 3443The global and constant memory spaces both use global virtual addresses, which 3444are the same virtual address space used by the CPU. However, some virtual 3445addresses may only be accessible to the CPU, some only accessible by the GPU, 3446and some by both. 3447 3448Using the constant memory space indicates that the data will not change during 3449the execution of the kernel. This allows scalar read instructions to be 3450used. The vector and scalar L1 caches are invalidated of volatile data before 3451each kernel dispatch execution to allow constant memory to change values between 3452kernel dispatches. 3453 3454The local memory space uses the hardware Local Data Store (LDS) which is 3455automatically allocated when the hardware creates work-groups of wavefronts, and 3456freed when all the wavefronts of a work-group have terminated. The data store 3457(DS) instructions can be used to access it. 3458 3459The private memory space uses the hardware scratch memory support. If the kernel 3460uses scratch, then the hardware allocates memory that is accessed using 3461wavefront lane dword (4 byte) interleaving. The mapping used from private 3462address to physical address is: 3463 3464 ``wavefront-scratch-base + 3465 (private-address * wavefront-size * 4) + 3466 (wavefront-lane-id * 4)`` 3467 3468There are different ways that the wavefront scratch base address is determined 3469by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3470memory can be accessed in an interleaved manner using buffer instruction with 3471the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3472instructions, or by flat instructions. If each lane of a wavefront accesses the 3473same private address, the interleaving results in adjacent dwords being accessed 3474and hence requires fewer cache lines to be fetched. Multi-dword access is not 3475supported except by flat and scratch instructions in GFX9-GFX10. 3476 3477The generic address space uses the hardware flat address support available in 3478GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3479local apertures), that are outside the range of addressible global memory, to 3480map from a flat address to a private or local address. 3481 3482FLAT instructions can take a flat address and access global, private (scratch) 3483and group (LDS) memory depending in if the address is within one of the 3484aperture ranges. Flat access to scratch requires hardware aperture setup and 3485setup in the kernel prologue (see 3486:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3487hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3488:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3489 3490To convert between a segment address and a flat address the base address of the 3491apertures address can be used. For GFX7-GFX8 these are available in the 3492:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3493Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3494GFX9-GFX10 the aperture base addresses are directly available as inline constant 3495registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3496address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3497which makes it easier to convert from flat to segment or segment to flat. 3498 3499Image and Samplers 3500~~~~~~~~~~~~~~~~~~ 3501 3502Image and sample handles created by an HSA compatible runtime (see 3503:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3504object respectively. In order to support the HSA ``query_sampler`` operations 3505two extra dwords are used to store the HSA BRIG enumeration values for the 3506queries that are not trivially deducible from the S# representation. 3507 3508HSA Signals 3509~~~~~~~~~~~ 3510 3511HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3512are 64-bit addresses of a structure allocated in memory accessible from both the 3513CPU and GPU. The structure is defined by the runtime and subject to change 3514between releases. For example, see [AMD-ROCm-github]_. 3515 3516.. _amdgpu-amdhsa-hsa-aql-queue: 3517 3518HSA AQL Queue 3519~~~~~~~~~~~~~ 3520 3521The HSA AQL queue structure is defined by an HSA compatible runtime (see 3522:ref:`amdgpu-os`) and subject to change between releases. For example, see 3523[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3524certain language features such as the flat address aperture bases. It also 3525contains fields used by CP such as managing the allocation of scratch memory. 3526 3527.. _amdgpu-amdhsa-kernel-descriptor: 3528 3529Kernel Descriptor 3530~~~~~~~~~~~~~~~~~ 3531 3532A kernel descriptor consists of the information needed by CP to initiate the 3533execution of a kernel, including the entry point address of the machine code 3534that implements the kernel. 3535 3536Code Object V3 Kernel Descriptor 3537++++++++++++++++++++++++++++++++ 3538 3539CP microcode requires the Kernel descriptor to be allocated on 64-byte 3540alignment. 3541 3542The fields used by CP for code objects before V3 also match those specified in 3543:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3544 3545 .. table:: Code Object V3 Kernel Descriptor 3546 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3547 3548 ======= ======= =============================== ============================ 3549 Bits Size Field Name Description 3550 ======= ======= =============================== ============================ 3551 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3552 address space memory 3553 required for a work-group 3554 in bytes. This does not 3555 include any dynamically 3556 allocated local address 3557 space memory that may be 3558 added when the kernel is 3559 dispatched. 3560 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3561 private address space 3562 memory required for a 3563 work-item in bytes. 3564 Additional space may need to 3565 be added to this value if 3566 the call stack has 3567 non-inlined function calls. 3568 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3569 memory pointed to by the 3570 AQL dispatch packet. The 3571 kernarg memory is used to 3572 pass arguments to the 3573 kernel. 3574 3575 * If the kernarg pointer in 3576 the dispatch packet is NULL 3577 then there are no kernel 3578 arguments. 3579 * If the kernarg pointer in 3580 the dispatch packet is 3581 not NULL and this value 3582 is 0 then the kernarg 3583 memory size is 3584 unspecified. 3585 * If the kernarg pointer in 3586 the dispatch packet is 3587 not NULL and this value 3588 is not 0 then the value 3589 specifies the kernarg 3590 memory size in bytes. It 3591 is recommended to provide 3592 a value as it may be used 3593 by CP to optimize making 3594 the kernarg memory 3595 visible to the kernel 3596 code. 3597 3598 127:96 4 bytes Reserved, must be 0. 3599 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3600 negative) from base 3601 address of kernel 3602 descriptor to kernel's 3603 entry point instruction 3604 which must be 256 byte 3605 aligned. 3606 351:272 20 Reserved, must be 0. 3607 bytes 3608 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3609 Reserved, must be 0. 3610 GFX90A 3611 Compute Shader (CS) 3612 program settings used by 3613 CP to set up 3614 ``COMPUTE_PGM_RSRC3`` 3615 configuration 3616 register. See 3617 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3618 GFX10 3619 Compute Shader (CS) 3620 program settings used by 3621 CP to set up 3622 ``COMPUTE_PGM_RSRC3`` 3623 configuration 3624 register. See 3625 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3626 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3627 program settings used by 3628 CP to set up 3629 ``COMPUTE_PGM_RSRC1`` 3630 configuration 3631 register. See 3632 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3633 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3634 program settings used by 3635 CP to set up 3636 ``COMPUTE_PGM_RSRC2`` 3637 configuration 3638 register. See 3639 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3640 458:448 7 bits *See separate bits below.* Enable the setup of the 3641 SGPR user data registers 3642 (see 3643 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3644 3645 The total number of SGPR 3646 user data registers 3647 requested must not exceed 3648 16 and match value in 3649 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3650 Any requests beyond 16 3651 will be ignored. 3652 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3653 _BUFFER 3654 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3655 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3656 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3657 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3658 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT 3659 3660 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3661 _SIZE 3662 457:455 3 bits Reserved, must be 0. 3663 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3664 Reserved, must be 0. 3665 GFX10 3666 - If 0 execute in 3667 wavefront size 64 mode. 3668 - If 1 execute in 3669 native wavefront size 3670 32 mode. 3671 463:459 1 bit Reserved, must be 0. 3672 464 1 bit RESERVED_464 Deprecated, must be 0. 3673 467:465 3 bits Reserved, must be 0. 3674 468 1 bit RESERVED_468 Deprecated, must be 0. 3675 469:471 3 bits Reserved, must be 0. 3676 511:472 5 bytes Reserved, must be 0. 3677 512 **Total size 64 bytes.** 3678 ======= ==================================================================== 3679 3680.. 3681 3682 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3683 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3684 3685 ======= ======= =============================== =========================================================================== 3686 Bits Size Field Name Description 3687 ======= ======= =============================== =========================================================================== 3688 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3689 blocks used by each work-item; 3690 granularity is device 3691 specific: 3692 3693 GFX6-GFX9 3694 - vgprs_used 0..256 3695 - max(0, ceil(vgprs_used / 4) - 1) 3696 GFX90A 3697 - vgprs_used 0..512 3698 - vgprs_used = align(arch_vgprs, 4) 3699 + acc_vgprs 3700 - max(0, ceil(vgprs_used / 8) - 1) 3701 GFX10 (wavefront size 64) 3702 - max_vgpr 1..256 3703 - max(0, ceil(vgprs_used / 4) - 1) 3704 GFX10 (wavefront size 32) 3705 - max_vgpr 1..256 3706 - max(0, ceil(vgprs_used / 8) - 1) 3707 3708 Where vgprs_used is defined 3709 as the highest VGPR number 3710 explicitly referenced plus 3711 one. 3712 3713 Used by CP to set up 3714 ``COMPUTE_PGM_RSRC1.VGPRS``. 3715 3716 The 3717 :ref:`amdgpu-assembler` 3718 calculates this 3719 automatically for the 3720 selected processor from 3721 values provided to the 3722 `.amdhsa_kernel` directive 3723 by the 3724 `.amdhsa_next_free_vgpr` 3725 nested directive (see 3726 :ref:`amdhsa-kernel-directives-table`). 3727 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3728 blocks used by a wavefront; 3729 granularity is device 3730 specific: 3731 3732 GFX6-GFX8 3733 - sgprs_used 0..112 3734 - max(0, ceil(sgprs_used / 8) - 1) 3735 GFX9 3736 - sgprs_used 0..112 3737 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3738 GFX10 3739 Reserved, must be 0. 3740 (128 SGPRs always 3741 allocated.) 3742 3743 Where sgprs_used is 3744 defined as the highest 3745 SGPR number explicitly 3746 referenced plus one, plus 3747 a target specific number 3748 of additional special 3749 SGPRs for VCC, 3750 FLAT_SCRATCH (GFX7+) and 3751 XNACK_MASK (GFX8+), and 3752 any additional 3753 target specific 3754 limitations. It does not 3755 include the 16 SGPRs added 3756 if a trap handler is 3757 enabled. 3758 3759 The target specific 3760 limitations and special 3761 SGPR layout are defined in 3762 the hardware 3763 documentation, which can 3764 be found in the 3765 :ref:`amdgpu-processors` 3766 table. 3767 3768 Used by CP to set up 3769 ``COMPUTE_PGM_RSRC1.SGPRS``. 3770 3771 The 3772 :ref:`amdgpu-assembler` 3773 calculates this 3774 automatically for the 3775 selected processor from 3776 values provided to the 3777 `.amdhsa_kernel` directive 3778 by the 3779 `.amdhsa_next_free_sgpr` 3780 and `.amdhsa_reserve_*` 3781 nested directives (see 3782 :ref:`amdhsa-kernel-directives-table`). 3783 11:10 2 bits PRIORITY Must be 0. 3784 3785 Start executing wavefront 3786 at the specified priority. 3787 3788 CP is responsible for 3789 filling in 3790 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3791 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3792 with specified rounding 3793 mode for single (32 3794 bit) floating point 3795 precision floating point 3796 operations. 3797 3798 Floating point rounding 3799 mode values are defined in 3800 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3801 3802 Used by CP to set up 3803 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3804 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3805 with specified rounding 3806 denorm mode for half/double (16 3807 and 64-bit) floating point 3808 precision floating point 3809 operations. 3810 3811 Floating point rounding 3812 mode values are defined in 3813 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3814 3815 Used by CP to set up 3816 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3817 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3818 with specified denorm mode 3819 for single (32 3820 bit) floating point 3821 precision floating point 3822 operations. 3823 3824 Floating point denorm mode 3825 values are defined in 3826 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3827 3828 Used by CP to set up 3829 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3830 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3831 with specified denorm mode 3832 for half/double (16 3833 and 64-bit) floating point 3834 precision floating point 3835 operations. 3836 3837 Floating point denorm mode 3838 values are defined in 3839 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3840 3841 Used by CP to set up 3842 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3843 20 1 bit PRIV Must be 0. 3844 3845 Start executing wavefront 3846 in privilege trap handler 3847 mode. 3848 3849 CP is responsible for 3850 filling in 3851 ``COMPUTE_PGM_RSRC1.PRIV``. 3852 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3853 with DX10 clamp mode 3854 enabled. Used by the vector 3855 ALU to force DX10 style 3856 treatment of NaN's (when 3857 set, clamp NaN to zero, 3858 otherwise pass NaN 3859 through). 3860 3861 Used by CP to set up 3862 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3863 22 1 bit DEBUG_MODE Must be 0. 3864 3865 Start executing wavefront 3866 in single step mode. 3867 3868 CP is responsible for 3869 filling in 3870 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3871 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3872 with IEEE mode 3873 enabled. Floating point 3874 opcodes that support 3875 exception flag gathering 3876 will quiet and propagate 3877 signaling-NaN inputs per 3878 IEEE 754-2008. Min_dx10 and 3879 max_dx10 become IEEE 3880 754-2008 compliant due to 3881 signaling-NaN propagation 3882 and quieting. 3883 3884 Used by CP to set up 3885 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3886 24 1 bit BULKY Must be 0. 3887 3888 Only one work-group allowed 3889 to execute on a compute 3890 unit. 3891 3892 CP is responsible for 3893 filling in 3894 ``COMPUTE_PGM_RSRC1.BULKY``. 3895 25 1 bit CDBG_USER Must be 0. 3896 3897 Flag that can be used to 3898 control debugging code. 3899 3900 CP is responsible for 3901 filling in 3902 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3903 26 1 bit FP16_OVFL GFX6-GFX8 3904 Reserved, must be 0. 3905 GFX9-GFX10 3906 Wavefront starts execution 3907 with specified fp16 overflow 3908 mode. 3909 3910 - If 0, fp16 overflow generates 3911 +/-INF values. 3912 - If 1, fp16 overflow that is the 3913 result of an +/-INF input value 3914 or divide by 0 produces a +/-INF, 3915 otherwise clamps computed 3916 overflow to +/-MAX_FP16 as 3917 appropriate. 3918 3919 Used by CP to set up 3920 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3921 28:27 2 bits Reserved, must be 0. 3922 29 1 bit WGP_MODE GFX6-GFX9 3923 Reserved, must be 0. 3924 GFX10 3925 - If 0 execute work-groups in 3926 CU wavefront execution mode. 3927 - If 1 execute work-groups on 3928 in WGP wavefront execution mode. 3929 3930 See :ref:`amdgpu-amdhsa-memory-model`. 3931 3932 Used by CP to set up 3933 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3934 30 1 bit MEM_ORDERED GFX6-GFX9 3935 Reserved, must be 0. 3936 GFX10 3937 Controls the behavior of the 3938 s_waitcnt's vmcnt and vscnt 3939 counters. 3940 3941 - If 0 vmcnt reports completion 3942 of load and atomic with return 3943 out of order with sample 3944 instructions, and the vscnt 3945 reports the completion of 3946 store and atomic without 3947 return in order. 3948 - If 1 vmcnt reports completion 3949 of load, atomic with return 3950 and sample instructions in 3951 order, and the vscnt reports 3952 the completion of store and 3953 atomic without return in order. 3954 3955 Used by CP to set up 3956 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3957 31 1 bit FWD_PROGRESS GFX6-GFX9 3958 Reserved, must be 0. 3959 GFX10 3960 - If 0 execute SIMD wavefronts 3961 using oldest first policy. 3962 - If 1 execute SIMD wavefronts to 3963 ensure wavefronts will make some 3964 forward progress. 3965 3966 Used by CP to set up 3967 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3968 32 **Total size 4 bytes** 3969 ======= =================================================================================================================== 3970 3971.. 3972 3973 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3974 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3975 3976 ======= ======= =============================== =========================================================================== 3977 Bits Size Field Name Description 3978 ======= ======= =============================== =========================================================================== 3979 0 1 bit ENABLE_PRIVATE_SEGMENT Enable the setup of the 3980 private segment. 3981 3982 In addition, enable the 3983 setup of the SGPR 3984 wavefront scratch offset 3985 system register (see 3986 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3987 3988 Used by CP to set up 3989 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3990 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3991 user data registers 3992 requested. This number must 3993 match the number of user 3994 data registers enabled. 3995 3996 Used by CP to set up 3997 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 3998 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 3999 4000 This bit represents 4001 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4002 which is set by the CP if 4003 the runtime has installed a 4004 trap handler. 4005 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4006 system SGPR register for 4007 the work-group id in the X 4008 dimension (see 4009 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4010 4011 Used by CP to set up 4012 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4013 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4014 system SGPR register for 4015 the work-group id in the Y 4016 dimension (see 4017 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4018 4019 Used by CP to set up 4020 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4021 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4022 system SGPR register for 4023 the work-group id in the Z 4024 dimension (see 4025 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4026 4027 Used by CP to set up 4028 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4029 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4030 system SGPR register for 4031 work-group information (see 4032 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4033 4034 Used by CP to set up 4035 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4036 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4037 VGPR system registers used 4038 for the work-item ID. 4039 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4040 defines the values. 4041 4042 Used by CP to set up 4043 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4044 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4045 4046 Wavefront starts execution 4047 with address watch 4048 exceptions enabled which 4049 are generated when L1 has 4050 witnessed a thread access 4051 an *address of 4052 interest*. 4053 4054 CP is responsible for 4055 filling in the address 4056 watch bit in 4057 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4058 according to what the 4059 runtime requests. 4060 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4061 4062 Wavefront starts execution 4063 with memory violation 4064 exceptions exceptions 4065 enabled which are generated 4066 when a memory violation has 4067 occurred for this wavefront from 4068 L1 or LDS 4069 (write-to-read-only-memory, 4070 mis-aligned atomic, LDS 4071 address out of range, 4072 illegal address, etc.). 4073 4074 CP sets the memory 4075 violation bit in 4076 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4077 according to what the 4078 runtime requests. 4079 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4080 4081 CP uses the rounded value 4082 from the dispatch packet, 4083 not this value, as the 4084 dispatch may contain 4085 dynamically allocated group 4086 segment memory. CP writes 4087 directly to 4088 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4089 4090 Amount of group segment 4091 (LDS) to allocate for each 4092 work-group. Granularity is 4093 device specific: 4094 4095 GFX6 4096 roundup(lds-size / (64 * 4)) 4097 GFX7-GFX10 4098 roundup(lds-size / (128 * 4)) 4099 4100 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4101 _INVALID_OPERATION with specified exceptions 4102 enabled. 4103 4104 Used by CP to set up 4105 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4106 (set from bits 0..6). 4107 4108 IEEE 754 FP Invalid 4109 Operation 4110 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4111 _SOURCE input operands is a 4112 denormal number 4113 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4114 _DIVISION_BY_ZERO Zero 4115 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4116 _OVERFLOW 4117 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4118 _UNDERFLOW 4119 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4120 _INEXACT 4121 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4122 _ZERO (rcp_iflag_f32 instruction 4123 only) 4124 31 1 bit Reserved, must be 0. 4125 32 **Total size 4 bytes.** 4126 ======= =================================================================================================================== 4127 4128.. 4129 4130 .. table:: compute_pgm_rsrc3 for GFX90A 4131 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4132 4133 ======= ======= =============================== =========================================================================== 4134 Bits Size Field Name Description 4135 ======= ======= =============================== =========================================================================== 4136 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4137 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4138 63 - accum-offset = 256. 4139 6:15 10 Reserved, must be 0. 4140 bits 4141 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4142 launched in the same CU. 4143 - If 1 the waves of a work-group can be 4144 launched in different CUs. The waves 4145 cannot use S_BARRIER or LDS. 4146 17:31 15 Reserved, must be 0. 4147 bits 4148 32 **Total size 4 bytes.** 4149 ======= =================================================================================================================== 4150 4151.. 4152 4153 .. table:: compute_pgm_rsrc3 for GFX10 4154 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4155 4156 ======= ======= =============================== =========================================================================== 4157 Bits Size Field Name Description 4158 ======= ======= =============================== =========================================================================== 4159 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4160 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4161 31:4 28 Reserved, must be 0. 4162 bits 4163 32 **Total size 4 bytes.** 4164 ======= =================================================================================================================== 4165 4166.. 4167 4168 .. table:: Floating Point Rounding Mode Enumeration Values 4169 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4170 4171 ====================================== ===== ============================== 4172 Enumeration Name Value Description 4173 ====================================== ===== ============================== 4174 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4175 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4176 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4177 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4178 ====================================== ===== ============================== 4179 4180.. 4181 4182 .. table:: Floating Point Denorm Mode Enumeration Values 4183 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4184 4185 ====================================== ===== ============================== 4186 Enumeration Name Value Description 4187 ====================================== ===== ============================== 4188 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4189 Denorms 4190 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4191 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4192 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4193 ====================================== ===== ============================== 4194 4195.. 4196 4197 .. table:: System VGPR Work-Item ID Enumeration Values 4198 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4199 4200 ======================================== ===== ============================ 4201 Enumeration Name Value Description 4202 ======================================== ===== ============================ 4203 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4204 ID. 4205 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4206 dimensions ID. 4207 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4208 dimensions ID. 4209 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4210 ======================================== ===== ============================ 4211 4212.. _amdgpu-amdhsa-initial-kernel-execution-state: 4213 4214Initial Kernel Execution State 4215~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4216 4217This section defines the register state that will be set up by the packet 4218processor prior to the start of execution of every wavefront. This is limited by 4219the constraints of the hardware controllers of CP/ADC/SPI. 4220 4221The order of the SGPR registers is defined, but the compiler can specify which 4222ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4223fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4224for enabled registers are dense starting at SGPR0: the first enabled register is 4225SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4226an SGPR number. 4227 4228The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4229all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4230using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4231actually initialized. These are then immediately followed by the System SGPRs 4232that are set up by ADC/SPI and can have different values for each wavefront of 4233the grid dispatch. 4234 4235SGPR register initial state is defined in 4236:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4237 4238 .. table:: SGPR Register Set Up Order 4239 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4240 4241 ========== ========================== ====== ============================== 4242 SGPR Order Name Number Description 4243 (kernel descriptor enable of 4244 field) SGPRs 4245 ========== ========================== ====== ============================== 4246 First Private Segment Buffer 4 See 4247 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4248 _segment_buffer) 4249 then Dispatch Ptr 2 64-bit address of AQL dispatch 4250 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4251 actually executing. 4252 then Queue Ptr 2 64-bit address of amd_queue_t 4253 (enable_sgpr_queue_ptr) object for AQL queue on which 4254 the dispatch packet was 4255 queued. 4256 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4257 (enable_sgpr_kernarg segment. This is directly 4258 _segment_ptr) copied from the 4259 kernarg_address in the kernel 4260 dispatch packet. 4261 4262 Having CP load it once avoids 4263 loading it at the beginning of 4264 every wavefront. 4265 then Dispatch Id 2 64-bit Dispatch ID of the 4266 (enable_sgpr_dispatch_id) dispatch packet being 4267 executed. 4268 then Flat Scratch Init 2 See 4269 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4270 _init) 4271 then Private Segment Size 1 The 32-bit byte size of a 4272 (enable_sgpr_private single work-item's memory 4273 _segment_size) allocation. This is the 4274 value from the kernel 4275 dispatch packet Private 4276 Segment Byte Size rounded up 4277 by CP to a multiple of 4278 DWORD. 4279 4280 Having CP load it once avoids 4281 loading it at the beginning of 4282 every wavefront. 4283 4284 This is not used for 4285 GFX7-GFX8 since it is the same 4286 value as the second SGPR of 4287 Flat Scratch Init. However, it 4288 may be needed for GFX9-GFX10 which 4289 changes the meaning of the 4290 Flat Scratch Init value. 4291 then Work-Group Id X 1 32-bit work-group id in X 4292 (enable_sgpr_workgroup_id dimension of grid for 4293 _X) wavefront. 4294 then Work-Group Id Y 1 32-bit work-group id in Y 4295 (enable_sgpr_workgroup_id dimension of grid for 4296 _Y) wavefront. 4297 then Work-Group Id Z 1 32-bit work-group id in Z 4298 (enable_sgpr_workgroup_id dimension of grid for 4299 _Z) wavefront. 4300 then Work-Group Info 1 {first_wavefront, 14'b0000, 4301 (enable_sgpr_workgroup ordered_append_term[10:0], 4302 _info) threadgroup_size_in_wavefronts[5:0]} 4303 then Scratch Wavefront Offset 1 See 4304 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4305 _segment_wavefront_offset) and 4306 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4307 ========== ========================== ====== ============================== 4308 4309The order of the VGPR registers is defined, but the compiler can specify which 4310ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4311fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4312for enabled registers are dense starting at VGPR0: the first enabled register is 4313VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4314VGPR number. 4315 4316There are different methods used for the VGPR initial state: 4317 4318* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4319 specifies otherwise, a separate VGPR register is used per work-item ID. The 4320 VGPR register initial state for this method is defined in 4321 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4322* If *Target Properties* column of :ref:`amdgpu-processor-table` 4323 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4324 for all work-item IDs. The register layout for this method is defined in 4325 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4326 4327 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4328 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4329 4330 ========== ========================== ====== ============================== 4331 VGPR Order Name Number Description 4332 (kernel descriptor enable of 4333 field) VGPRs 4334 ========== ========================== ====== ============================== 4335 First Work-Item Id X 1 32-bit work-item id in X 4336 (Always initialized) dimension of work-group for 4337 wavefront lane. 4338 then Work-Item Id Y 1 32-bit work-item id in Y 4339 (enable_vgpr_workitem_id dimension of work-group for 4340 > 0) wavefront lane. 4341 then Work-Item Id Z 1 32-bit work-item id in Z 4342 (enable_vgpr_workitem_id dimension of work-group for 4343 > 1) wavefront lane. 4344 ========== ========================== ====== ============================== 4345 4346.. 4347 4348 .. table:: Register Layout for Packed Work-Item ID Method 4349 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4350 4351 ======= ======= ================ ========================================= 4352 Bits Size Field Name Description 4353 ======= ======= ================ ========================================= 4354 0:9 10 bits Work-Item Id X Work-item id in X 4355 dimension of work-group for 4356 wavefront lane. 4357 4358 Always initialized. 4359 4360 10:19 10 bits Work-Item Id Y Work-item id in Y 4361 dimension of work-group for 4362 wavefront lane. 4363 4364 Initialized if enable_vgpr_workitem_id > 4365 0, otherwise set to 0. 4366 20:29 10 bits Work-Item Id Z Work-item id in Z 4367 dimension of work-group for 4368 wavefront lane. 4369 4370 Initialized if enable_vgpr_workitem_id > 4371 1, otherwise set to 0. 4372 30:31 2 bits Reserved, set to 0. 4373 ======= ======= ================ ========================================= 4374 4375The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4376 43771. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4378 registers. 43792. Work-group Id registers X, Y, Z are set by ADC which supports any 4380 combination including none. 43813. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4382 its value cannot be included with the flat scratch init value which is per 4383 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 43844. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4385 or (X, Y, Z). 43865. Flat Scratch register pair initialization is described in 4387 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4388 4389The global segment can be accessed either using buffer instructions (GFX6 which 4390has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4391instructions (GFX9-GFX10). 4392 4393If buffer operations are used, then the compiler can generate a V# with the 4394following properties: 4395 4396* base address of 0 4397* no swizzle 4398* ATC: 1 if IOMMU present (such as APU) 4399* ptr64: 1 4400* MTYPE set to support memory coherence that matches the runtime (such as CC for 4401 APU and NC for dGPU). 4402 4403.. _amdgpu-amdhsa-kernel-prolog: 4404 4405Kernel Prolog 4406~~~~~~~~~~~~~ 4407 4408The compiler performs initialization in the kernel prologue depending on the 4409target and information about things like stack usage in the kernel and called 4410functions. Some of this initialization requires the compiler to request certain 4411User and System SGPRs be present in the 4412:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4413:ref:`amdgpu-amdhsa-kernel-descriptor`. 4414 4415.. _amdgpu-amdhsa-kernel-prolog-cfi: 4416 4417CFI 4418+++ 4419 44201. The CFI return address is undefined. 4421 44222. The CFI CFA is defined using an expression which evaluates to a location 4423 description that comprises one memory location description for the 4424 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4425 4426.. _amdgpu-amdhsa-kernel-prolog-m0: 4427 4428M0 4429++ 4430 4431GFX6-GFX8 4432 The M0 register must be initialized with a value at least the total LDS size 4433 if the kernel may access LDS via DS or flat operations. Total LDS size is 4434 available in dispatch packet. For M0, it is also possible to use maximum 4435 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4436 GFX7-GFX8). 4437GFX9-GFX10 4438 The M0 register is not used for range checking LDS accesses and so does not 4439 need to be initialized in the prolog. 4440 4441.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4442 4443Stack Pointer 4444+++++++++++++ 4445 4446If the kernel has function calls it must set up the ABI stack pointer described 4447in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4448SGPR32 to the unswizzled scratch offset of the address past the last local 4449allocation. 4450 4451.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4452 4453Frame Pointer 4454+++++++++++++ 4455 4456If the kernel needs a frame pointer for the reasons defined in 4457``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4458kernel prolog. If a frame pointer is not required then all uses of the frame 4459pointer are replaced with immediate ``0`` offsets. 4460 4461.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4462 4463Flat Scratch 4464++++++++++++ 4465 4466There are different methods used for initializing flat scratch: 4467 4468* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4469 specifies *Does not support generic address space*: 4470 4471 Flat scratch is not supported and there is no flat scratch register pair. 4472 4473* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4474 specifies *Offset flat scratch*: 4475 4476 If the kernel or any function it calls may use flat operations to access 4477 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4478 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4479 Scratch Wavefront Offset SGPR registers (see 4480 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4481 4482 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4483 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4484 being managed by SPI for the queue executing the kernel dispatch. This is 4485 the same value used in the Scratch Segment Buffer V# base address. 4486 4487 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4488 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4489 4490 The prolog must add the value of Scratch Wavefront Offset to get the 4491 wavefront's byte scratch backing memory offset from 4492 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4493 4494 The Scratch Wavefront Offset must also be used as an offset with Private 4495 segment address when using the Scratch Segment Buffer. 4496 4497 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4498 shifted by 8 before moving into FLAT_SCRATCH_HI. 4499 4500 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4501 SGPRn is the highest numbered SGPR allocated to the wavefront). 4502 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4503 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4504 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4505 aperture. 4506 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4507 work-items scratch memory usage. 4508 4509 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4510 checks that the value in the kernel dispatch packet Private Segment Byte 4511 Size is not larger and requests the runtime to increase the queue's scratch 4512 size if necessary. 4513 4514 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4515 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4516 loading it at the beginning of every wavefront. 4517 4518 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4519 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4520 in flat memory instructions. 4521 4522* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4523 specifies *Absolute flat scratch*: 4524 4525 If the kernel or any function it calls may use flat operations to access 4526 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4527 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4528 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4529 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4530 4531 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4532 memory being managed by SPI for the queue executing the kernel dispatch. 4533 4534 CP obtains this from the runtime. 4535 4536 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4537 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4538 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4539 memory instructions. 4540 4541 The Scratch Wavefront Offset must also be used as an offset with Private 4542 segment address when using the Scratch Segment Buffer (see 4543 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4544 4545.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4546 4547Private Segment Buffer 4548++++++++++++++++++++++ 4549 4550Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4551that are used as a V# to access scratch. CP uses the value provided by the 4552runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4553access the private memory space using a segment address. See 4554:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4555 4556The scratch V# is a four-aligned SGPR and always selected for the kernel as 4557follows: 4558 4559 - If it is known during instruction selection that there is stack usage, 4560 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4561 optimizations are disabled (``-O0``), if stack objects already exist (for 4562 locals, etc.), or if there are any function calls. 4563 4564 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4565 are reserved for the tentative scratch V#. These will be used if it is 4566 determined that spilling is needed. 4567 4568 - If no use is made of the tentative scratch V#, then it is unreserved, 4569 and the register count is determined ignoring it. 4570 - If use is made of the tentative scratch V#, then its register numbers 4571 are shifted to the first four-aligned SGPR index after the highest one 4572 allocated by the register allocator, and all uses are updated. The 4573 register count includes them in the shifted location. 4574 - In either case, if the processor has the SGPR allocation bug, the 4575 tentative allocation is not shifted or unreserved in order to ensure 4576 the register count is higher to workaround the bug. 4577 4578 .. note:: 4579 4580 This approach of using a tentative scratch V# and shifting the register 4581 numbers if used avoids having to perform register allocation a second 4582 time if the tentative V# is eliminated. This is more efficient and 4583 avoids the problem that the second register allocation may perform 4584 spilling which will fail as there is no longer a scratch V#. 4585 4586When the kernel prolog code is being emitted it is known whether the scratch V# 4587described above is actually used. If it is, the prolog code must set it up by 4588copying the Private Segment Buffer to the scratch V# registers and then adding 4589the Private Segment Wavefront Offset to the queue base address in the V#. The 4590result is a V# with a base address pointing to the beginning of the wavefront 4591scratch backing memory. 4592 4593The Private Segment Buffer is always requested, but the Private Segment 4594Wavefront Offset is only requested if it is used (see 4595:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4596 4597.. _amdgpu-amdhsa-memory-model: 4598 4599Memory Model 4600~~~~~~~~~~~~ 4601 4602This section describes the mapping of the LLVM memory model onto AMDGPU machine 4603code (see :ref:`memmodel`). 4604 4605The AMDGPU backend supports the memory synchronization scopes specified in 4606:ref:`amdgpu-memory-scopes`. 4607 4608The code sequences used to implement the memory model specify the order of 4609instructions that a single thread must execute. The ``s_waitcnt`` and cache 4610management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4611to other memory instructions executed by the same thread. This allows them to be 4612moved earlier or later which can allow them to be combined with other instances 4613of the same instruction, or hoisted/sunk out of loops to improve performance. 4614Only the instructions related to the memory model are given; additional 4615``s_waitcnt`` instructions are required to ensure registers are defined before 4616being used. These may be able to be combined with the memory model ``s_waitcnt`` 4617instructions as described above. 4618 4619The AMDGPU backend supports the following memory models: 4620 4621 HSA Memory Model [HSA]_ 4622 The HSA memory model uses a single happens-before relation for all address 4623 spaces (see :ref:`amdgpu-address-spaces`). 4624 OpenCL Memory Model [OpenCL]_ 4625 The OpenCL memory model which has separate happens-before relations for the 4626 global and local address spaces. Only a fence specifying both global and 4627 local address space, and seq_cst instructions join the relationships. Since 4628 the LLVM ``memfence`` instruction does not allow an address space to be 4629 specified the OpenCL fence has to conservatively assume both local and 4630 global address space was specified. However, optimizations can often be 4631 done to eliminate the additional ``s_waitcnt`` instructions when there are 4632 no intervening memory instructions which access the corresponding address 4633 space. The code sequences in the table indicate what can be omitted for the 4634 OpenCL memory. The target triple environment is used to determine if the 4635 source language is OpenCL (see :ref:`amdgpu-opencl`). 4636 4637``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4638operations. 4639 4640``buffer/global/flat_load/store/atomic`` instructions to global memory are 4641termed vector memory operations. 4642 4643Private address space uses ``buffer_load/store`` using the scratch V# 4644(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4645is accessing the memory, atomic memory orderings are not meaningful, and all 4646accesses are treated as non-atomic. 4647 4648Constant address space uses ``buffer/global_load`` instructions (or equivalent 4649scalar memory instructions). Since the constant address space contents do not 4650change during the execution of a kernel dispatch it is not legal to perform 4651stores, and atomic memory orderings are not meaningful, and all accesses are 4652treated as non-atomic. 4653 4654A memory synchronization scope wider than work-group is not meaningful for the 4655group (LDS) address space and is treated as work-group. 4656 4657The memory model does not support the region address space which is treated as 4658non-atomic. 4659 4660Acquire memory ordering is not meaningful on store atomic instructions and is 4661treated as non-atomic. 4662 4663Release memory ordering is not meaningful on load atomic instructions and is 4664treated a non-atomic. 4665 4666Acquire-release memory ordering is not meaningful on load or store atomic 4667instructions and is treated as acquire and release respectively. 4668 4669The memory order also adds the single thread optimization constraints defined in 4670table 4671:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4672 4673 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4674 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4675 4676 ============ ============================================================== 4677 LLVM Memory Optimization Constraints 4678 Ordering 4679 ============ ============================================================== 4680 unordered *none* 4681 monotonic *none* 4682 acquire - If a load atomic/atomicrmw then no following load/load 4683 atomic/store/store atomic/atomicrmw/fence instruction can be 4684 moved before the acquire. 4685 - If a fence then same as load atomic, plus no preceding 4686 associated fence-paired-atomic can be moved after the fence. 4687 release - If a store atomic/atomicrmw then no preceding load/load 4688 atomic/store/store atomic/atomicrmw/fence instruction can be 4689 moved after the release. 4690 - If a fence then same as store atomic, plus no following 4691 associated fence-paired-atomic can be moved before the 4692 fence. 4693 acq_rel Same constraints as both acquire and release. 4694 seq_cst - If a load atomic then same constraints as acquire, plus no 4695 preceding sequentially consistent load atomic/store 4696 atomic/atomicrmw/fence instruction can be moved after the 4697 seq_cst. 4698 - If a store atomic then the same constraints as release, plus 4699 no following sequentially consistent load atomic/store 4700 atomic/atomicrmw/fence instruction can be moved before the 4701 seq_cst. 4702 - If an atomicrmw/fence then same constraints as acq_rel. 4703 ============ ============================================================== 4704 4705The code sequences used to implement the memory model are defined in the 4706following sections: 4707 4708* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4709* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4710* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4711 4712.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4713 4714Memory Model GFX6-GFX9 4715++++++++++++++++++++++ 4716 4717For GFX6-GFX9: 4718 4719* Each agent has multiple shader arrays (SA). 4720* Each SA has multiple compute units (CU). 4721* Each CU has multiple SIMDs that execute wavefronts. 4722* The wavefronts for a single work-group are executed in the same CU but may be 4723 executed by different SIMDs. 4724* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4725 executing on it. 4726* All LDS operations of a CU are performed as wavefront wide operations in a 4727 global order and involve no caching. Completion is reported to a wavefront in 4728 execution order. 4729* The LDS memory has multiple request queues shared by the SIMDs of a 4730 CU. Therefore, the LDS operations performed by different wavefronts of a 4731 work-group can be reordered relative to each other, which can result in 4732 reordering the visibility of vector memory operations with respect to LDS 4733 operations of other wavefronts in the same work-group. A ``s_waitcnt 4734 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4735 vector memory operations between wavefronts of a work-group, but not between 4736 operations performed by the same wavefront. 4737* The vector memory operations are performed as wavefront wide operations and 4738 completion is reported to a wavefront in execution order. The exception is 4739 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4740 vector memory order if they access LDS memory, and out of LDS operation order 4741 if they access global memory. 4742* The vector memory operations access a single vector L1 cache shared by all 4743 SIMDs a CU. Therefore, no special action is required for coherence between the 4744 lanes of a single wavefront, or for coherence between wavefronts in the same 4745 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4746 wavefronts executing in different work-groups as they may be executing on 4747 different CUs. 4748* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4749 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4750 scalar operations are used in a restricted way so do not impact the memory 4751 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4752* The vector and scalar memory operations use an L2 cache shared by all CUs on 4753 the same agent. 4754* The L2 cache has independent channels to service disjoint ranges of virtual 4755 addresses. 4756* Each CU has a separate request queue per channel. Therefore, the vector and 4757 scalar memory operations performed by wavefronts executing in different 4758 work-groups (which may be executing on different CUs) of an agent can be 4759 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4760 ensure synchronization between vector memory operations of different CUs. It 4761 ensures a previous vector memory operation has completed before executing a 4762 subsequent vector memory or LDS operation and so can be used to meet the 4763 requirements of acquire and release. 4764* The L2 cache can be kept coherent with other agents on some targets, or ranges 4765 of virtual addresses can be set up to bypass it to ensure system coherence. 4766 4767Scalar memory operations are only used to access memory that is proven to not 4768change during the execution of the kernel dispatch. This includes constant 4769address space and global address space for program scope ``const`` variables. 4770Therefore, the kernel machine code does not have to maintain the scalar cache to 4771ensure it is coherent with the vector caches. The scalar and vector caches are 4772invalidated between kernel dispatches by CP since constant address space data 4773may change between kernel dispatch executions. See 4774:ref:`amdgpu-amdhsa-memory-spaces`. 4775 4776The one exception is if scalar writes are used to spill SGPR registers. In this 4777case the AMDGPU backend ensures the memory location used to spill is never 4778accessed by vector memory operations at the same time. If scalar writes are used 4779then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4780return since the locations may be used for vector memory instructions by a 4781future wavefront that uses the same scratch area, or a function call that 4782creates a frame at the same address, respectively. There is no need for a 4783``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4784 4785For kernarg backing memory: 4786 4787* CP invalidates the L1 cache at the start of each kernel dispatch. 4788* On dGPU the kernarg backing memory is allocated in host memory accessed as 4789 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4790 causes it to be treated as non-volatile and so is not invalidated by 4791 ``*_vol``. 4792* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4793 and so the L2 cache will be coherent with the CPU and other agents. 4794 4795Scratch backing memory (which is used for the private address space) is accessed 4796with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4797only accessed by a single thread, and is always write-before-read, there is 4798never a need to invalidate these entries from the L1 cache. Hence all cache 4799invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4800 4801The code sequences used to implement the memory model for GFX6-GFX9 are defined 4802in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4803 4804 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4805 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4806 4807 ============ ============ ============== ========== ================================ 4808 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4809 Ordering Sync Scope Address GFX6-GFX9 4810 Space 4811 ============ ============ ============== ========== ================================ 4812 **Non-Atomic** 4813 ------------------------------------------------------------------------------------ 4814 load *none* *none* - global - !volatile & !nontemporal 4815 - generic 4816 - private 1. buffer/global/flat_load 4817 - constant 4818 - !volatile & nontemporal 4819 4820 1. buffer/global/flat_load 4821 glc=1 slc=1 4822 4823 - volatile 4824 4825 1. buffer/global/flat_load 4826 glc=1 4827 2. s_waitcnt vmcnt(0) 4828 4829 - Must happen before 4830 any following volatile 4831 global/generic 4832 load/store. 4833 - Ensures that 4834 volatile 4835 operations to 4836 different 4837 addresses will not 4838 be reordered by 4839 hardware. 4840 4841 load *none* *none* - local 1. ds_load 4842 store *none* *none* - global - !volatile & !nontemporal 4843 - generic 4844 - private 1. buffer/global/flat_store 4845 - constant 4846 - !volatile & nontemporal 4847 4848 1. buffer/global/flat_store 4849 glc=1 slc=1 4850 4851 - volatile 4852 4853 1. buffer/global/flat_store 4854 2. s_waitcnt vmcnt(0) 4855 4856 - Must happen before 4857 any following volatile 4858 global/generic 4859 load/store. 4860 - Ensures that 4861 volatile 4862 operations to 4863 different 4864 addresses will not 4865 be reordered by 4866 hardware. 4867 4868 store *none* *none* - local 1. ds_store 4869 **Unordered Atomic** 4870 ------------------------------------------------------------------------------------ 4871 load atomic unordered *any* *any* *Same as non-atomic*. 4872 store atomic unordered *any* *any* *Same as non-atomic*. 4873 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4874 **Monotonic Atomic** 4875 ------------------------------------------------------------------------------------ 4876 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4877 - wavefront - local 4878 - workgroup - generic 4879 load atomic monotonic - agent - global 1. buffer/global/flat_load 4880 - system - generic glc=1 4881 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4882 - wavefront - generic 4883 - workgroup 4884 - agent 4885 - system 4886 store atomic monotonic - singlethread - local 1. ds_store 4887 - wavefront 4888 - workgroup 4889 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4890 - wavefront - generic 4891 - workgroup 4892 - agent 4893 - system 4894 atomicrmw monotonic - singlethread - local 1. ds_atomic 4895 - wavefront 4896 - workgroup 4897 **Acquire Atomic** 4898 ------------------------------------------------------------------------------------ 4899 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4900 - wavefront - local 4901 - generic 4902 load atomic acquire - workgroup - global 1. buffer/global_load 4903 load atomic acquire - workgroup - local 1. ds/flat_load 4904 - generic 2. s_waitcnt lgkmcnt(0) 4905 4906 - If OpenCL, omit. 4907 - Must happen before 4908 any following 4909 global/generic 4910 load/load 4911 atomic/store/store 4912 atomic/atomicrmw. 4913 - Ensures any 4914 following global 4915 data read is no 4916 older than a local load 4917 atomic value being 4918 acquired. 4919 4920 load atomic acquire - agent - global 1. buffer/global_load 4921 - system glc=1 4922 2. s_waitcnt vmcnt(0) 4923 4924 - Must happen before 4925 following 4926 buffer_wbinvl1_vol. 4927 - Ensures the load 4928 has completed 4929 before invalidating 4930 the cache. 4931 4932 3. buffer_wbinvl1_vol 4933 4934 - Must happen before 4935 any following 4936 global/generic 4937 load/load 4938 atomic/atomicrmw. 4939 - Ensures that 4940 following 4941 loads will not see 4942 stale global data. 4943 4944 load atomic acquire - agent - generic 1. flat_load glc=1 4945 - system 2. s_waitcnt vmcnt(0) & 4946 lgkmcnt(0) 4947 4948 - If OpenCL omit 4949 lgkmcnt(0). 4950 - Must happen before 4951 following 4952 buffer_wbinvl1_vol. 4953 - Ensures the flat_load 4954 has completed 4955 before invalidating 4956 the cache. 4957 4958 3. buffer_wbinvl1_vol 4959 4960 - Must happen before 4961 any following 4962 global/generic 4963 load/load 4964 atomic/atomicrmw. 4965 - Ensures that 4966 following loads 4967 will not see stale 4968 global data. 4969 4970 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 4971 - wavefront - local 4972 - generic 4973 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 4974 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 4975 - generic 2. s_waitcnt lgkmcnt(0) 4976 4977 - If OpenCL, omit. 4978 - Must happen before 4979 any following 4980 global/generic 4981 load/load 4982 atomic/store/store 4983 atomic/atomicrmw. 4984 - Ensures any 4985 following global 4986 data read is no 4987 older than a local 4988 atomicrmw value 4989 being acquired. 4990 4991 atomicrmw acquire - agent - global 1. buffer/global_atomic 4992 - system 2. s_waitcnt vmcnt(0) 4993 4994 - Must happen before 4995 following 4996 buffer_wbinvl1_vol. 4997 - Ensures the 4998 atomicrmw has 4999 completed before 5000 invalidating the 5001 cache. 5002 5003 3. buffer_wbinvl1_vol 5004 5005 - Must happen before 5006 any following 5007 global/generic 5008 load/load 5009 atomic/atomicrmw. 5010 - Ensures that 5011 following loads 5012 will not see stale 5013 global data. 5014 5015 atomicrmw acquire - agent - generic 1. flat_atomic 5016 - system 2. s_waitcnt vmcnt(0) & 5017 lgkmcnt(0) 5018 5019 - If OpenCL, omit 5020 lgkmcnt(0). 5021 - Must happen before 5022 following 5023 buffer_wbinvl1_vol. 5024 - Ensures the 5025 atomicrmw has 5026 completed before 5027 invalidating the 5028 cache. 5029 5030 3. buffer_wbinvl1_vol 5031 5032 - Must happen before 5033 any following 5034 global/generic 5035 load/load 5036 atomic/atomicrmw. 5037 - Ensures that 5038 following loads 5039 will not see stale 5040 global data. 5041 5042 fence acquire - singlethread *none* *none* 5043 - wavefront 5044 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5045 5046 - If OpenCL and 5047 address space is 5048 not generic, omit. 5049 - However, since LLVM 5050 currently has no 5051 address space on 5052 the fence need to 5053 conservatively 5054 always generate. If 5055 fence had an 5056 address space then 5057 set to address 5058 space of OpenCL 5059 fence flag, or to 5060 generic if both 5061 local and global 5062 flags are 5063 specified. 5064 - Must happen after 5065 any preceding 5066 local/generic load 5067 atomic/atomicrmw 5068 with an equal or 5069 wider sync scope 5070 and memory ordering 5071 stronger than 5072 unordered (this is 5073 termed the 5074 fence-paired-atomic). 5075 - Must happen before 5076 any following 5077 global/generic 5078 load/load 5079 atomic/store/store 5080 atomic/atomicrmw. 5081 - Ensures any 5082 following global 5083 data read is no 5084 older than the 5085 value read by the 5086 fence-paired-atomic. 5087 5088 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5089 - system vmcnt(0) 5090 5091 - If OpenCL and 5092 address space is 5093 not generic, omit 5094 lgkmcnt(0). 5095 - However, since LLVM 5096 currently has no 5097 address space on 5098 the fence need to 5099 conservatively 5100 always generate 5101 (see comment for 5102 previous fence). 5103 - Could be split into 5104 separate s_waitcnt 5105 vmcnt(0) and 5106 s_waitcnt 5107 lgkmcnt(0) to allow 5108 them to be 5109 independently moved 5110 according to the 5111 following rules. 5112 - s_waitcnt vmcnt(0) 5113 must happen after 5114 any preceding 5115 global/generic load 5116 atomic/atomicrmw 5117 with an equal or 5118 wider sync scope 5119 and memory ordering 5120 stronger than 5121 unordered (this is 5122 termed the 5123 fence-paired-atomic). 5124 - s_waitcnt lgkmcnt(0) 5125 must happen after 5126 any preceding 5127 local/generic load 5128 atomic/atomicrmw 5129 with an equal or 5130 wider sync scope 5131 and memory ordering 5132 stronger than 5133 unordered (this is 5134 termed the 5135 fence-paired-atomic). 5136 - Must happen before 5137 the following 5138 buffer_wbinvl1_vol. 5139 - Ensures that the 5140 fence-paired atomic 5141 has completed 5142 before invalidating 5143 the 5144 cache. Therefore 5145 any following 5146 locations read must 5147 be no older than 5148 the value read by 5149 the 5150 fence-paired-atomic. 5151 5152 2. buffer_wbinvl1_vol 5153 5154 - Must happen before any 5155 following global/generic 5156 load/load 5157 atomic/store/store 5158 atomic/atomicrmw. 5159 - Ensures that 5160 following loads 5161 will not see stale 5162 global data. 5163 5164 **Release Atomic** 5165 ------------------------------------------------------------------------------------ 5166 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5167 - wavefront - local 5168 - generic 5169 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5170 - generic 5171 - If OpenCL, omit. 5172 - Must happen after 5173 any preceding 5174 local/generic 5175 load/store/load 5176 atomic/store 5177 atomic/atomicrmw. 5178 - Must happen before 5179 the following 5180 store. 5181 - Ensures that all 5182 memory operations 5183 to local have 5184 completed before 5185 performing the 5186 store that is being 5187 released. 5188 5189 2. buffer/global/flat_store 5190 store atomic release - workgroup - local 1. ds_store 5191 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5192 - system - generic vmcnt(0) 5193 5194 - If OpenCL and 5195 address space is 5196 not generic, omit 5197 lgkmcnt(0). 5198 - Could be split into 5199 separate s_waitcnt 5200 vmcnt(0) and 5201 s_waitcnt 5202 lgkmcnt(0) to allow 5203 them to be 5204 independently moved 5205 according to the 5206 following rules. 5207 - s_waitcnt vmcnt(0) 5208 must happen after 5209 any preceding 5210 global/generic 5211 load/store/load 5212 atomic/store 5213 atomic/atomicrmw. 5214 - s_waitcnt lgkmcnt(0) 5215 must happen after 5216 any preceding 5217 local/generic 5218 load/store/load 5219 atomic/store 5220 atomic/atomicrmw. 5221 - Must happen before 5222 the following 5223 store. 5224 - Ensures that all 5225 memory operations 5226 to memory have 5227 completed before 5228 performing the 5229 store that is being 5230 released. 5231 5232 2. buffer/global/flat_store 5233 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5234 - wavefront - local 5235 - generic 5236 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5237 - generic 5238 - If OpenCL, omit. 5239 - Must happen after 5240 any preceding 5241 local/generic 5242 load/store/load 5243 atomic/store 5244 atomic/atomicrmw. 5245 - Must happen before 5246 the following 5247 atomicrmw. 5248 - Ensures that all 5249 memory operations 5250 to local have 5251 completed before 5252 performing the 5253 atomicrmw that is 5254 being released. 5255 5256 2. buffer/global/flat_atomic 5257 atomicrmw release - workgroup - local 1. ds_atomic 5258 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5259 - system - generic vmcnt(0) 5260 5261 - If OpenCL, omit 5262 lgkmcnt(0). 5263 - Could be split into 5264 separate s_waitcnt 5265 vmcnt(0) and 5266 s_waitcnt 5267 lgkmcnt(0) to allow 5268 them to be 5269 independently moved 5270 according to the 5271 following rules. 5272 - s_waitcnt vmcnt(0) 5273 must happen after 5274 any preceding 5275 global/generic 5276 load/store/load 5277 atomic/store 5278 atomic/atomicrmw. 5279 - s_waitcnt lgkmcnt(0) 5280 must happen after 5281 any preceding 5282 local/generic 5283 load/store/load 5284 atomic/store 5285 atomic/atomicrmw. 5286 - Must happen before 5287 the following 5288 atomicrmw. 5289 - Ensures that all 5290 memory operations 5291 to global and local 5292 have completed 5293 before performing 5294 the atomicrmw that 5295 is being released. 5296 5297 2. buffer/global/flat_atomic 5298 fence release - singlethread *none* *none* 5299 - wavefront 5300 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5301 5302 - If OpenCL and 5303 address space is 5304 not generic, omit. 5305 - However, since LLVM 5306 currently has no 5307 address space on 5308 the fence need to 5309 conservatively 5310 always generate. If 5311 fence had an 5312 address space then 5313 set to address 5314 space of OpenCL 5315 fence flag, or to 5316 generic if both 5317 local and global 5318 flags are 5319 specified. 5320 - Must happen after 5321 any preceding 5322 local/generic 5323 load/load 5324 atomic/store/store 5325 atomic/atomicrmw. 5326 - Must happen before 5327 any following store 5328 atomic/atomicrmw 5329 with an equal or 5330 wider sync scope 5331 and memory ordering 5332 stronger than 5333 unordered (this is 5334 termed the 5335 fence-paired-atomic). 5336 - Ensures that all 5337 memory operations 5338 to local have 5339 completed before 5340 performing the 5341 following 5342 fence-paired-atomic. 5343 5344 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5345 - system vmcnt(0) 5346 5347 - If OpenCL and 5348 address space is 5349 not generic, omit 5350 lgkmcnt(0). 5351 - If OpenCL and 5352 address space is 5353 local, omit 5354 vmcnt(0). 5355 - However, since LLVM 5356 currently has no 5357 address space on 5358 the fence need to 5359 conservatively 5360 always generate. If 5361 fence had an 5362 address space then 5363 set to address 5364 space of OpenCL 5365 fence flag, or to 5366 generic if both 5367 local and global 5368 flags are 5369 specified. 5370 - Could be split into 5371 separate s_waitcnt 5372 vmcnt(0) and 5373 s_waitcnt 5374 lgkmcnt(0) to allow 5375 them to be 5376 independently moved 5377 according to the 5378 following rules. 5379 - s_waitcnt vmcnt(0) 5380 must happen after 5381 any preceding 5382 global/generic 5383 load/store/load 5384 atomic/store 5385 atomic/atomicrmw. 5386 - s_waitcnt lgkmcnt(0) 5387 must happen after 5388 any preceding 5389 local/generic 5390 load/store/load 5391 atomic/store 5392 atomic/atomicrmw. 5393 - Must happen before 5394 any following store 5395 atomic/atomicrmw 5396 with an equal or 5397 wider sync scope 5398 and memory ordering 5399 stronger than 5400 unordered (this is 5401 termed the 5402 fence-paired-atomic). 5403 - Ensures that all 5404 memory operations 5405 have 5406 completed before 5407 performing the 5408 following 5409 fence-paired-atomic. 5410 5411 **Acquire-Release Atomic** 5412 ------------------------------------------------------------------------------------ 5413 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5414 - wavefront - local 5415 - generic 5416 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5417 5418 - If OpenCL, omit. 5419 - Must happen after 5420 any preceding 5421 local/generic 5422 load/store/load 5423 atomic/store 5424 atomic/atomicrmw. 5425 - Must happen before 5426 the following 5427 atomicrmw. 5428 - Ensures that all 5429 memory operations 5430 to local have 5431 completed before 5432 performing the 5433 atomicrmw that is 5434 being released. 5435 5436 2. buffer/global_atomic 5437 5438 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5439 2. s_waitcnt lgkmcnt(0) 5440 5441 - If OpenCL, omit. 5442 - Must happen before 5443 any following 5444 global/generic 5445 load/load 5446 atomic/store/store 5447 atomic/atomicrmw. 5448 - Ensures any 5449 following global 5450 data read is no 5451 older than the local load 5452 atomic value being 5453 acquired. 5454 5455 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5456 5457 - If OpenCL, omit. 5458 - Must happen after 5459 any preceding 5460 local/generic 5461 load/store/load 5462 atomic/store 5463 atomic/atomicrmw. 5464 - Must happen before 5465 the following 5466 atomicrmw. 5467 - Ensures that all 5468 memory operations 5469 to local have 5470 completed before 5471 performing the 5472 atomicrmw that is 5473 being released. 5474 5475 2. flat_atomic 5476 3. s_waitcnt lgkmcnt(0) 5477 5478 - If OpenCL, omit. 5479 - Must happen before 5480 any following 5481 global/generic 5482 load/load 5483 atomic/store/store 5484 atomic/atomicrmw. 5485 - Ensures any 5486 following global 5487 data read is no 5488 older than a local load 5489 atomic value being 5490 acquired. 5491 5492 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5493 - system vmcnt(0) 5494 5495 - If OpenCL, omit 5496 lgkmcnt(0). 5497 - Could be split into 5498 separate s_waitcnt 5499 vmcnt(0) and 5500 s_waitcnt 5501 lgkmcnt(0) to allow 5502 them to be 5503 independently moved 5504 according to the 5505 following rules. 5506 - s_waitcnt vmcnt(0) 5507 must happen after 5508 any preceding 5509 global/generic 5510 load/store/load 5511 atomic/store 5512 atomic/atomicrmw. 5513 - s_waitcnt lgkmcnt(0) 5514 must happen after 5515 any preceding 5516 local/generic 5517 load/store/load 5518 atomic/store 5519 atomic/atomicrmw. 5520 - Must happen before 5521 the following 5522 atomicrmw. 5523 - Ensures that all 5524 memory operations 5525 to global have 5526 completed before 5527 performing the 5528 atomicrmw that is 5529 being released. 5530 5531 2. buffer/global_atomic 5532 3. s_waitcnt vmcnt(0) 5533 5534 - Must happen before 5535 following 5536 buffer_wbinvl1_vol. 5537 - Ensures the 5538 atomicrmw has 5539 completed before 5540 invalidating the 5541 cache. 5542 5543 4. buffer_wbinvl1_vol 5544 5545 - Must happen before 5546 any following 5547 global/generic 5548 load/load 5549 atomic/atomicrmw. 5550 - Ensures that 5551 following loads 5552 will not see stale 5553 global data. 5554 5555 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5556 - system vmcnt(0) 5557 5558 - If OpenCL, omit 5559 lgkmcnt(0). 5560 - Could be split into 5561 separate s_waitcnt 5562 vmcnt(0) and 5563 s_waitcnt 5564 lgkmcnt(0) to allow 5565 them to be 5566 independently moved 5567 according to the 5568 following rules. 5569 - s_waitcnt vmcnt(0) 5570 must happen after 5571 any preceding 5572 global/generic 5573 load/store/load 5574 atomic/store 5575 atomic/atomicrmw. 5576 - s_waitcnt lgkmcnt(0) 5577 must happen after 5578 any preceding 5579 local/generic 5580 load/store/load 5581 atomic/store 5582 atomic/atomicrmw. 5583 - Must happen before 5584 the following 5585 atomicrmw. 5586 - Ensures that all 5587 memory operations 5588 to global have 5589 completed before 5590 performing the 5591 atomicrmw that is 5592 being released. 5593 5594 2. flat_atomic 5595 3. s_waitcnt vmcnt(0) & 5596 lgkmcnt(0) 5597 5598 - If OpenCL, omit 5599 lgkmcnt(0). 5600 - Must happen before 5601 following 5602 buffer_wbinvl1_vol. 5603 - Ensures the 5604 atomicrmw has 5605 completed before 5606 invalidating the 5607 cache. 5608 5609 4. buffer_wbinvl1_vol 5610 5611 - Must happen before 5612 any following 5613 global/generic 5614 load/load 5615 atomic/atomicrmw. 5616 - Ensures that 5617 following loads 5618 will not see stale 5619 global data. 5620 5621 fence acq_rel - singlethread *none* *none* 5622 - wavefront 5623 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5624 5625 - If OpenCL and 5626 address space is 5627 not generic, omit. 5628 - However, 5629 since LLVM 5630 currently has no 5631 address space on 5632 the fence need to 5633 conservatively 5634 always generate 5635 (see comment for 5636 previous fence). 5637 - Must happen after 5638 any preceding 5639 local/generic 5640 load/load 5641 atomic/store/store 5642 atomic/atomicrmw. 5643 - Must happen before 5644 any following 5645 global/generic 5646 load/load 5647 atomic/store/store 5648 atomic/atomicrmw. 5649 - Ensures that all 5650 memory operations 5651 to local have 5652 completed before 5653 performing any 5654 following global 5655 memory operations. 5656 - Ensures that the 5657 preceding 5658 local/generic load 5659 atomic/atomicrmw 5660 with an equal or 5661 wider sync scope 5662 and memory ordering 5663 stronger than 5664 unordered (this is 5665 termed the 5666 acquire-fence-paired-atomic) 5667 has completed 5668 before following 5669 global memory 5670 operations. This 5671 satisfies the 5672 requirements of 5673 acquire. 5674 - Ensures that all 5675 previous memory 5676 operations have 5677 completed before a 5678 following 5679 local/generic store 5680 atomic/atomicrmw 5681 with an equal or 5682 wider sync scope 5683 and memory ordering 5684 stronger than 5685 unordered (this is 5686 termed the 5687 release-fence-paired-atomic). 5688 This satisfies the 5689 requirements of 5690 release. 5691 5692 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5693 - system vmcnt(0) 5694 5695 - If OpenCL and 5696 address space is 5697 not generic, omit 5698 lgkmcnt(0). 5699 - However, since LLVM 5700 currently has no 5701 address space on 5702 the fence need to 5703 conservatively 5704 always generate 5705 (see comment for 5706 previous fence). 5707 - Could be split into 5708 separate s_waitcnt 5709 vmcnt(0) and 5710 s_waitcnt 5711 lgkmcnt(0) to allow 5712 them to be 5713 independently moved 5714 according to the 5715 following rules. 5716 - s_waitcnt vmcnt(0) 5717 must happen after 5718 any preceding 5719 global/generic 5720 load/store/load 5721 atomic/store 5722 atomic/atomicrmw. 5723 - s_waitcnt lgkmcnt(0) 5724 must happen after 5725 any preceding 5726 local/generic 5727 load/store/load 5728 atomic/store 5729 atomic/atomicrmw. 5730 - Must happen before 5731 the following 5732 buffer_wbinvl1_vol. 5733 - Ensures that the 5734 preceding 5735 global/local/generic 5736 load 5737 atomic/atomicrmw 5738 with an equal or 5739 wider sync scope 5740 and memory ordering 5741 stronger than 5742 unordered (this is 5743 termed the 5744 acquire-fence-paired-atomic) 5745 has completed 5746 before invalidating 5747 the cache. This 5748 satisfies the 5749 requirements of 5750 acquire. 5751 - Ensures that all 5752 previous memory 5753 operations have 5754 completed before a 5755 following 5756 global/local/generic 5757 store 5758 atomic/atomicrmw 5759 with an equal or 5760 wider sync scope 5761 and memory ordering 5762 stronger than 5763 unordered (this is 5764 termed the 5765 release-fence-paired-atomic). 5766 This satisfies the 5767 requirements of 5768 release. 5769 5770 2. buffer_wbinvl1_vol 5771 5772 - Must happen before 5773 any following 5774 global/generic 5775 load/load 5776 atomic/store/store 5777 atomic/atomicrmw. 5778 - Ensures that 5779 following loads 5780 will not see stale 5781 global data. This 5782 satisfies the 5783 requirements of 5784 acquire. 5785 5786 **Sequential Consistent Atomic** 5787 ------------------------------------------------------------------------------------ 5788 load atomic seq_cst - singlethread - global *Same as corresponding 5789 - wavefront - local load atomic acquire, 5790 - generic except must generated 5791 all instructions even 5792 for OpenCL.* 5793 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5794 - generic 5795 5796 - Must 5797 happen after 5798 preceding 5799 local/generic load 5800 atomic/store 5801 atomic/atomicrmw 5802 with memory 5803 ordering of seq_cst 5804 and with equal or 5805 wider sync scope. 5806 (Note that seq_cst 5807 fences have their 5808 own s_waitcnt 5809 lgkmcnt(0) and so do 5810 not need to be 5811 considered.) 5812 - Ensures any 5813 preceding 5814 sequential 5815 consistent local 5816 memory instructions 5817 have completed 5818 before executing 5819 this sequentially 5820 consistent 5821 instruction. This 5822 prevents reordering 5823 a seq_cst store 5824 followed by a 5825 seq_cst load. (Note 5826 that seq_cst is 5827 stronger than 5828 acquire/release as 5829 the reordering of 5830 load acquire 5831 followed by a store 5832 release is 5833 prevented by the 5834 s_waitcnt of 5835 the release, but 5836 there is nothing 5837 preventing a store 5838 release followed by 5839 load acquire from 5840 completing out of 5841 order. The s_waitcnt 5842 could be placed after 5843 seq_store or before 5844 the seq_load. We 5845 choose the load to 5846 make the s_waitcnt be 5847 as late as possible 5848 so that the store 5849 may have already 5850 completed.) 5851 5852 2. *Following 5853 instructions same as 5854 corresponding load 5855 atomic acquire, 5856 except must generated 5857 all instructions even 5858 for OpenCL.* 5859 load atomic seq_cst - workgroup - local *Same as corresponding 5860 load atomic acquire, 5861 except must generated 5862 all instructions even 5863 for OpenCL.* 5864 5865 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5866 - system - generic vmcnt(0) 5867 5868 - Could be split into 5869 separate s_waitcnt 5870 vmcnt(0) 5871 and s_waitcnt 5872 lgkmcnt(0) to allow 5873 them to be 5874 independently moved 5875 according to the 5876 following rules. 5877 - s_waitcnt lgkmcnt(0) 5878 must happen after 5879 preceding 5880 global/generic load 5881 atomic/store 5882 atomic/atomicrmw 5883 with memory 5884 ordering of seq_cst 5885 and with equal or 5886 wider sync scope. 5887 (Note that seq_cst 5888 fences have their 5889 own s_waitcnt 5890 lgkmcnt(0) and so do 5891 not need to be 5892 considered.) 5893 - s_waitcnt vmcnt(0) 5894 must happen after 5895 preceding 5896 global/generic load 5897 atomic/store 5898 atomic/atomicrmw 5899 with memory 5900 ordering of seq_cst 5901 and with equal or 5902 wider sync scope. 5903 (Note that seq_cst 5904 fences have their 5905 own s_waitcnt 5906 vmcnt(0) and so do 5907 not need to be 5908 considered.) 5909 - Ensures any 5910 preceding 5911 sequential 5912 consistent global 5913 memory instructions 5914 have completed 5915 before executing 5916 this sequentially 5917 consistent 5918 instruction. This 5919 prevents reordering 5920 a seq_cst store 5921 followed by a 5922 seq_cst load. (Note 5923 that seq_cst is 5924 stronger than 5925 acquire/release as 5926 the reordering of 5927 load acquire 5928 followed by a store 5929 release is 5930 prevented by the 5931 s_waitcnt of 5932 the release, but 5933 there is nothing 5934 preventing a store 5935 release followed by 5936 load acquire from 5937 completing out of 5938 order. The s_waitcnt 5939 could be placed after 5940 seq_store or before 5941 the seq_load. We 5942 choose the load to 5943 make the s_waitcnt be 5944 as late as possible 5945 so that the store 5946 may have already 5947 completed.) 5948 5949 2. *Following 5950 instructions same as 5951 corresponding load 5952 atomic acquire, 5953 except must generated 5954 all instructions even 5955 for OpenCL.* 5956 store atomic seq_cst - singlethread - global *Same as corresponding 5957 - wavefront - local store atomic release, 5958 - workgroup - generic except must generated 5959 - agent all instructions even 5960 - system for OpenCL.* 5961 atomicrmw seq_cst - singlethread - global *Same as corresponding 5962 - wavefront - local atomicrmw acq_rel, 5963 - workgroup - generic except must generated 5964 - agent all instructions even 5965 - system for OpenCL.* 5966 fence seq_cst - singlethread *none* *Same as corresponding 5967 - wavefront fence acq_rel, 5968 - workgroup except must generated 5969 - agent all instructions even 5970 - system for OpenCL.* 5971 ============ ============ ============== ========== ================================ 5972 5973.. _amdgpu-amdhsa-memory-model-gfx90a: 5974 5975Memory Model GFX90A 5976+++++++++++++++++++ 5977 5978For GFX90A: 5979 5980* Each agent has multiple shader arrays (SA). 5981* Each SA has multiple compute units (CU). 5982* Each CU has multiple SIMDs that execute wavefronts. 5983* The wavefronts for a single work-group are executed in the same CU but may be 5984 executed by different SIMDs. The exception is when in tgsplit execution mode 5985 when the wavefronts may be executed by different SIMDs in different CUs. 5986* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5987 executing on it. The exception is when in tgsplit execution mode when no LDS 5988 is allocated as wavefronts of the same work-group can be in different CUs. 5989* All LDS operations of a CU are performed as wavefront wide operations in a 5990 global order and involve no caching. Completion is reported to a wavefront in 5991 execution order. 5992* The LDS memory has multiple request queues shared by the SIMDs of a 5993 CU. Therefore, the LDS operations performed by different wavefronts of a 5994 work-group can be reordered relative to each other, which can result in 5995 reordering the visibility of vector memory operations with respect to LDS 5996 operations of other wavefronts in the same work-group. A ``s_waitcnt 5997 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5998 vector memory operations between wavefronts of a work-group, but not between 5999 operations performed by the same wavefront. 6000* The vector memory operations are performed as wavefront wide operations and 6001 completion is reported to a wavefront in execution order. The exception is 6002 that ``flat_load/store/atomic`` instructions can report out of vector memory 6003 order if they access LDS memory, and out of LDS operation order if they access 6004 global memory. 6005* The vector memory operations access a single vector L1 cache shared by all 6006 SIMDs a CU. Therefore: 6007 6008 * No special action is required for coherence between the lanes of a single 6009 wavefront. 6010 6011 * No special action is required for coherence between wavefronts in the same 6012 work-group since they execute on the same CU. The exception is when in 6013 tgsplit execution mode as wavefronts of the same work-group can be in 6014 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6015 the following item. 6016 6017 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6018 executing in different work-groups as they may be executing on different 6019 CUs. 6020 6021* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6022 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6023 scalar operations are used in a restricted way so do not impact the memory 6024 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6025* The vector and scalar memory operations use an L2 cache shared by all CUs on 6026 the same agent. 6027 6028 * The L2 cache has independent channels to service disjoint ranges of virtual 6029 addresses. 6030 * Each CU has a separate request queue per channel. Therefore, the vector and 6031 scalar memory operations performed by wavefronts executing in different 6032 work-groups (which may be executing on different CUs), or the same 6033 work-group if executing in tgsplit mode, of an agent can be reordered 6034 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6035 synchronization between vector memory operations of different CUs. It 6036 ensures a previous vector memory operation has completed before executing a 6037 subsequent vector memory or LDS operation and so can be used to meet the 6038 requirements of acquire and release. 6039 * The L2 cache of one agent can be kept coherent with other agents by using 6040 the MTYPE CC (cache-coherent) with the PTE C-bit for memory local to the L2, 6041 and MTYPE UC (uncached) with the PTE C-bit set for memory not local to the 6042 L2. 6043 6044 * Any local memory cache lines will be automatically invalidated by writes 6045 from CUs associated with other L2 caches, or writes from the CPU, due to 6046 the cache probe caused by coherent requests. Coherent requests are caused 6047 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6048 XGMI, and by PCIe requests that are configured to be coherent requests. 6049 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6050 Subsequent access from the GPU will automatically invalidate or writeback 6051 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6052 * Since all work-groups on the same agent share the same L2, no L2 6053 invalidation or writeback is required for coherence. 6054 * Since local memory reads and writes of work-groups in different agents 6055 access memory using MTYPE CC, no L2 invalidate or writeback is required 6056 for coherence. MTYPE CC causes write through to DRAM and local reads to be 6057 invalidated by remote writes with with the PTE C-bit. 6058 * Since remote memory reads and writes of work-groups in different agents 6059 access memory using MTYPE UC, no L2 invalidate or writeback is required 6060 for coherence. MTYPE UC causes direct accesses to DRAM. 6061 6062 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6063 MTYPE UC (uncached) which bypasses the L2. 6064 6065Scalar memory operations are only used to access memory that is proven to not 6066change during the execution of the kernel dispatch. This includes constant 6067address space and global address space for program scope ``const`` variables. 6068Therefore, the kernel machine code does not have to maintain the scalar cache to 6069ensure it is coherent with the vector caches. The scalar and vector caches are 6070invalidated between kernel dispatches by CP since constant address space data 6071may change between kernel dispatch executions. See 6072:ref:`amdgpu-amdhsa-memory-spaces`. 6073 6074The one exception is if scalar writes are used to spill SGPR registers. In this 6075case the AMDGPU backend ensures the memory location used to spill is never 6076accessed by vector memory operations at the same time. If scalar writes are used 6077then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6078return since the locations may be used for vector memory instructions by a 6079future wavefront that uses the same scratch area, or a function call that 6080creates a frame at the same address, respectively. There is no need for a 6081``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6082 6083For kernarg backing memory: 6084 6085* CP invalidates the L1 cache at the start of each kernel dispatch. 6086* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6087 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6088 cache. This also causes it to be treated as non-volatile and so is not 6089 invalidated by ``*_vol``. 6090* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6091 so the L2 cache will be coherent with the CPU and other agents. 6092 6093Scratch backing memory (which is used for the private address space) is accessed 6094with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6095only accessed by a single thread, and is always write-before-read, there is 6096never a need to invalidate these entries from the L1 cache. Hence all cache 6097invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6098 6099The code sequences used to implement the memory model for GFX90A are defined 6100in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6101 6102 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6103 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6104 6105 ============ ============ ============== ========== ================================ 6106 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6107 Ordering Sync Scope Address GFX90A 6108 Space 6109 ============ ============ ============== ========== ================================ 6110 **Non-Atomic** 6111 ------------------------------------------------------------------------------------ 6112 load *none* *none* - global - !volatile & !nontemporal 6113 - generic 6114 - private 1. buffer/global/flat_load 6115 - constant 6116 - !volatile & nontemporal 6117 6118 1. buffer/global/flat_load 6119 glc=1 slc=1 6120 6121 - volatile 6122 6123 1. buffer/global/flat_load 6124 glc=1 6125 2. s_waitcnt vmcnt(0) 6126 6127 - Must happen before 6128 any following volatile 6129 global/generic 6130 load/store. 6131 - Ensures that 6132 volatile 6133 operations to 6134 different 6135 addresses will not 6136 be reordered by 6137 hardware. 6138 6139 load *none* *none* - local 1. ds_load 6140 store *none* *none* - global - !volatile & !nontemporal 6141 - generic 6142 - private 1. buffer/global/flat_store 6143 - constant 6144 - !volatile & nontemporal 6145 6146 1. buffer/global/flat_store 6147 glc=1 slc=1 6148 6149 - volatile 6150 6151 1. buffer/global/flat_store 6152 2. s_waitcnt vmcnt(0) 6153 6154 - Must happen before 6155 any following volatile 6156 global/generic 6157 load/store. 6158 - Ensures that 6159 volatile 6160 operations to 6161 different 6162 addresses will not 6163 be reordered by 6164 hardware. 6165 6166 store *none* *none* - local 1. ds_store 6167 **Unordered Atomic** 6168 ------------------------------------------------------------------------------------ 6169 load atomic unordered *any* *any* *Same as non-atomic*. 6170 store atomic unordered *any* *any* *Same as non-atomic*. 6171 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6172 **Monotonic Atomic** 6173 ------------------------------------------------------------------------------------ 6174 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6175 - wavefront - generic 6176 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6177 - generic glc=1 6178 6179 - If not TgSplit execution 6180 mode, omit glc=1. 6181 6182 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6183 - wavefront local address space cannot 6184 - workgroup be used.* 6185 6186 1. ds_load 6187 load atomic monotonic - agent - global 1. buffer/global/flat_load 6188 - generic glc=1 6189 load atomic monotonic - system - global 1. buffer/global/flat_load 6190 - generic glc=1 6191 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6192 - wavefront - generic 6193 - workgroup 6194 - agent 6195 store atomic monotonic - system - global 1. buffer/global/flat_store 6196 - generic 6197 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6198 - wavefront local address space cannot 6199 - workgroup be used.* 6200 6201 1. ds_store 6202 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6203 - wavefront - generic 6204 - workgroup 6205 - agent 6206 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6207 - generic 6208 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6209 - wavefront local address space cannot 6210 - workgroup be used.* 6211 6212 1. ds_atomic 6213 **Acquire Atomic** 6214 ------------------------------------------------------------------------------------ 6215 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6216 - wavefront - local 6217 - generic 6218 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6219 6220 - If not TgSplit execution 6221 mode, omit glc=1. 6222 6223 2. s_waitcnt vmcnt(0) 6224 6225 - If not TgSplit execution 6226 mode, omit. 6227 - Must happen before the 6228 following buffer_wbinvl1_vol. 6229 6230 3. buffer_wbinvl1_vol 6231 6232 - If not TgSplit execution 6233 mode, omit. 6234 - Must happen before 6235 any following 6236 global/generic 6237 load/load 6238 atomic/store/store 6239 atomic/atomicrmw. 6240 - Ensures that 6241 following 6242 loads will not see 6243 stale data. 6244 6245 load atomic acquire - workgroup - local *If TgSplit execution mode, 6246 local address space cannot 6247 be used.* 6248 6249 1. ds_load 6250 2. s_waitcnt lgkmcnt(0) 6251 6252 - If OpenCL, omit. 6253 - Must happen before 6254 any following 6255 global/generic 6256 load/load 6257 atomic/store/store 6258 atomic/atomicrmw. 6259 - Ensures any 6260 following global 6261 data read is no 6262 older than the local load 6263 atomic value being 6264 acquired. 6265 6266 load atomic acquire - workgroup - generic 1. flat_load glc=1 6267 6268 - If not TgSplit execution 6269 mode, omit glc=1. 6270 6271 2. s_waitcnt lgkm/vmcnt(0) 6272 6273 - Use lgkmcnt(0) if not 6274 TgSplit execution mode 6275 and vmcnt(0) if TgSplit 6276 execution mode. 6277 - If OpenCL, omit lgkmcnt(0). 6278 - Must happen before 6279 the following 6280 buffer_wbinvl1_vol and any 6281 following global/generic 6282 load/load 6283 atomic/store/store 6284 atomic/atomicrmw. 6285 - Ensures any 6286 following global 6287 data read is no 6288 older than a local load 6289 atomic value being 6290 acquired. 6291 6292 3. buffer_wbinvl1_vol 6293 6294 - If not TgSplit execution 6295 mode, omit. 6296 - Ensures that 6297 following 6298 loads will not see 6299 stale data. 6300 6301 load atomic acquire - agent - global 1. buffer/global_load 6302 glc=1 6303 2. s_waitcnt vmcnt(0) 6304 6305 - Must happen before 6306 following 6307 buffer_wbinvl1_vol. 6308 - Ensures the load 6309 has completed 6310 before invalidating 6311 the cache. 6312 6313 3. buffer_wbinvl1_vol 6314 6315 - Must happen before 6316 any following 6317 global/generic 6318 load/load 6319 atomic/atomicrmw. 6320 - Ensures that 6321 following 6322 loads will not see 6323 stale global data. 6324 6325 load atomic acquire - system - global 1. buffer/global/flat_load 6326 glc=1 6327 2. s_waitcnt vmcnt(0) 6328 6329 - Must happen before 6330 following 6331 buffer_wbinvl1_vol. 6332 - Ensures the load 6333 has completed 6334 before invalidating 6335 the cache. 6336 6337 3. buffer_wbinvl1_vol 6338 6339 - Must happen before 6340 any following 6341 global/generic 6342 load/load 6343 atomic/atomicrmw. 6344 - Ensures that 6345 following 6346 loads will not see 6347 stale L1 global data. 6348 MTYPE RW and CC memory will 6349 never be stale in L2 due to 6350 the memory probes. 6351 6352 load atomic acquire - agent - generic 1. flat_load glc=1 6353 2. s_waitcnt vmcnt(0) & 6354 lgkmcnt(0) 6355 6356 - If TgSplit execution mode, 6357 omit lgkmcnt(0). 6358 - If OpenCL omit 6359 lgkmcnt(0). 6360 - Must happen before 6361 following 6362 buffer_wbinvl1_vol. 6363 - Ensures the flat_load 6364 has completed 6365 before invalidating 6366 the cache. 6367 6368 3. buffer_wbinvl1_vol 6369 6370 - Must happen before 6371 any following 6372 global/generic 6373 load/load 6374 atomic/atomicrmw. 6375 - Ensures that 6376 following loads 6377 will not see stale 6378 global data. 6379 6380 load atomic acquire - system - generic 1. flat_load glc=1 6381 2. s_waitcnt vmcnt(0) & 6382 lgkmcnt(0) 6383 6384 - If TgSplit execution mode, 6385 omit lgkmcnt(0). 6386 - If OpenCL omit 6387 lgkmcnt(0). 6388 - Must happen before 6389 following 6390 buffer_wbinvl1_vol. 6391 - Ensures the flat_load 6392 has completed 6393 before invalidating 6394 the caches. 6395 6396 3. buffer_wbinvl1_vol 6397 6398 - Must happen before 6399 any following 6400 global/generic 6401 load/load 6402 atomic/atomicrmw. 6403 - Ensures that 6404 following 6405 L1 loads will not see 6406 stale global data. 6407 MTYPE RW and CC memory will 6408 never be stale in L2 due to 6409 the memory probes. 6410 6411 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6412 - wavefront - generic 6413 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6414 - wavefront local address space cannot 6415 be used.* 6416 6417 1. ds_atomic 6418 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6419 2. s_waitcnt vmcnt(0) 6420 6421 - If not TgSplit execution 6422 mode, omit. 6423 - Must happen before the 6424 following buffer_wbinvl1_vol. 6425 - Ensures the atomicrmw 6426 has completed 6427 before invalidating 6428 the cache. 6429 6430 3. buffer_wbinvl1_vol 6431 6432 - If not TgSplit execution 6433 mode, omit. 6434 - Must happen before 6435 any following 6436 global/generic 6437 load/load 6438 atomic/atomicrmw. 6439 - Ensures that 6440 following loads 6441 will not see stale 6442 global data. 6443 6444 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6445 local address space cannot 6446 be used.* 6447 6448 1. ds_atomic 6449 2. s_waitcnt lgkmcnt(0) 6450 6451 - If OpenCL, omit. 6452 - Must happen before 6453 any following 6454 global/generic 6455 load/load 6456 atomic/store/store 6457 atomic/atomicrmw. 6458 - Ensures any 6459 following global 6460 data read is no 6461 older than the local 6462 atomicrmw value 6463 being acquired. 6464 6465 atomicrmw acquire - workgroup - generic 1. flat_atomic 6466 2. s_waitcnt lgkm/vmcnt(0) 6467 6468 - Use lgkmcnt(0) if not 6469 TgSplit execution mode 6470 and vmcnt(0) if TgSplit 6471 execution mode. 6472 - If OpenCL, omit lgkmcnt(0). 6473 - Must happen before 6474 the following 6475 buffer_wbinvl1_vol and 6476 any following 6477 global/generic 6478 load/load 6479 atomic/store/store 6480 atomic/atomicrmw. 6481 - Ensures any 6482 following global 6483 data read is no 6484 older than a local 6485 atomicrmw value 6486 being acquired. 6487 6488 3. buffer_wbinvl1_vol 6489 6490 - If not TgSplit execution 6491 mode, omit. 6492 - Ensures that 6493 following 6494 loads will not see 6495 stale data. 6496 6497 atomicrmw acquire - agent - global 1. buffer/global_atomic 6498 2. s_waitcnt vmcnt(0) 6499 6500 - Must happen before 6501 following 6502 buffer_wbinvl1_vol. 6503 - Ensures the 6504 atomicrmw has 6505 completed before 6506 invalidating the 6507 cache. 6508 6509 3. buffer_wbinvl1_vol 6510 6511 - Must happen before 6512 any following 6513 global/generic 6514 load/load 6515 atomic/atomicrmw. 6516 - Ensures that 6517 following loads 6518 will not see stale 6519 global data. 6520 6521 atomicrmw acquire - system - global 1. buffer/global_atomic 6522 2. s_waitcnt vmcnt(0) 6523 6524 - Must happen before 6525 following 6526 buffer_wbinvl1_vol. 6527 - Ensures the 6528 atomicrmw has 6529 completed before 6530 invalidating the 6531 caches. 6532 6533 3. buffer_wbinvl1_vol 6534 6535 - Must happen before 6536 any following 6537 global/generic 6538 load/load 6539 atomic/atomicrmw. 6540 - Ensures that 6541 following 6542 loads will not see 6543 stale L1 global data. 6544 MTYPE RW and CC L2 memory 6545 never be stale in L2 due to 6546 the memory probes. 6547 6548 atomicrmw acquire - agent - generic 1. flat_atomic 6549 2. s_waitcnt vmcnt(0) & 6550 lgkmcnt(0) 6551 6552 - If TgSplit execution mode, 6553 omit lgkmcnt(0). 6554 - If OpenCL, omit 6555 lgkmcnt(0). 6556 - Must happen before 6557 following 6558 buffer_wbinvl1_vol. 6559 - Ensures the 6560 atomicrmw has 6561 completed before 6562 invalidating the 6563 cache. 6564 6565 3. buffer_wbinvl1_vol 6566 6567 - Must happen before 6568 any following 6569 global/generic 6570 load/load 6571 atomic/atomicrmw. 6572 - Ensures that 6573 following loads 6574 will not see stale 6575 global data. 6576 6577 atomicrmw acquire - system - generic 1. flat_atomic 6578 2. s_waitcnt vmcnt(0) & 6579 lgkmcnt(0) 6580 6581 - If TgSplit execution mode, 6582 omit lgkmcnt(0). 6583 - If OpenCL, omit 6584 lgkmcnt(0). 6585 - Must happen before 6586 following 6587 buffer_wbinvl1_vol. 6588 - Ensures the 6589 atomicrmw has 6590 completed before 6591 invalidating the 6592 caches. 6593 6594 3. buffer_wbinvl1_vol 6595 6596 - Must happen before 6597 any following 6598 global/generic 6599 load/load 6600 atomic/atomicrmw. 6601 - Ensures that 6602 following 6603 loads will not see 6604 stale L1 global data. 6605 MTYPE RW and CC memory will 6606 never be stale in L2 due to 6607 the memory probes. 6608 6609 fence acquire - singlethread *none* *none* 6610 - wavefront 6611 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6612 6613 - Use lgkmcnt(0) if not 6614 TgSplit execution mode 6615 and vmcnt(0) if TgSplit 6616 execution mode. 6617 - If OpenCL and 6618 address space is 6619 not generic, omit 6620 lgkmcnt(0). 6621 - If OpenCL and 6622 address space is 6623 local, omit 6624 vmcnt(0). 6625 - However, since LLVM 6626 currently has no 6627 address space on 6628 the fence need to 6629 conservatively 6630 always generate. If 6631 fence had an 6632 address space then 6633 set to address 6634 space of OpenCL 6635 fence flag, or to 6636 generic if both 6637 local and global 6638 flags are 6639 specified. 6640 - s_waitcnt vmcnt(0) 6641 must happen after 6642 any preceding 6643 global/generic load 6644 atomic/ 6645 atomicrmw 6646 with an equal or 6647 wider sync scope 6648 and memory ordering 6649 stronger than 6650 unordered (this is 6651 termed the 6652 fence-paired-atomic). 6653 - s_waitcnt lgkmcnt(0) 6654 must happen after 6655 any preceding 6656 local/generic load 6657 atomic/atomicrmw 6658 with an equal or 6659 wider sync scope 6660 and memory ordering 6661 stronger than 6662 unordered (this is 6663 termed the 6664 fence-paired-atomic). 6665 - Must happen before 6666 the following 6667 buffer_wbinvl1_vol and 6668 any following 6669 global/generic 6670 load/load 6671 atomic/store/store 6672 atomic/atomicrmw. 6673 - Ensures any 6674 following global 6675 data read is no 6676 older than the 6677 value read by the 6678 fence-paired-atomic. 6679 6680 3. buffer_wbinvl1_vol 6681 6682 - If not TgSplit execution 6683 mode, omit. 6684 - Ensures that 6685 following 6686 loads will not see 6687 stale data. 6688 6689 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6690 vmcnt(0) 6691 6692 - If TgSplit execution mode, 6693 omit lgkmcnt(0). 6694 - If OpenCL and 6695 address space is 6696 not generic, omit 6697 lgkmcnt(0). 6698 - However, since LLVM 6699 currently has no 6700 address space on 6701 the fence need to 6702 conservatively 6703 always generate 6704 (see comment for 6705 previous fence). 6706 - Could be split into 6707 separate s_waitcnt 6708 vmcnt(0) and 6709 s_waitcnt 6710 lgkmcnt(0) to allow 6711 them to be 6712 independently moved 6713 according to the 6714 following rules. 6715 - s_waitcnt vmcnt(0) 6716 must happen after 6717 any preceding 6718 global/generic load 6719 atomic/atomicrmw 6720 with an equal or 6721 wider sync scope 6722 and memory ordering 6723 stronger than 6724 unordered (this is 6725 termed the 6726 fence-paired-atomic). 6727 - s_waitcnt lgkmcnt(0) 6728 must happen after 6729 any preceding 6730 local/generic load 6731 atomic/atomicrmw 6732 with an equal or 6733 wider sync scope 6734 and memory ordering 6735 stronger than 6736 unordered (this is 6737 termed the 6738 fence-paired-atomic). 6739 - Must happen before 6740 the following 6741 buffer_wbinvl1_vol. 6742 - Ensures that the 6743 fence-paired atomic 6744 has completed 6745 before invalidating 6746 the 6747 cache. Therefore 6748 any following 6749 locations read must 6750 be no older than 6751 the value read by 6752 the 6753 fence-paired-atomic. 6754 6755 2. buffer_wbinvl1_vol 6756 6757 - Must happen before any 6758 following global/generic 6759 load/load 6760 atomic/store/store 6761 atomic/atomicrmw. 6762 - Ensures that 6763 following loads 6764 will not see stale 6765 global data. 6766 6767 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6768 vmcnt(0) 6769 6770 - If TgSplit execution mode, 6771 omit lgkmcnt(0). 6772 - If OpenCL and 6773 address space is 6774 not generic, omit 6775 lgkmcnt(0). 6776 - However, since LLVM 6777 currently has no 6778 address space on 6779 the fence need to 6780 conservatively 6781 always generate 6782 (see comment for 6783 previous fence). 6784 - Could be split into 6785 separate s_waitcnt 6786 vmcnt(0) and 6787 s_waitcnt 6788 lgkmcnt(0) to allow 6789 them to be 6790 independently moved 6791 according to the 6792 following rules. 6793 - s_waitcnt vmcnt(0) 6794 must happen after 6795 any preceding 6796 global/generic load 6797 atomic/atomicrmw 6798 with an equal or 6799 wider sync scope 6800 and memory ordering 6801 stronger than 6802 unordered (this is 6803 termed the 6804 fence-paired-atomic). 6805 - s_waitcnt lgkmcnt(0) 6806 must happen after 6807 any preceding 6808 local/generic load 6809 atomic/atomicrmw 6810 with an equal or 6811 wider sync scope 6812 and memory ordering 6813 stronger than 6814 unordered (this is 6815 termed the 6816 fence-paired-atomic). 6817 - Must happen before 6818 the following 6819 buffer_wbinvl1_vol. 6820 - Ensures that the 6821 fence-paired atomic 6822 has completed 6823 before invalidating 6824 the 6825 cache. Therefore 6826 any following 6827 locations read must 6828 be no older than 6829 the value read by 6830 the 6831 fence-paired-atomic. 6832 6833 2. buffer_wbinvl1_vol 6834 6835 - Must happen before any 6836 following global/generic 6837 load/load 6838 atomic/store/store 6839 atomic/atomicrmw. 6840 - Ensures that 6841 following 6842 loads will not see 6843 stale L1 global data. 6844 MTYPE RW and CC memory will 6845 never be stale in L2 due to 6846 the memory probes. 6847 **Release Atomic** 6848 ------------------------------------------------------------------------------------ 6849 store atomic release - singlethread - global 1. buffer/global/flat_store 6850 - wavefront - generic 6851 store atomic release - singlethread - local *If TgSplit execution mode, 6852 - wavefront local address space cannot 6853 be used.* 6854 6855 1. ds_store 6856 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6857 - generic 6858 - Use lgkmcnt(0) if not 6859 TgSplit execution mode 6860 and vmcnt(0) if TgSplit 6861 execution mode. 6862 - If OpenCL, omit lgkmcnt(0). 6863 - s_waitcnt vmcnt(0) 6864 must happen after 6865 any preceding 6866 global/generic load/store/ 6867 load atomic/store atomic/ 6868 atomicrmw. 6869 - s_waitcnt lgkmcnt(0) 6870 must happen after 6871 any preceding 6872 local/generic 6873 load/store/load 6874 atomic/store 6875 atomic/atomicrmw. 6876 - Must happen before 6877 the following 6878 store. 6879 - Ensures that all 6880 memory operations 6881 have 6882 completed before 6883 performing the 6884 store that is being 6885 released. 6886 6887 2. buffer/global/flat_store 6888 store atomic release - workgroup - local *If TgSplit execution mode, 6889 local address space cannot 6890 be used.* 6891 6892 1. ds_store 6893 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6894 - generic vmcnt(0) 6895 6896 - If TgSplit execution mode, 6897 omit lgkmcnt(0). 6898 - If OpenCL and 6899 address space is 6900 not generic, omit 6901 lgkmcnt(0). 6902 - Could be split into 6903 separate s_waitcnt 6904 vmcnt(0) and 6905 s_waitcnt 6906 lgkmcnt(0) to allow 6907 them to be 6908 independently moved 6909 according to the 6910 following rules. 6911 - s_waitcnt vmcnt(0) 6912 must happen after 6913 any preceding 6914 global/generic 6915 load/store/load 6916 atomic/store 6917 atomic/atomicrmw. 6918 - s_waitcnt lgkmcnt(0) 6919 must happen after 6920 any preceding 6921 local/generic 6922 load/store/load 6923 atomic/store 6924 atomic/atomicrmw. 6925 - Must happen before 6926 the following 6927 store. 6928 - Ensures that all 6929 memory operations 6930 to memory have 6931 completed before 6932 performing the 6933 store that is being 6934 released. 6935 6936 2. buffer/global/flat_store 6937 store atomic release - system - global 1. s_waitcnt lgkmcnt(0) & 6938 - generic vmcnt(0) 6939 6940 - If TgSplit execution mode, 6941 omit lgkmcnt(0). 6942 - If OpenCL and 6943 address space is 6944 not generic, omit 6945 lgkmcnt(0). 6946 - Could be split into 6947 separate s_waitcnt 6948 vmcnt(0) and 6949 s_waitcnt 6950 lgkmcnt(0) to allow 6951 them to be 6952 independently moved 6953 according to the 6954 following rules. 6955 - s_waitcnt vmcnt(0) 6956 must happen after any 6957 preceding 6958 global/generic 6959 load/store/load 6960 atomic/store 6961 atomic/atomicrmw. 6962 - s_waitcnt lgkmcnt(0) 6963 must happen after any 6964 preceding 6965 local/generic 6966 load/store/load 6967 atomic/store 6968 atomic/atomicrmw. 6969 - Must happen before 6970 the following 6971 store. 6972 - Ensures that all 6973 memory operations 6974 to memory and the L2 6975 writeback have 6976 completed before 6977 performing the 6978 store that is being 6979 released. 6980 6981 2. buffer/global/flat_store 6982 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 6983 - wavefront - generic 6984 atomicrmw release - singlethread - local *If TgSplit execution mode, 6985 - wavefront local address space cannot 6986 be used.* 6987 6988 1. ds_atomic 6989 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6990 - generic 6991 - Use lgkmcnt(0) if not 6992 TgSplit execution mode 6993 and vmcnt(0) if TgSplit 6994 execution mode. 6995 - If OpenCL, omit 6996 lgkmcnt(0). 6997 - s_waitcnt vmcnt(0) 6998 must happen after 6999 any preceding 7000 global/generic load/store/ 7001 load atomic/store atomic/ 7002 atomicrmw. 7003 - s_waitcnt lgkmcnt(0) 7004 must happen after 7005 any preceding 7006 local/generic 7007 load/store/load 7008 atomic/store 7009 atomic/atomicrmw. 7010 - Must happen before 7011 the following 7012 atomicrmw. 7013 - Ensures that all 7014 memory operations 7015 have 7016 completed before 7017 performing the 7018 atomicrmw that is 7019 being released. 7020 7021 2. buffer/global/flat_atomic 7022 atomicrmw release - workgroup - local *If TgSplit execution mode, 7023 local address space cannot 7024 be used.* 7025 7026 1. ds_atomic 7027 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7028 - generic vmcnt(0) 7029 7030 - If TgSplit execution mode, 7031 omit lgkmcnt(0). 7032 - If OpenCL, omit 7033 lgkmcnt(0). 7034 - Could be split into 7035 separate s_waitcnt 7036 vmcnt(0) and 7037 s_waitcnt 7038 lgkmcnt(0) to allow 7039 them to be 7040 independently moved 7041 according to the 7042 following rules. 7043 - s_waitcnt vmcnt(0) 7044 must happen after 7045 any preceding 7046 global/generic 7047 load/store/load 7048 atomic/store 7049 atomic/atomicrmw. 7050 - s_waitcnt lgkmcnt(0) 7051 must happen after 7052 any preceding 7053 local/generic 7054 load/store/load 7055 atomic/store 7056 atomic/atomicrmw. 7057 - Must happen before 7058 the following 7059 atomicrmw. 7060 - Ensures that all 7061 memory operations 7062 to global and local 7063 have completed 7064 before performing 7065 the atomicrmw that 7066 is being released. 7067 7068 2. buffer/global/flat_atomic 7069 atomicrmw release - system - global 1. s_waitcnt lgkmcnt(0) & 7070 - generic vmcnt(0) 7071 7072 - If TgSplit execution mode, 7073 omit lgkmcnt(0). 7074 - If OpenCL, omit 7075 lgkmcnt(0). 7076 - Could be split into 7077 separate s_waitcnt 7078 vmcnt(0) and 7079 s_waitcnt 7080 lgkmcnt(0) to allow 7081 them to be 7082 independently moved 7083 according to the 7084 following rules. 7085 - s_waitcnt vmcnt(0) 7086 must happen after 7087 any preceding 7088 global/generic 7089 load/store/load 7090 atomic/store 7091 atomic/atomicrmw. 7092 - s_waitcnt lgkmcnt(0) 7093 must happen after 7094 any preceding 7095 local/generic 7096 load/store/load 7097 atomic/store 7098 atomic/atomicrmw. 7099 - Must happen before 7100 the following 7101 atomicrmw. 7102 - Ensures that all 7103 memory operations 7104 to memory and the L2 7105 writeback have 7106 completed before 7107 performing the 7108 store that is being 7109 released. 7110 7111 2. buffer/global/flat_atomic 7112 fence release - singlethread *none* *none* 7113 - wavefront 7114 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7115 7116 - Use lgkmcnt(0) if not 7117 TgSplit execution mode 7118 and vmcnt(0) if TgSplit 7119 execution mode. 7120 - If OpenCL and 7121 address space is 7122 not generic, omit 7123 lgkmcnt(0). 7124 - If OpenCL and 7125 address space is 7126 local, omit 7127 vmcnt(0). 7128 - However, since LLVM 7129 currently has no 7130 address space on 7131 the fence need to 7132 conservatively 7133 always generate. If 7134 fence had an 7135 address space then 7136 set to address 7137 space of OpenCL 7138 fence flag, or to 7139 generic if both 7140 local and global 7141 flags are 7142 specified. 7143 - s_waitcnt vmcnt(0) 7144 must happen after 7145 any preceding 7146 global/generic 7147 load/store/ 7148 load atomic/store atomic/ 7149 atomicrmw. 7150 - s_waitcnt lgkmcnt(0) 7151 must happen after 7152 any preceding 7153 local/generic 7154 load/load 7155 atomic/store/store 7156 atomic/atomicrmw. 7157 - Must happen before 7158 any following store 7159 atomic/atomicrmw 7160 with an equal or 7161 wider sync scope 7162 and memory ordering 7163 stronger than 7164 unordered (this is 7165 termed the 7166 fence-paired-atomic). 7167 - Ensures that all 7168 memory operations 7169 have 7170 completed before 7171 performing the 7172 following 7173 fence-paired-atomic. 7174 7175 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7176 vmcnt(0) 7177 7178 - If TgSplit execution mode, 7179 omit lgkmcnt(0). 7180 - If OpenCL and 7181 address space is 7182 not generic, omit 7183 lgkmcnt(0). 7184 - If OpenCL and 7185 address space is 7186 local, omit 7187 vmcnt(0). 7188 - However, since LLVM 7189 currently has no 7190 address space on 7191 the fence need to 7192 conservatively 7193 always generate. If 7194 fence had an 7195 address space then 7196 set to address 7197 space of OpenCL 7198 fence flag, or to 7199 generic if both 7200 local and global 7201 flags are 7202 specified. 7203 - Could be split into 7204 separate s_waitcnt 7205 vmcnt(0) and 7206 s_waitcnt 7207 lgkmcnt(0) to allow 7208 them to be 7209 independently moved 7210 according to the 7211 following rules. 7212 - s_waitcnt vmcnt(0) 7213 must happen after 7214 any preceding 7215 global/generic 7216 load/store/load 7217 atomic/store 7218 atomic/atomicrmw. 7219 - s_waitcnt lgkmcnt(0) 7220 must happen after 7221 any preceding 7222 local/generic 7223 load/store/load 7224 atomic/store 7225 atomic/atomicrmw. 7226 - Must happen before 7227 any following store 7228 atomic/atomicrmw 7229 with an equal or 7230 wider sync scope 7231 and memory ordering 7232 stronger than 7233 unordered (this is 7234 termed the 7235 fence-paired-atomic). 7236 - Ensures that all 7237 memory operations 7238 have 7239 completed before 7240 performing the 7241 following 7242 fence-paired-atomic. 7243 7244 fence release - system *none* 1. s_waitcnt lgkmcnt(0) & 7245 vmcnt(0) 7246 7247 - If TgSplit execution mode, 7248 omit lgkmcnt(0). 7249 - If OpenCL and 7250 address space is 7251 not generic, omit 7252 lgkmcnt(0). 7253 - If OpenCL and 7254 address space is 7255 local, omit 7256 vmcnt(0). 7257 - However, since LLVM 7258 currently has no 7259 address space on 7260 the fence need to 7261 conservatively 7262 always generate. If 7263 fence had an 7264 address space then 7265 set to address 7266 space of OpenCL 7267 fence flag, or to 7268 generic if both 7269 local and global 7270 flags are 7271 specified. 7272 - Could be split into 7273 separate s_waitcnt 7274 vmcnt(0) and 7275 s_waitcnt 7276 lgkmcnt(0) to allow 7277 them to be 7278 independently moved 7279 according to the 7280 following rules. 7281 - s_waitcnt vmcnt(0) 7282 must happen after 7283 any preceding 7284 global/generic 7285 load/store/load 7286 atomic/store 7287 atomic/atomicrmw. 7288 - s_waitcnt lgkmcnt(0) 7289 must happen after 7290 any preceding 7291 local/generic 7292 load/store/load 7293 atomic/store 7294 atomic/atomicrmw. 7295 - Must happen before 7296 any following store 7297 atomic/atomicrmw 7298 with an equal or 7299 wider sync scope 7300 and memory ordering 7301 stronger than 7302 unordered (this is 7303 termed the 7304 fence-paired-atomic). 7305 - Ensures that all 7306 memory operations 7307 have 7308 completed before 7309 performing the 7310 following 7311 fence-paired-atomic. 7312 7313 **Acquire-Release Atomic** 7314 ------------------------------------------------------------------------------------ 7315 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7316 - wavefront - generic 7317 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7318 - wavefront local address space cannot 7319 be used.* 7320 7321 1. ds_atomic 7322 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7323 7324 - Use lgkmcnt(0) if not 7325 TgSplit execution mode 7326 and vmcnt(0) if TgSplit 7327 execution mode. 7328 - If OpenCL, omit 7329 lgkmcnt(0). 7330 - Must happen after 7331 any preceding 7332 local/generic 7333 load/store/load 7334 atomic/store 7335 atomic/atomicrmw. 7336 - s_waitcnt vmcnt(0) 7337 must happen after 7338 any preceding 7339 global/generic load/store/ 7340 load atomic/store atomic/ 7341 atomicrmw. 7342 - s_waitcnt lgkmcnt(0) 7343 must happen after 7344 any preceding 7345 local/generic 7346 load/store/load 7347 atomic/store 7348 atomic/atomicrmw. 7349 - Must happen before 7350 the following 7351 atomicrmw. 7352 - Ensures that all 7353 memory operations 7354 have 7355 completed before 7356 performing the 7357 atomicrmw that is 7358 being released. 7359 7360 2. buffer/global_atomic 7361 3. s_waitcnt vmcnt(0) 7362 7363 - If not TgSplit execution 7364 mode, omit. 7365 - Must happen before 7366 the following 7367 buffer_wbinvl1_vol. 7368 - Ensures any 7369 following global 7370 data read is no 7371 older than the 7372 atomicrmw value 7373 being acquired. 7374 7375 4. buffer_wbinvl1_vol 7376 7377 - If not TgSplit execution 7378 mode, omit. 7379 - Ensures that 7380 following 7381 loads will not see 7382 stale data. 7383 7384 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7385 local address space cannot 7386 be used.* 7387 7388 1. ds_atomic 7389 2. s_waitcnt lgkmcnt(0) 7390 7391 - If OpenCL, omit. 7392 - Must happen before 7393 any following 7394 global/generic 7395 load/load 7396 atomic/store/store 7397 atomic/atomicrmw. 7398 - Ensures any 7399 following global 7400 data read is no 7401 older than the local load 7402 atomic value being 7403 acquired. 7404 7405 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7406 7407 - Use lgkmcnt(0) if not 7408 TgSplit execution mode 7409 and vmcnt(0) if TgSplit 7410 execution mode. 7411 - If OpenCL, omit 7412 lgkmcnt(0). 7413 - s_waitcnt vmcnt(0) 7414 must happen after 7415 any preceding 7416 global/generic load/store/ 7417 load atomic/store atomic/ 7418 atomicrmw. 7419 - s_waitcnt lgkmcnt(0) 7420 must happen after 7421 any preceding 7422 local/generic 7423 load/store/load 7424 atomic/store 7425 atomic/atomicrmw. 7426 - Must happen before 7427 the following 7428 atomicrmw. 7429 - Ensures that all 7430 memory operations 7431 have 7432 completed before 7433 performing the 7434 atomicrmw that is 7435 being released. 7436 7437 2. flat_atomic 7438 3. s_waitcnt lgkmcnt(0) & 7439 vmcnt(0) 7440 7441 - If not TgSplit execution 7442 mode, omit vmcnt(0). 7443 - If OpenCL, omit 7444 lgkmcnt(0). 7445 - Must happen before 7446 the following 7447 buffer_wbinvl1_vol and 7448 any following 7449 global/generic 7450 load/load 7451 atomic/store/store 7452 atomic/atomicrmw. 7453 - Ensures any 7454 following global 7455 data read is no 7456 older than a local load 7457 atomic value being 7458 acquired. 7459 7460 3. buffer_wbinvl1_vol 7461 7462 - If not TgSplit execution 7463 mode, omit. 7464 - Ensures that 7465 following 7466 loads will not see 7467 stale data. 7468 7469 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7470 vmcnt(0) 7471 7472 - If TgSplit execution mode, 7473 omit lgkmcnt(0). 7474 - If OpenCL, omit 7475 lgkmcnt(0). 7476 - Could be split into 7477 separate s_waitcnt 7478 vmcnt(0) and 7479 s_waitcnt 7480 lgkmcnt(0) to allow 7481 them to be 7482 independently moved 7483 according to the 7484 following rules. 7485 - s_waitcnt vmcnt(0) 7486 must happen after 7487 any preceding 7488 global/generic 7489 load/store/load 7490 atomic/store 7491 atomic/atomicrmw. 7492 - s_waitcnt lgkmcnt(0) 7493 must happen after 7494 any preceding 7495 local/generic 7496 load/store/load 7497 atomic/store 7498 atomic/atomicrmw. 7499 - Must happen before 7500 the following 7501 atomicrmw. 7502 - Ensures that all 7503 memory operations 7504 to global have 7505 completed before 7506 performing the 7507 atomicrmw that is 7508 being released. 7509 7510 2. buffer/global_atomic 7511 3. s_waitcnt vmcnt(0) 7512 7513 - Must happen before 7514 following 7515 buffer_wbinvl1_vol. 7516 - Ensures the 7517 atomicrmw has 7518 completed before 7519 invalidating the 7520 cache. 7521 7522 4. buffer_wbinvl1_vol 7523 7524 - Must happen before 7525 any following 7526 global/generic 7527 load/load 7528 atomic/atomicrmw. 7529 - Ensures that 7530 following loads 7531 will not see stale 7532 global data. 7533 7534 atomicrmw acq_rel - system - global 1. s_waitcnt lgkmcnt(0) & 7535 vmcnt(0) 7536 7537 - If TgSplit execution mode, 7538 omit lgkmcnt(0). 7539 - If OpenCL, omit 7540 lgkmcnt(0). 7541 - Could be split into 7542 separate s_waitcnt 7543 vmcnt(0) and 7544 s_waitcnt 7545 lgkmcnt(0) to allow 7546 them to be 7547 independently moved 7548 according to the 7549 following rules. 7550 - s_waitcnt vmcnt(0) 7551 must happen after 7552 any preceding 7553 global/generic 7554 load/store/load 7555 atomic/store 7556 atomic/atomicrmw. 7557 - s_waitcnt lgkmcnt(0) 7558 must happen after 7559 any preceding 7560 local/generic 7561 load/store/load 7562 atomic/store 7563 atomic/atomicrmw. 7564 - Must happen before 7565 the following 7566 atomicrmw. 7567 - Ensures that all 7568 memory operations 7569 to global and L2 writeback 7570 have completed before 7571 performing the 7572 atomicrmw that is 7573 being released. 7574 7575 2. buffer/global_atomic 7576 3. s_waitcnt vmcnt(0) 7577 7578 - Must happen before 7579 following 7580 buffer_wbinvl1_vol. 7581 - Ensures the 7582 atomicrmw has 7583 completed before 7584 invalidating the 7585 caches. 7586 7587 4. buffer_wbinvl1_vol 7588 7589 - Must happen before 7590 any following 7591 global/generic 7592 load/load 7593 atomic/atomicrmw. 7594 - Ensures that 7595 following 7596 loads will not see 7597 stale L1 global data. 7598 MTYPE RW and CC memory will 7599 never be stale in L2 due to 7600 the memory probes. 7601 7602 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7603 vmcnt(0) 7604 7605 - If TgSplit execution mode, 7606 omit lgkmcnt(0). 7607 - If OpenCL, omit 7608 lgkmcnt(0). 7609 - Could be split into 7610 separate s_waitcnt 7611 vmcnt(0) and 7612 s_waitcnt 7613 lgkmcnt(0) to allow 7614 them to be 7615 independently moved 7616 according to the 7617 following rules. 7618 - s_waitcnt vmcnt(0) 7619 must happen after 7620 any preceding 7621 global/generic 7622 load/store/load 7623 atomic/store 7624 atomic/atomicrmw. 7625 - s_waitcnt lgkmcnt(0) 7626 must happen after 7627 any preceding 7628 local/generic 7629 load/store/load 7630 atomic/store 7631 atomic/atomicrmw. 7632 - Must happen before 7633 the following 7634 atomicrmw. 7635 - Ensures that all 7636 memory operations 7637 to global have 7638 completed before 7639 performing the 7640 atomicrmw that is 7641 being released. 7642 7643 2. flat_atomic 7644 3. s_waitcnt vmcnt(0) & 7645 lgkmcnt(0) 7646 7647 - If TgSplit execution mode, 7648 omit lgkmcnt(0). 7649 - If OpenCL, omit 7650 lgkmcnt(0). 7651 - Must happen before 7652 following 7653 buffer_wbinvl1_vol. 7654 - Ensures the 7655 atomicrmw has 7656 completed before 7657 invalidating the 7658 cache. 7659 7660 4. buffer_wbinvl1_vol 7661 7662 - Must happen before 7663 any following 7664 global/generic 7665 load/load 7666 atomic/atomicrmw. 7667 - Ensures that 7668 following loads 7669 will not see stale 7670 global data. 7671 7672 atomicrmw acq_rel - system - generic 1. s_waitcnt lgkmcnt(0) & 7673 vmcnt(0) 7674 7675 - If TgSplit execution mode, 7676 omit lgkmcnt(0). 7677 - If OpenCL, omit 7678 lgkmcnt(0). 7679 - Could be split into 7680 separate s_waitcnt 7681 vmcnt(0) and 7682 s_waitcnt 7683 lgkmcnt(0) to allow 7684 them to be 7685 independently moved 7686 according to the 7687 following rules. 7688 - s_waitcnt vmcnt(0) 7689 must happen after 7690 any preceding 7691 global/generic 7692 load/store/load 7693 atomic/store 7694 atomic/atomicrmw. 7695 - s_waitcnt lgkmcnt(0) 7696 must happen after 7697 any preceding 7698 local/generic 7699 load/store/load 7700 atomic/store 7701 atomic/atomicrmw. 7702 - Must happen before 7703 the following 7704 atomicrmw. 7705 - Ensures that all 7706 memory operations 7707 to global and L2 writeback 7708 have completed before 7709 performing the 7710 atomicrmw that is 7711 being released. 7712 7713 2. flat_atomic 7714 3. s_waitcnt vmcnt(0) & 7715 lgkmcnt(0) 7716 7717 - If TgSplit execution mode, 7718 omit lgkmcnt(0). 7719 - If OpenCL, omit 7720 lgkmcnt(0). 7721 - Must happen before 7722 following 7723 buffer_wbinvl1_vol. 7724 - Ensures the 7725 atomicrmw has 7726 completed before 7727 invalidating the 7728 caches. 7729 7730 4. buffer_wbinvl1_vol 7731 7732 - Must happen before 7733 any following 7734 global/generic 7735 load/load 7736 atomic/atomicrmw. 7737 - Ensures that 7738 following 7739 loads will not see 7740 stale L1 global data. 7741 MTYPE RW and CC memory will 7742 never be stale in L2 due to 7743 the memory probes. 7744 7745 fence acq_rel - singlethread *none* *none* 7746 - wavefront 7747 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7748 7749 - Use lgkmcnt(0) if not 7750 TgSplit execution mode 7751 and vmcnt(0) if TgSplit 7752 execution mode. 7753 - If OpenCL and 7754 address space is 7755 not generic, omit 7756 lgkmcnt(0). 7757 - If OpenCL and 7758 address space is 7759 local, omit 7760 vmcnt(0). 7761 - However, 7762 since LLVM 7763 currently has no 7764 address space on 7765 the fence need to 7766 conservatively 7767 always generate 7768 (see comment for 7769 previous fence). 7770 - s_waitcnt vmcnt(0) 7771 must happen after 7772 any preceding 7773 global/generic 7774 load/store/ 7775 load atomic/store atomic/ 7776 atomicrmw. 7777 - s_waitcnt lgkmcnt(0) 7778 must happen after 7779 any preceding 7780 local/generic 7781 load/load 7782 atomic/store/store 7783 atomic/atomicrmw. 7784 - Must happen before 7785 any following 7786 global/generic 7787 load/load 7788 atomic/store/store 7789 atomic/atomicrmw. 7790 - Ensures that all 7791 memory operations 7792 have 7793 completed before 7794 performing any 7795 following global 7796 memory operations. 7797 - Ensures that the 7798 preceding 7799 local/generic load 7800 atomic/atomicrmw 7801 with an equal or 7802 wider sync scope 7803 and memory ordering 7804 stronger than 7805 unordered (this is 7806 termed the 7807 acquire-fence-paired-atomic) 7808 has completed 7809 before following 7810 global memory 7811 operations. This 7812 satisfies the 7813 requirements of 7814 acquire. 7815 - Ensures that all 7816 previous memory 7817 operations have 7818 completed before a 7819 following 7820 local/generic store 7821 atomic/atomicrmw 7822 with an equal or 7823 wider sync scope 7824 and memory ordering 7825 stronger than 7826 unordered (this is 7827 termed the 7828 release-fence-paired-atomic). 7829 This satisfies the 7830 requirements of 7831 release. 7832 - Must happen before 7833 the following 7834 buffer_wbinvl1_vol. 7835 - Ensures that the 7836 acquire-fence-paired 7837 atomic has completed 7838 before invalidating 7839 the 7840 cache. Therefore 7841 any following 7842 locations read must 7843 be no older than 7844 the value read by 7845 the 7846 acquire-fence-paired-atomic. 7847 7848 3. buffer_wbinvl1_vol 7849 7850 - If not TgSplit execution 7851 mode, omit. 7852 - Ensures that 7853 following 7854 loads will not see 7855 stale data. 7856 7857 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7858 vmcnt(0) 7859 7860 - If TgSplit execution mode, 7861 omit lgkmcnt(0). 7862 - If OpenCL and 7863 address space is 7864 not generic, omit 7865 lgkmcnt(0). 7866 - However, since LLVM 7867 currently has no 7868 address space on 7869 the fence need to 7870 conservatively 7871 always generate 7872 (see comment for 7873 previous fence). 7874 - Could be split into 7875 separate s_waitcnt 7876 vmcnt(0) and 7877 s_waitcnt 7878 lgkmcnt(0) to allow 7879 them to be 7880 independently moved 7881 according to the 7882 following rules. 7883 - s_waitcnt vmcnt(0) 7884 must happen after 7885 any preceding 7886 global/generic 7887 load/store/load 7888 atomic/store 7889 atomic/atomicrmw. 7890 - s_waitcnt lgkmcnt(0) 7891 must happen after 7892 any preceding 7893 local/generic 7894 load/store/load 7895 atomic/store 7896 atomic/atomicrmw. 7897 - Must happen before 7898 the following 7899 buffer_wbinvl1_vol. 7900 - Ensures that the 7901 preceding 7902 global/local/generic 7903 load 7904 atomic/atomicrmw 7905 with an equal or 7906 wider sync scope 7907 and memory ordering 7908 stronger than 7909 unordered (this is 7910 termed the 7911 acquire-fence-paired-atomic) 7912 has completed 7913 before invalidating 7914 the cache. This 7915 satisfies the 7916 requirements of 7917 acquire. 7918 - Ensures that all 7919 previous memory 7920 operations have 7921 completed before a 7922 following 7923 global/local/generic 7924 store 7925 atomic/atomicrmw 7926 with an equal or 7927 wider sync scope 7928 and memory ordering 7929 stronger than 7930 unordered (this is 7931 termed the 7932 release-fence-paired-atomic). 7933 This satisfies the 7934 requirements of 7935 release. 7936 7937 2. buffer_wbinvl1_vol 7938 7939 - Must happen before 7940 any following 7941 global/generic 7942 load/load 7943 atomic/store/store 7944 atomic/atomicrmw. 7945 - Ensures that 7946 following loads 7947 will not see stale 7948 global data. This 7949 satisfies the 7950 requirements of 7951 acquire. 7952 7953 fence acq_rel - system *none* 1. s_waitcnt lgkmcnt(0) & 7954 vmcnt(0) 7955 7956 - If TgSplit execution mode, 7957 omit lgkmcnt(0). 7958 - If OpenCL and 7959 address space is 7960 not generic, omit 7961 lgkmcnt(0). 7962 - However, since LLVM 7963 currently has no 7964 address space on 7965 the fence need to 7966 conservatively 7967 always generate 7968 (see comment for 7969 previous fence). 7970 - Could be split into 7971 separate s_waitcnt 7972 vmcnt(0) and 7973 s_waitcnt 7974 lgkmcnt(0) to allow 7975 them to be 7976 independently moved 7977 according to the 7978 following rules. 7979 - s_waitcnt vmcnt(0) 7980 must happen after 7981 any preceding 7982 global/generic 7983 load/store/load 7984 atomic/store 7985 atomic/atomicrmw. 7986 - s_waitcnt lgkmcnt(0) 7987 must happen after 7988 any preceding 7989 local/generic 7990 load/store/load 7991 atomic/store 7992 atomic/atomicrmw. 7993 - Must happen before 7994 the following 7995 buffer_wbinvl1_vol. 7996 - Ensures that the 7997 preceding 7998 global/local/generic 7999 load 8000 atomic/atomicrmw 8001 with an equal or 8002 wider sync scope 8003 and memory ordering 8004 stronger than 8005 unordered (this is 8006 termed the 8007 acquire-fence-paired-atomic) 8008 has completed 8009 before invalidating 8010 the cache. This 8011 satisfies the 8012 requirements of 8013 acquire. 8014 - Ensures that all 8015 previous memory 8016 operations have 8017 completed before a 8018 following 8019 global/local/generic 8020 store 8021 atomic/atomicrmw 8022 with an equal or 8023 wider sync scope 8024 and memory ordering 8025 stronger than 8026 unordered (this is 8027 termed the 8028 release-fence-paired-atomic). 8029 This satisfies the 8030 requirements of 8031 release. 8032 8033 2. buffer_wbinvl1_vol 8034 8035 - Must happen before 8036 any following 8037 global/generic 8038 load/load 8039 atomic/store/store 8040 atomic/atomicrmw. 8041 - Ensures that 8042 following 8043 loads will not see 8044 stale L1 global data. 8045 MTYPE RW and CC memory will 8046 never be stale in L2 due to 8047 the memory probes. 8048 8049 **Sequential Consistent Atomic** 8050 ------------------------------------------------------------------------------------ 8051 load atomic seq_cst - singlethread - global *Same as corresponding 8052 - wavefront - local load atomic acquire, 8053 - generic except must generated 8054 all instructions even 8055 for OpenCL.* 8056 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8057 - generic 8058 - Use lgkmcnt(0) if not 8059 TgSplit execution mode 8060 and vmcnt(0) if TgSplit 8061 execution mode. 8062 - s_waitcnt lgkmcnt(0) must 8063 happen after 8064 preceding 8065 local/generic load 8066 atomic/store 8067 atomic/atomicrmw 8068 with memory 8069 ordering of seq_cst 8070 and with equal or 8071 wider sync scope. 8072 (Note that seq_cst 8073 fences have their 8074 own s_waitcnt 8075 lgkmcnt(0) and so do 8076 not need to be 8077 considered.) 8078 - s_waitcnt vmcnt(0) 8079 must happen after 8080 preceding 8081 global/generic load 8082 atomic/store 8083 atomic/atomicrmw 8084 with memory 8085 ordering of seq_cst 8086 and with equal or 8087 wider sync scope. 8088 (Note that seq_cst 8089 fences have their 8090 own s_waitcnt 8091 vmcnt(0) and so do 8092 not need to be 8093 considered.) 8094 - Ensures any 8095 preceding 8096 sequential 8097 consistent global/local 8098 memory instructions 8099 have completed 8100 before executing 8101 this sequentially 8102 consistent 8103 instruction. This 8104 prevents reordering 8105 a seq_cst store 8106 followed by a 8107 seq_cst load. (Note 8108 that seq_cst is 8109 stronger than 8110 acquire/release as 8111 the reordering of 8112 load acquire 8113 followed by a store 8114 release is 8115 prevented by the 8116 s_waitcnt of 8117 the release, but 8118 there is nothing 8119 preventing a store 8120 release followed by 8121 load acquire from 8122 completing out of 8123 order. The s_waitcnt 8124 could be placed after 8125 seq_store or before 8126 the seq_load. We 8127 choose the load to 8128 make the s_waitcnt be 8129 as late as possible 8130 so that the store 8131 may have already 8132 completed.) 8133 8134 2. *Following 8135 instructions same as 8136 corresponding load 8137 atomic acquire, 8138 except must generated 8139 all instructions even 8140 for OpenCL.* 8141 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8142 local address space cannot 8143 be used.* 8144 8145 *Same as corresponding 8146 load atomic acquire, 8147 except must generated 8148 all instructions even 8149 for OpenCL.* 8150 8151 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8152 - system - generic vmcnt(0) 8153 8154 - If TgSplit execution mode, 8155 omit lgkmcnt(0). 8156 - Could be split into 8157 separate s_waitcnt 8158 vmcnt(0) 8159 and s_waitcnt 8160 lgkmcnt(0) to allow 8161 them to be 8162 independently moved 8163 according to the 8164 following rules. 8165 - s_waitcnt lgkmcnt(0) 8166 must happen after 8167 preceding 8168 global/generic load 8169 atomic/store 8170 atomic/atomicrmw 8171 with memory 8172 ordering of seq_cst 8173 and with equal or 8174 wider sync scope. 8175 (Note that seq_cst 8176 fences have their 8177 own s_waitcnt 8178 lgkmcnt(0) and so do 8179 not need to be 8180 considered.) 8181 - s_waitcnt vmcnt(0) 8182 must happen after 8183 preceding 8184 global/generic load 8185 atomic/store 8186 atomic/atomicrmw 8187 with memory 8188 ordering of seq_cst 8189 and with equal or 8190 wider sync scope. 8191 (Note that seq_cst 8192 fences have their 8193 own s_waitcnt 8194 vmcnt(0) and so do 8195 not need to be 8196 considered.) 8197 - Ensures any 8198 preceding 8199 sequential 8200 consistent global 8201 memory instructions 8202 have completed 8203 before executing 8204 this sequentially 8205 consistent 8206 instruction. This 8207 prevents reordering 8208 a seq_cst store 8209 followed by a 8210 seq_cst load. (Note 8211 that seq_cst is 8212 stronger than 8213 acquire/release as 8214 the reordering of 8215 load acquire 8216 followed by a store 8217 release is 8218 prevented by the 8219 s_waitcnt of 8220 the release, but 8221 there is nothing 8222 preventing a store 8223 release followed by 8224 load acquire from 8225 completing out of 8226 order. The s_waitcnt 8227 could be placed after 8228 seq_store or before 8229 the seq_load. We 8230 choose the load to 8231 make the s_waitcnt be 8232 as late as possible 8233 so that the store 8234 may have already 8235 completed.) 8236 8237 2. *Following 8238 instructions same as 8239 corresponding load 8240 atomic acquire, 8241 except must generated 8242 all instructions even 8243 for OpenCL.* 8244 store atomic seq_cst - singlethread - global *Same as corresponding 8245 - wavefront - local store atomic release, 8246 - workgroup - generic except must generated 8247 - agent all instructions even 8248 - system for OpenCL.* 8249 atomicrmw seq_cst - singlethread - global *Same as corresponding 8250 - wavefront - local atomicrmw acq_rel, 8251 - workgroup - generic except must generated 8252 - agent all instructions even 8253 - system for OpenCL.* 8254 fence seq_cst - singlethread *none* *Same as corresponding 8255 - wavefront fence acq_rel, 8256 - workgroup except must generated 8257 - agent all instructions even 8258 - system for OpenCL.* 8259 ============ ============ ============== ========== ================================ 8260 8261.. _amdgpu-amdhsa-memory-model-gfx10: 8262 8263Memory Model GFX10 8264++++++++++++++++++ 8265 8266For GFX10: 8267 8268* Each agent has multiple shader arrays (SA). 8269* Each SA has multiple work-group processors (WGP). 8270* Each WGP has multiple compute units (CU). 8271* Each CU has multiple SIMDs that execute wavefronts. 8272* The wavefronts for a single work-group are executed in the same 8273 WGP. In CU wavefront execution mode the wavefronts may be executed by 8274 different SIMDs in the same CU. In WGP wavefront execution mode the 8275 wavefronts may be executed by different SIMDs in different CUs in the same 8276 WGP. 8277* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8278 executing on it. 8279* All LDS operations of a WGP are performed as wavefront wide operations in a 8280 global order and involve no caching. Completion is reported to a wavefront in 8281 execution order. 8282* The LDS memory has multiple request queues shared by the SIMDs of a 8283 WGP. Therefore, the LDS operations performed by different wavefronts of a 8284 work-group can be reordered relative to each other, which can result in 8285 reordering the visibility of vector memory operations with respect to LDS 8286 operations of other wavefronts in the same work-group. A ``s_waitcnt 8287 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8288 vector memory operations between wavefronts of a work-group, but not between 8289 operations performed by the same wavefront. 8290* The vector memory operations are performed as wavefront wide operations. 8291 Completion of load/store/sample operations are reported to a wavefront in 8292 execution order of other load/store/sample operations performed by that 8293 wavefront. 8294* The vector memory operations access a vector L0 cache. There is a single L0 8295 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8296 special action is required for coherence between the lanes of a single 8297 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8298 wavefronts executing in the same work-group as they may be executing on SIMDs 8299 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8300 required for coherence between wavefronts executing in different work-groups 8301 as they may be executing on different WGPs. 8302* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8303 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8304 operations are used in a restricted way so do not impact the memory model. See 8305 :ref:`amdgpu-amdhsa-memory-spaces`. 8306* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8307 the same SA. Therefore, no special action is required for coherence between 8308 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8309 required for coherence between wavefronts executing in different work-groups 8310 as they may be executing on different SAs that access different L1s. 8311* The L1 caches have independent quadrants to service disjoint ranges of virtual 8312 addresses. 8313* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8314 vector and scalar memory operations performed by different wavefronts, whether 8315 executing in the same or different work-groups (which may be executing on 8316 different CUs accessing different L0s), can be reordered relative to each 8317 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8318 synchronization between vector memory operations of different wavefronts. It 8319 ensures a previous vector memory operation has completed before executing a 8320 subsequent vector memory or LDS operation and so can be used to meet the 8321 requirements of acquire, release and sequential consistency. 8322* The L1 caches use an L2 cache shared by all SAs on the same agent. 8323* The L2 cache has independent channels to service disjoint ranges of virtual 8324 addresses. 8325* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8326 quadrant has a separate request queue per L2 channel. Therefore, the vector 8327 and scalar memory operations performed by wavefronts executing in different 8328 work-groups (which may be executing on different SAs) of an agent can be 8329 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8330 required to ensure synchronization between vector memory operations of 8331 different SAs. It ensures a previous vector memory operation has completed 8332 before executing a subsequent vector memory and so can be used to meet the 8333 requirements of acquire, release and sequential consistency. 8334* The L2 cache can be kept coherent with other agents on some targets, or ranges 8335 of virtual addresses can be set up to bypass it to ensure system coherence. 8336 8337Scalar memory operations are only used to access memory that is proven to not 8338change during the execution of the kernel dispatch. This includes constant 8339address space and global address space for program scope ``const`` variables. 8340Therefore, the kernel machine code does not have to maintain the scalar cache to 8341ensure it is coherent with the vector caches. The scalar and vector caches are 8342invalidated between kernel dispatches by CP since constant address space data 8343may change between kernel dispatch executions. See 8344:ref:`amdgpu-amdhsa-memory-spaces`. 8345 8346The one exception is if scalar writes are used to spill SGPR registers. In this 8347case the AMDGPU backend ensures the memory location used to spill is never 8348accessed by vector memory operations at the same time. If scalar writes are used 8349then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8350return since the locations may be used for vector memory instructions by a 8351future wavefront that uses the same scratch area, or a function call that 8352creates a frame at the same address, respectively. There is no need for a 8353``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8354 8355For kernarg backing memory: 8356 8357* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8358* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8359 needing to invalidate the L2 cache. 8360* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8361 so the L2 cache will be coherent with the CPU and other agents. 8362 8363Scratch backing memory (which is used for the private address space) is accessed 8364with MTYPE NC (non-coherent). Since the private address space is only accessed 8365by a single thread, and is always write-before-read, there is never a need to 8366invalidate these entries from the L0 or L1 caches. 8367 8368Wavefronts are executed in native mode with in-order reporting of loads and 8369sample instructions. In this mode vmcnt reports completion of load, atomic with 8370return and sample instructions in order, and the vscnt reports the completion of 8371store and atomic without return in order. See ``MEM_ORDERED`` field in 8372:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8373 8374Wavefronts can be executed in WGP or CU wavefront execution mode: 8375 8376* In WGP wavefront execution mode the wavefronts of a work-group are executed 8377 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8378 CU L0 caches is required for work-group synchronization. Also accesses to L1 8379 at work-group scope need to be explicitly ordered as the accesses from 8380 different CUs are not ordered. 8381* In CU wavefront execution mode the wavefronts of a work-group are executed on 8382 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8383 the work-group access the same L0 which in turn ensures L1 accesses are 8384 ordered and so do not require explicit management of the caches for 8385 work-group synchronization. 8386 8387See ``WGP_MODE`` field in 8388:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8389:ref:`amdgpu-target-features`. 8390 8391The code sequences used to implement the memory model for GFX10 are defined in 8392table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8393 8394 .. table:: AMDHSA Memory Model Code Sequences GFX10 8395 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8396 8397 ============ ============ ============== ========== ================================ 8398 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8399 Ordering Sync Scope Address GFX10 8400 Space 8401 ============ ============ ============== ========== ================================ 8402 **Non-Atomic** 8403 ------------------------------------------------------------------------------------ 8404 load *none* *none* - global - !volatile & !nontemporal 8405 - generic 8406 - private 1. buffer/global/flat_load 8407 - constant 8408 - !volatile & nontemporal 8409 8410 1. buffer/global/flat_load 8411 slc=1 8412 8413 - volatile 8414 8415 1. buffer/global/flat_load 8416 glc=1 dlc=1 8417 2. s_waitcnt vmcnt(0) 8418 8419 - Must happen before 8420 any following volatile 8421 global/generic 8422 load/store. 8423 - Ensures that 8424 volatile 8425 operations to 8426 different 8427 addresses will not 8428 be reordered by 8429 hardware. 8430 8431 load *none* *none* - local 1. ds_load 8432 store *none* *none* - global - !volatile & !nontemporal 8433 - generic 8434 - private 1. buffer/global/flat_store 8435 - constant 8436 - !volatile & nontemporal 8437 8438 1. buffer/global/flat_store 8439 slc=1 8440 8441 - volatile 8442 8443 1. buffer/global/flat_store 8444 2. s_waitcnt vscnt(0) 8445 8446 - Must happen before 8447 any following volatile 8448 global/generic 8449 load/store. 8450 - Ensures that 8451 volatile 8452 operations to 8453 different 8454 addresses will not 8455 be reordered by 8456 hardware. 8457 8458 store *none* *none* - local 1. ds_store 8459 **Unordered Atomic** 8460 ------------------------------------------------------------------------------------ 8461 load atomic unordered *any* *any* *Same as non-atomic*. 8462 store atomic unordered *any* *any* *Same as non-atomic*. 8463 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8464 **Monotonic Atomic** 8465 ------------------------------------------------------------------------------------ 8466 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8467 - wavefront - generic 8468 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8469 - generic glc=1 8470 8471 - If CU wavefront execution 8472 mode, omit glc=1. 8473 8474 load atomic monotonic - singlethread - local 1. ds_load 8475 - wavefront 8476 - workgroup 8477 load atomic monotonic - agent - global 1. buffer/global/flat_load 8478 - system - generic glc=1 dlc=1 8479 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8480 - wavefront - generic 8481 - workgroup 8482 - agent 8483 - system 8484 store atomic monotonic - singlethread - local 1. ds_store 8485 - wavefront 8486 - workgroup 8487 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8488 - wavefront - generic 8489 - workgroup 8490 - agent 8491 - system 8492 atomicrmw monotonic - singlethread - local 1. ds_atomic 8493 - wavefront 8494 - workgroup 8495 **Acquire Atomic** 8496 ------------------------------------------------------------------------------------ 8497 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8498 - wavefront - local 8499 - generic 8500 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8501 8502 - If CU wavefront execution 8503 mode, omit glc=1. 8504 8505 2. s_waitcnt vmcnt(0) 8506 8507 - If CU wavefront execution 8508 mode, omit. 8509 - Must happen before 8510 the following buffer_gl0_inv 8511 and before any following 8512 global/generic 8513 load/load 8514 atomic/store/store 8515 atomic/atomicrmw. 8516 8517 3. buffer_gl0_inv 8518 8519 - If CU wavefront execution 8520 mode, omit. 8521 - Ensures that 8522 following 8523 loads will not see 8524 stale data. 8525 8526 load atomic acquire - workgroup - local 1. ds_load 8527 2. s_waitcnt lgkmcnt(0) 8528 8529 - If OpenCL, omit. 8530 - Must happen before 8531 the following buffer_gl0_inv 8532 and before any following 8533 global/generic load/load 8534 atomic/store/store 8535 atomic/atomicrmw. 8536 - Ensures any 8537 following global 8538 data read is no 8539 older than the local load 8540 atomic value being 8541 acquired. 8542 8543 3. buffer_gl0_inv 8544 8545 - If CU wavefront execution 8546 mode, omit. 8547 - If OpenCL, omit. 8548 - Ensures that 8549 following 8550 loads will not see 8551 stale data. 8552 8553 load atomic acquire - workgroup - generic 1. flat_load glc=1 8554 8555 - If CU wavefront execution 8556 mode, omit glc=1. 8557 8558 2. s_waitcnt lgkmcnt(0) & 8559 vmcnt(0) 8560 8561 - If CU wavefront execution 8562 mode, omit vmcnt(0). 8563 - If OpenCL, omit 8564 lgkmcnt(0). 8565 - Must happen before 8566 the following 8567 buffer_gl0_inv and any 8568 following global/generic 8569 load/load 8570 atomic/store/store 8571 atomic/atomicrmw. 8572 - Ensures any 8573 following global 8574 data read is no 8575 older than a local load 8576 atomic value being 8577 acquired. 8578 8579 3. buffer_gl0_inv 8580 8581 - If CU wavefront execution 8582 mode, omit. 8583 - Ensures that 8584 following 8585 loads will not see 8586 stale data. 8587 8588 load atomic acquire - agent - global 1. buffer/global_load 8589 - system glc=1 dlc=1 8590 2. s_waitcnt vmcnt(0) 8591 8592 - Must happen before 8593 following 8594 buffer_gl*_inv. 8595 - Ensures the load 8596 has completed 8597 before invalidating 8598 the caches. 8599 8600 3. buffer_gl0_inv; 8601 buffer_gl1_inv 8602 8603 - Must happen before 8604 any following 8605 global/generic 8606 load/load 8607 atomic/atomicrmw. 8608 - Ensures that 8609 following 8610 loads will not see 8611 stale global data. 8612 8613 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8614 - system 2. s_waitcnt vmcnt(0) & 8615 lgkmcnt(0) 8616 8617 - If OpenCL omit 8618 lgkmcnt(0). 8619 - Must happen before 8620 following 8621 buffer_gl*_invl. 8622 - Ensures the flat_load 8623 has completed 8624 before invalidating 8625 the caches. 8626 8627 3. buffer_gl0_inv; 8628 buffer_gl1_inv 8629 8630 - Must happen before 8631 any following 8632 global/generic 8633 load/load 8634 atomic/atomicrmw. 8635 - Ensures that 8636 following loads 8637 will not see stale 8638 global data. 8639 8640 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8641 - wavefront - local 8642 - generic 8643 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8644 2. s_waitcnt vm/vscnt(0) 8645 8646 - If CU wavefront execution 8647 mode, omit. 8648 - Use vmcnt(0) if atomic with 8649 return and vscnt(0) if 8650 atomic with no-return. 8651 - Must happen before 8652 the following buffer_gl0_inv 8653 and before any following 8654 global/generic 8655 load/load 8656 atomic/store/store 8657 atomic/atomicrmw. 8658 8659 3. buffer_gl0_inv 8660 8661 - If CU wavefront execution 8662 mode, omit. 8663 - Ensures that 8664 following 8665 loads will not see 8666 stale data. 8667 8668 atomicrmw acquire - workgroup - local 1. ds_atomic 8669 2. s_waitcnt lgkmcnt(0) 8670 8671 - If OpenCL, omit. 8672 - Must happen before 8673 the following 8674 buffer_gl0_inv. 8675 - Ensures any 8676 following global 8677 data read is no 8678 older than the local 8679 atomicrmw value 8680 being acquired. 8681 8682 3. buffer_gl0_inv 8683 8684 - If OpenCL omit. 8685 - Ensures that 8686 following 8687 loads will not see 8688 stale data. 8689 8690 atomicrmw acquire - workgroup - generic 1. flat_atomic 8691 2. s_waitcnt lgkmcnt(0) & 8692 vm/vscnt(0) 8693 8694 - If CU wavefront execution 8695 mode, omit vm/vscnt(0). 8696 - If OpenCL, omit lgkmcnt(0). 8697 - Use vmcnt(0) if atomic with 8698 return and vscnt(0) if 8699 atomic with no-return. 8700 - Must happen before 8701 the following 8702 buffer_gl0_inv. 8703 - Ensures any 8704 following global 8705 data read is no 8706 older than a local 8707 atomicrmw value 8708 being acquired. 8709 8710 3. buffer_gl0_inv 8711 8712 - If CU wavefront execution 8713 mode, omit. 8714 - Ensures that 8715 following 8716 loads will not see 8717 stale data. 8718 8719 atomicrmw acquire - agent - global 1. buffer/global_atomic 8720 - system 2. s_waitcnt vm/vscnt(0) 8721 8722 - Use vmcnt(0) if atomic with 8723 return and vscnt(0) if 8724 atomic with no-return. 8725 - Must happen before 8726 following 8727 buffer_gl*_inv. 8728 - Ensures the 8729 atomicrmw has 8730 completed before 8731 invalidating the 8732 caches. 8733 8734 3. buffer_gl0_inv; 8735 buffer_gl1_inv 8736 8737 - Must happen before 8738 any following 8739 global/generic 8740 load/load 8741 atomic/atomicrmw. 8742 - Ensures that 8743 following loads 8744 will not see stale 8745 global data. 8746 8747 atomicrmw acquire - agent - generic 1. flat_atomic 8748 - system 2. s_waitcnt vm/vscnt(0) & 8749 lgkmcnt(0) 8750 8751 - If OpenCL, omit 8752 lgkmcnt(0). 8753 - Use vmcnt(0) if atomic with 8754 return and vscnt(0) if 8755 atomic with no-return. 8756 - Must happen before 8757 following 8758 buffer_gl*_inv. 8759 - Ensures the 8760 atomicrmw has 8761 completed before 8762 invalidating the 8763 caches. 8764 8765 3. buffer_gl0_inv; 8766 buffer_gl1_inv 8767 8768 - Must happen before 8769 any following 8770 global/generic 8771 load/load 8772 atomic/atomicrmw. 8773 - Ensures that 8774 following loads 8775 will not see stale 8776 global data. 8777 8778 fence acquire - singlethread *none* *none* 8779 - wavefront 8780 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 8781 vmcnt(0) & vscnt(0) 8782 8783 - If CU wavefront execution 8784 mode, omit vmcnt(0) and 8785 vscnt(0). 8786 - If OpenCL and 8787 address space is 8788 not generic, omit 8789 lgkmcnt(0). 8790 - If OpenCL and 8791 address space is 8792 local, omit 8793 vmcnt(0) and vscnt(0). 8794 - However, since LLVM 8795 currently has no 8796 address space on 8797 the fence need to 8798 conservatively 8799 always generate. If 8800 fence had an 8801 address space then 8802 set to address 8803 space of OpenCL 8804 fence flag, or to 8805 generic if both 8806 local and global 8807 flags are 8808 specified. 8809 - Could be split into 8810 separate s_waitcnt 8811 vmcnt(0), s_waitcnt 8812 vscnt(0) and s_waitcnt 8813 lgkmcnt(0) to allow 8814 them to be 8815 independently moved 8816 according to the 8817 following rules. 8818 - s_waitcnt vmcnt(0) 8819 must happen after 8820 any preceding 8821 global/generic load 8822 atomic/ 8823 atomicrmw-with-return-value 8824 with an equal or 8825 wider sync scope 8826 and memory ordering 8827 stronger than 8828 unordered (this is 8829 termed the 8830 fence-paired-atomic). 8831 - s_waitcnt vscnt(0) 8832 must happen after 8833 any preceding 8834 global/generic 8835 atomicrmw-no-return-value 8836 with an equal or 8837 wider sync scope 8838 and memory ordering 8839 stronger than 8840 unordered (this is 8841 termed the 8842 fence-paired-atomic). 8843 - s_waitcnt lgkmcnt(0) 8844 must happen after 8845 any preceding 8846 local/generic load 8847 atomic/atomicrmw 8848 with an equal or 8849 wider sync scope 8850 and memory ordering 8851 stronger than 8852 unordered (this is 8853 termed the 8854 fence-paired-atomic). 8855 - Must happen before 8856 the following 8857 buffer_gl0_inv. 8858 - Ensures that the 8859 fence-paired atomic 8860 has completed 8861 before invalidating 8862 the 8863 cache. Therefore 8864 any following 8865 locations read must 8866 be no older than 8867 the value read by 8868 the 8869 fence-paired-atomic. 8870 8871 3. buffer_gl0_inv 8872 8873 - If CU wavefront execution 8874 mode, omit. 8875 - Ensures that 8876 following 8877 loads will not see 8878 stale data. 8879 8880 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 8881 - system vmcnt(0) & vscnt(0) 8882 8883 - If OpenCL and 8884 address space is 8885 not generic, omit 8886 lgkmcnt(0). 8887 - If OpenCL and 8888 address space is 8889 local, omit 8890 vmcnt(0) and vscnt(0). 8891 - However, since LLVM 8892 currently has no 8893 address space on 8894 the fence need to 8895 conservatively 8896 always generate 8897 (see comment for 8898 previous fence). 8899 - Could be split into 8900 separate s_waitcnt 8901 vmcnt(0), s_waitcnt 8902 vscnt(0) and s_waitcnt 8903 lgkmcnt(0) to allow 8904 them to be 8905 independently moved 8906 according to the 8907 following rules. 8908 - s_waitcnt vmcnt(0) 8909 must happen after 8910 any preceding 8911 global/generic load 8912 atomic/ 8913 atomicrmw-with-return-value 8914 with an equal or 8915 wider sync scope 8916 and memory ordering 8917 stronger than 8918 unordered (this is 8919 termed the 8920 fence-paired-atomic). 8921 - s_waitcnt vscnt(0) 8922 must happen after 8923 any preceding 8924 global/generic 8925 atomicrmw-no-return-value 8926 with an equal or 8927 wider sync scope 8928 and memory ordering 8929 stronger than 8930 unordered (this is 8931 termed the 8932 fence-paired-atomic). 8933 - s_waitcnt lgkmcnt(0) 8934 must happen after 8935 any preceding 8936 local/generic load 8937 atomic/atomicrmw 8938 with an equal or 8939 wider sync scope 8940 and memory ordering 8941 stronger than 8942 unordered (this is 8943 termed the 8944 fence-paired-atomic). 8945 - Must happen before 8946 the following 8947 buffer_gl*_inv. 8948 - Ensures that the 8949 fence-paired atomic 8950 has completed 8951 before invalidating 8952 the 8953 caches. Therefore 8954 any following 8955 locations read must 8956 be no older than 8957 the value read by 8958 the 8959 fence-paired-atomic. 8960 8961 2. buffer_gl0_inv; 8962 buffer_gl1_inv 8963 8964 - Must happen before any 8965 following global/generic 8966 load/load 8967 atomic/store/store 8968 atomic/atomicrmw. 8969 - Ensures that 8970 following loads 8971 will not see stale 8972 global data. 8973 8974 **Release Atomic** 8975 ------------------------------------------------------------------------------------ 8976 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 8977 - wavefront - local 8978 - generic 8979 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 8980 - generic vmcnt(0) & vscnt(0) 8981 8982 - If CU wavefront execution 8983 mode, omit vmcnt(0) and 8984 vscnt(0). 8985 - If OpenCL, omit 8986 lgkmcnt(0). 8987 - Could be split into 8988 separate s_waitcnt 8989 vmcnt(0), s_waitcnt 8990 vscnt(0) and s_waitcnt 8991 lgkmcnt(0) to allow 8992 them to be 8993 independently moved 8994 according to the 8995 following rules. 8996 - s_waitcnt vmcnt(0) 8997 must happen after 8998 any preceding 8999 global/generic load/load 9000 atomic/ 9001 atomicrmw-with-return-value. 9002 - s_waitcnt vscnt(0) 9003 must happen after 9004 any preceding 9005 global/generic 9006 store/store 9007 atomic/ 9008 atomicrmw-no-return-value. 9009 - s_waitcnt lgkmcnt(0) 9010 must happen after 9011 any preceding 9012 local/generic 9013 load/store/load 9014 atomic/store 9015 atomic/atomicrmw. 9016 - Must happen before 9017 the following 9018 store. 9019 - Ensures that all 9020 memory operations 9021 have 9022 completed before 9023 performing the 9024 store that is being 9025 released. 9026 9027 2. buffer/global/flat_store 9028 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9029 9030 - If CU wavefront execution 9031 mode, omit. 9032 - If OpenCL, omit. 9033 - Could be split into 9034 separate s_waitcnt 9035 vmcnt(0) and s_waitcnt 9036 vscnt(0) to allow 9037 them to be 9038 independently moved 9039 according to the 9040 following rules. 9041 - s_waitcnt vmcnt(0) 9042 must happen after 9043 any preceding 9044 global/generic load/load 9045 atomic/ 9046 atomicrmw-with-return-value. 9047 - s_waitcnt vscnt(0) 9048 must happen after 9049 any preceding 9050 global/generic 9051 store/store atomic/ 9052 atomicrmw-no-return-value. 9053 - Must happen before 9054 the following 9055 store. 9056 - Ensures that all 9057 global memory 9058 operations have 9059 completed before 9060 performing the 9061 store that is being 9062 released. 9063 9064 2. ds_store 9065 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9066 - system - generic vmcnt(0) & vscnt(0) 9067 9068 - If OpenCL and 9069 address space is 9070 not generic, omit 9071 lgkmcnt(0). 9072 - Could be split into 9073 separate s_waitcnt 9074 vmcnt(0), s_waitcnt vscnt(0) 9075 and s_waitcnt 9076 lgkmcnt(0) to allow 9077 them to be 9078 independently moved 9079 according to the 9080 following rules. 9081 - s_waitcnt vmcnt(0) 9082 must happen after 9083 any preceding 9084 global/generic 9085 load/load 9086 atomic/ 9087 atomicrmw-with-return-value. 9088 - s_waitcnt vscnt(0) 9089 must happen after 9090 any preceding 9091 global/generic 9092 store/store atomic/ 9093 atomicrmw-no-return-value. 9094 - s_waitcnt lgkmcnt(0) 9095 must happen after 9096 any preceding 9097 local/generic 9098 load/store/load 9099 atomic/store 9100 atomic/atomicrmw. 9101 - Must happen before 9102 the following 9103 store. 9104 - Ensures that all 9105 memory operations 9106 have 9107 completed before 9108 performing the 9109 store that is being 9110 released. 9111 9112 2. buffer/global/flat_store 9113 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9114 - wavefront - local 9115 - generic 9116 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9117 - generic vmcnt(0) & vscnt(0) 9118 9119 - If CU wavefront execution 9120 mode, omit vmcnt(0) and 9121 vscnt(0). 9122 - If OpenCL, omit lgkmcnt(0). 9123 - Could be split into 9124 separate s_waitcnt 9125 vmcnt(0), s_waitcnt 9126 vscnt(0) and s_waitcnt 9127 lgkmcnt(0) to allow 9128 them to be 9129 independently moved 9130 according to the 9131 following rules. 9132 - s_waitcnt vmcnt(0) 9133 must happen after 9134 any preceding 9135 global/generic load/load 9136 atomic/ 9137 atomicrmw-with-return-value. 9138 - s_waitcnt vscnt(0) 9139 must happen after 9140 any preceding 9141 global/generic 9142 store/store 9143 atomic/ 9144 atomicrmw-no-return-value. 9145 - s_waitcnt lgkmcnt(0) 9146 must happen after 9147 any preceding 9148 local/generic 9149 load/store/load 9150 atomic/store 9151 atomic/atomicrmw. 9152 - Must happen before 9153 the following 9154 atomicrmw. 9155 - Ensures that all 9156 memory operations 9157 have 9158 completed before 9159 performing the 9160 atomicrmw that is 9161 being released. 9162 9163 2. buffer/global/flat_atomic 9164 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9165 9166 - If CU wavefront execution 9167 mode, omit. 9168 - If OpenCL, omit. 9169 - Could be split into 9170 separate s_waitcnt 9171 vmcnt(0) and s_waitcnt 9172 vscnt(0) to allow 9173 them to be 9174 independently moved 9175 according to the 9176 following rules. 9177 - s_waitcnt vmcnt(0) 9178 must happen after 9179 any preceding 9180 global/generic load/load 9181 atomic/ 9182 atomicrmw-with-return-value. 9183 - s_waitcnt vscnt(0) 9184 must happen after 9185 any preceding 9186 global/generic 9187 store/store atomic/ 9188 atomicrmw-no-return-value. 9189 - Must happen before 9190 the following 9191 store. 9192 - Ensures that all 9193 global memory 9194 operations have 9195 completed before 9196 performing the 9197 store that is being 9198 released. 9199 9200 2. ds_atomic 9201 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9202 - system - generic vmcnt(0) & vscnt(0) 9203 9204 - If OpenCL, omit 9205 lgkmcnt(0). 9206 - Could be split into 9207 separate s_waitcnt 9208 vmcnt(0), s_waitcnt 9209 vscnt(0) and s_waitcnt 9210 lgkmcnt(0) to allow 9211 them to be 9212 independently moved 9213 according to the 9214 following rules. 9215 - s_waitcnt vmcnt(0) 9216 must happen after 9217 any preceding 9218 global/generic 9219 load/load atomic/ 9220 atomicrmw-with-return-value. 9221 - s_waitcnt vscnt(0) 9222 must happen after 9223 any preceding 9224 global/generic 9225 store/store atomic/ 9226 atomicrmw-no-return-value. 9227 - s_waitcnt lgkmcnt(0) 9228 must happen after 9229 any preceding 9230 local/generic 9231 load/store/load 9232 atomic/store 9233 atomic/atomicrmw. 9234 - Must happen before 9235 the following 9236 atomicrmw. 9237 - Ensures that all 9238 memory operations 9239 to global and local 9240 have completed 9241 before performing 9242 the atomicrmw that 9243 is being released. 9244 9245 2. buffer/global/flat_atomic 9246 fence release - singlethread *none* *none* 9247 - wavefront 9248 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9249 vmcnt(0) & vscnt(0) 9250 9251 - If CU wavefront execution 9252 mode, omit vmcnt(0) and 9253 vscnt(0). 9254 - If OpenCL and 9255 address space is 9256 not generic, omit 9257 lgkmcnt(0). 9258 - If OpenCL and 9259 address space is 9260 local, omit 9261 vmcnt(0) and vscnt(0). 9262 - However, since LLVM 9263 currently has no 9264 address space on 9265 the fence need to 9266 conservatively 9267 always generate. If 9268 fence had an 9269 address space then 9270 set to address 9271 space of OpenCL 9272 fence flag, or to 9273 generic if both 9274 local and global 9275 flags are 9276 specified. 9277 - Could be split into 9278 separate s_waitcnt 9279 vmcnt(0), s_waitcnt 9280 vscnt(0) and s_waitcnt 9281 lgkmcnt(0) to allow 9282 them to be 9283 independently moved 9284 according to the 9285 following rules. 9286 - s_waitcnt vmcnt(0) 9287 must happen after 9288 any preceding 9289 global/generic 9290 load/load 9291 atomic/ 9292 atomicrmw-with-return-value. 9293 - s_waitcnt vscnt(0) 9294 must happen after 9295 any preceding 9296 global/generic 9297 store/store atomic/ 9298 atomicrmw-no-return-value. 9299 - s_waitcnt lgkmcnt(0) 9300 must happen after 9301 any preceding 9302 local/generic 9303 load/store/load 9304 atomic/store atomic/ 9305 atomicrmw. 9306 - Must happen before 9307 any following store 9308 atomic/atomicrmw 9309 with an equal or 9310 wider sync scope 9311 and memory ordering 9312 stronger than 9313 unordered (this is 9314 termed the 9315 fence-paired-atomic). 9316 - Ensures that all 9317 memory operations 9318 have 9319 completed before 9320 performing the 9321 following 9322 fence-paired-atomic. 9323 9324 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9325 - system vmcnt(0) & vscnt(0) 9326 9327 - If OpenCL and 9328 address space is 9329 not generic, omit 9330 lgkmcnt(0). 9331 - If OpenCL and 9332 address space is 9333 local, omit 9334 vmcnt(0) and vscnt(0). 9335 - However, since LLVM 9336 currently has no 9337 address space on 9338 the fence need to 9339 conservatively 9340 always generate. If 9341 fence had an 9342 address space then 9343 set to address 9344 space of OpenCL 9345 fence flag, or to 9346 generic if both 9347 local and global 9348 flags are 9349 specified. 9350 - Could be split into 9351 separate s_waitcnt 9352 vmcnt(0), s_waitcnt 9353 vscnt(0) and s_waitcnt 9354 lgkmcnt(0) to allow 9355 them to be 9356 independently moved 9357 according to the 9358 following rules. 9359 - s_waitcnt vmcnt(0) 9360 must happen after 9361 any preceding 9362 global/generic 9363 load/load atomic/ 9364 atomicrmw-with-return-value. 9365 - s_waitcnt vscnt(0) 9366 must happen after 9367 any preceding 9368 global/generic 9369 store/store atomic/ 9370 atomicrmw-no-return-value. 9371 - s_waitcnt lgkmcnt(0) 9372 must happen after 9373 any preceding 9374 local/generic 9375 load/store/load 9376 atomic/store 9377 atomic/atomicrmw. 9378 - Must happen before 9379 any following store 9380 atomic/atomicrmw 9381 with an equal or 9382 wider sync scope 9383 and memory ordering 9384 stronger than 9385 unordered (this is 9386 termed the 9387 fence-paired-atomic). 9388 - Ensures that all 9389 memory operations 9390 have 9391 completed before 9392 performing the 9393 following 9394 fence-paired-atomic. 9395 9396 **Acquire-Release Atomic** 9397 ------------------------------------------------------------------------------------ 9398 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9399 - wavefront - local 9400 - generic 9401 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9402 vmcnt(0) & vscnt(0) 9403 9404 - If CU wavefront execution 9405 mode, omit vmcnt(0) and 9406 vscnt(0). 9407 - If OpenCL, omit 9408 lgkmcnt(0). 9409 - Must happen after 9410 any preceding 9411 local/generic 9412 load/store/load 9413 atomic/store 9414 atomic/atomicrmw. 9415 - Could be split into 9416 separate s_waitcnt 9417 vmcnt(0), s_waitcnt 9418 vscnt(0), and s_waitcnt 9419 lgkmcnt(0) to allow 9420 them to be 9421 independently moved 9422 according to the 9423 following rules. 9424 - s_waitcnt vmcnt(0) 9425 must happen after 9426 any preceding 9427 global/generic load/load 9428 atomic/ 9429 atomicrmw-with-return-value. 9430 - s_waitcnt vscnt(0) 9431 must happen after 9432 any preceding 9433 global/generic 9434 store/store 9435 atomic/ 9436 atomicrmw-no-return-value. 9437 - s_waitcnt lgkmcnt(0) 9438 must happen after 9439 any preceding 9440 local/generic 9441 load/store/load 9442 atomic/store 9443 atomic/atomicrmw. 9444 - Must happen before 9445 the following 9446 atomicrmw. 9447 - Ensures that all 9448 memory operations 9449 have 9450 completed before 9451 performing the 9452 atomicrmw that is 9453 being released. 9454 9455 2. buffer/global_atomic 9456 3. s_waitcnt vm/vscnt(0) 9457 9458 - If CU wavefront execution 9459 mode, omit. 9460 - Use vmcnt(0) if atomic with 9461 return and vscnt(0) if 9462 atomic with no-return. 9463 - Must happen before 9464 the following 9465 buffer_gl0_inv. 9466 - Ensures any 9467 following global 9468 data read is no 9469 older than the 9470 atomicrmw value 9471 being acquired. 9472 9473 4. buffer_gl0_inv 9474 9475 - If CU wavefront execution 9476 mode, omit. 9477 - Ensures that 9478 following 9479 loads will not see 9480 stale data. 9481 9482 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9483 9484 - If CU wavefront execution 9485 mode, omit. 9486 - If OpenCL, omit. 9487 - Could be split into 9488 separate s_waitcnt 9489 vmcnt(0) and s_waitcnt 9490 vscnt(0) to allow 9491 them to be 9492 independently moved 9493 according to the 9494 following rules. 9495 - s_waitcnt vmcnt(0) 9496 must happen after 9497 any preceding 9498 global/generic load/load 9499 atomic/ 9500 atomicrmw-with-return-value. 9501 - s_waitcnt vscnt(0) 9502 must happen after 9503 any preceding 9504 global/generic 9505 store/store atomic/ 9506 atomicrmw-no-return-value. 9507 - Must happen before 9508 the following 9509 store. 9510 - Ensures that all 9511 global memory 9512 operations have 9513 completed before 9514 performing the 9515 store that is being 9516 released. 9517 9518 2. ds_atomic 9519 3. s_waitcnt lgkmcnt(0) 9520 9521 - If OpenCL, omit. 9522 - Must happen before 9523 the following 9524 buffer_gl0_inv. 9525 - Ensures any 9526 following global 9527 data read is no 9528 older than the local load 9529 atomic value being 9530 acquired. 9531 9532 4. buffer_gl0_inv 9533 9534 - If CU wavefront execution 9535 mode, omit. 9536 - If OpenCL omit. 9537 - Ensures that 9538 following 9539 loads will not see 9540 stale data. 9541 9542 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9543 vmcnt(0) & vscnt(0) 9544 9545 - If CU wavefront execution 9546 mode, omit vmcnt(0) and 9547 vscnt(0). 9548 - If OpenCL, omit lgkmcnt(0). 9549 - Could be split into 9550 separate s_waitcnt 9551 vmcnt(0), s_waitcnt 9552 vscnt(0) and s_waitcnt 9553 lgkmcnt(0) to allow 9554 them to be 9555 independently moved 9556 according to the 9557 following rules. 9558 - s_waitcnt vmcnt(0) 9559 must happen after 9560 any preceding 9561 global/generic load/load 9562 atomic/ 9563 atomicrmw-with-return-value. 9564 - s_waitcnt vscnt(0) 9565 must happen after 9566 any preceding 9567 global/generic 9568 store/store 9569 atomic/ 9570 atomicrmw-no-return-value. 9571 - s_waitcnt lgkmcnt(0) 9572 must happen after 9573 any preceding 9574 local/generic 9575 load/store/load 9576 atomic/store 9577 atomic/atomicrmw. 9578 - Must happen before 9579 the following 9580 atomicrmw. 9581 - Ensures that all 9582 memory operations 9583 have 9584 completed before 9585 performing the 9586 atomicrmw that is 9587 being released. 9588 9589 2. flat_atomic 9590 3. s_waitcnt lgkmcnt(0) & 9591 vmcnt(0) & vscnt(0) 9592 9593 - If CU wavefront execution 9594 mode, omit vmcnt(0) and 9595 vscnt(0). 9596 - If OpenCL, omit lgkmcnt(0). 9597 - Must happen before 9598 the following 9599 buffer_gl0_inv. 9600 - Ensures any 9601 following global 9602 data read is no 9603 older than the load 9604 atomic value being 9605 acquired. 9606 9607 3. buffer_gl0_inv 9608 9609 - If CU wavefront execution 9610 mode, omit. 9611 - Ensures that 9612 following 9613 loads will not see 9614 stale data. 9615 9616 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9617 - system vmcnt(0) & vscnt(0) 9618 9619 - If OpenCL, omit 9620 lgkmcnt(0). 9621 - Could be split into 9622 separate s_waitcnt 9623 vmcnt(0), s_waitcnt 9624 vscnt(0) and s_waitcnt 9625 lgkmcnt(0) to allow 9626 them to be 9627 independently moved 9628 according to the 9629 following rules. 9630 - s_waitcnt vmcnt(0) 9631 must happen after 9632 any preceding 9633 global/generic 9634 load/load atomic/ 9635 atomicrmw-with-return-value. 9636 - s_waitcnt vscnt(0) 9637 must happen after 9638 any preceding 9639 global/generic 9640 store/store atomic/ 9641 atomicrmw-no-return-value. 9642 - s_waitcnt lgkmcnt(0) 9643 must happen after 9644 any preceding 9645 local/generic 9646 load/store/load 9647 atomic/store 9648 atomic/atomicrmw. 9649 - Must happen before 9650 the following 9651 atomicrmw. 9652 - Ensures that all 9653 memory operations 9654 to global have 9655 completed before 9656 performing the 9657 atomicrmw that is 9658 being released. 9659 9660 2. buffer/global_atomic 9661 3. s_waitcnt vm/vscnt(0) 9662 9663 - Use vmcnt(0) if atomic with 9664 return and vscnt(0) if 9665 atomic with no-return. 9666 - Must happen before 9667 following 9668 buffer_gl*_inv. 9669 - Ensures the 9670 atomicrmw has 9671 completed before 9672 invalidating the 9673 caches. 9674 9675 4. buffer_gl0_inv; 9676 buffer_gl1_inv 9677 9678 - Must happen before 9679 any following 9680 global/generic 9681 load/load 9682 atomic/atomicrmw. 9683 - Ensures that 9684 following loads 9685 will not see stale 9686 global data. 9687 9688 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9689 - system vmcnt(0) & vscnt(0) 9690 9691 - If OpenCL, omit 9692 lgkmcnt(0). 9693 - Could be split into 9694 separate s_waitcnt 9695 vmcnt(0), s_waitcnt 9696 vscnt(0), and s_waitcnt 9697 lgkmcnt(0) to allow 9698 them to be 9699 independently moved 9700 according to the 9701 following rules. 9702 - s_waitcnt vmcnt(0) 9703 must happen after 9704 any preceding 9705 global/generic 9706 load/load atomic 9707 atomicrmw-with-return-value. 9708 - s_waitcnt vscnt(0) 9709 must happen after 9710 any preceding 9711 global/generic 9712 store/store atomic/ 9713 atomicrmw-no-return-value. 9714 - s_waitcnt lgkmcnt(0) 9715 must happen after 9716 any preceding 9717 local/generic 9718 load/store/load 9719 atomic/store 9720 atomic/atomicrmw. 9721 - Must happen before 9722 the following 9723 atomicrmw. 9724 - Ensures that all 9725 memory operations 9726 have 9727 completed before 9728 performing the 9729 atomicrmw that is 9730 being released. 9731 9732 2. flat_atomic 9733 3. s_waitcnt vm/vscnt(0) & 9734 lgkmcnt(0) 9735 9736 - If OpenCL, omit 9737 lgkmcnt(0). 9738 - Use vmcnt(0) if atomic with 9739 return and vscnt(0) if 9740 atomic with no-return. 9741 - Must happen before 9742 following 9743 buffer_gl*_inv. 9744 - Ensures the 9745 atomicrmw has 9746 completed before 9747 invalidating the 9748 caches. 9749 9750 4. buffer_gl0_inv; 9751 buffer_gl1_inv 9752 9753 - Must happen before 9754 any following 9755 global/generic 9756 load/load 9757 atomic/atomicrmw. 9758 - Ensures that 9759 following loads 9760 will not see stale 9761 global data. 9762 9763 fence acq_rel - singlethread *none* *none* 9764 - wavefront 9765 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9766 vmcnt(0) & vscnt(0) 9767 9768 - If CU wavefront execution 9769 mode, omit vmcnt(0) and 9770 vscnt(0). 9771 - If OpenCL and 9772 address space is 9773 not generic, omit 9774 lgkmcnt(0). 9775 - If OpenCL and 9776 address space is 9777 local, omit 9778 vmcnt(0) and vscnt(0). 9779 - However, 9780 since LLVM 9781 currently has no 9782 address space on 9783 the fence need to 9784 conservatively 9785 always generate 9786 (see comment for 9787 previous fence). 9788 - Could be split into 9789 separate s_waitcnt 9790 vmcnt(0), s_waitcnt 9791 vscnt(0) and s_waitcnt 9792 lgkmcnt(0) to allow 9793 them to be 9794 independently moved 9795 according to the 9796 following rules. 9797 - s_waitcnt vmcnt(0) 9798 must happen after 9799 any preceding 9800 global/generic 9801 load/load 9802 atomic/ 9803 atomicrmw-with-return-value. 9804 - s_waitcnt vscnt(0) 9805 must happen after 9806 any preceding 9807 global/generic 9808 store/store atomic/ 9809 atomicrmw-no-return-value. 9810 - s_waitcnt lgkmcnt(0) 9811 must happen after 9812 any preceding 9813 local/generic 9814 load/store/load 9815 atomic/store atomic/ 9816 atomicrmw. 9817 - Must happen before 9818 any following 9819 global/generic 9820 load/load 9821 atomic/store/store 9822 atomic/atomicrmw. 9823 - Ensures that all 9824 memory operations 9825 have 9826 completed before 9827 performing any 9828 following global 9829 memory operations. 9830 - Ensures that the 9831 preceding 9832 local/generic load 9833 atomic/atomicrmw 9834 with an equal or 9835 wider sync scope 9836 and memory ordering 9837 stronger than 9838 unordered (this is 9839 termed the 9840 acquire-fence-paired-atomic) 9841 has completed 9842 before following 9843 global memory 9844 operations. This 9845 satisfies the 9846 requirements of 9847 acquire. 9848 - Ensures that all 9849 previous memory 9850 operations have 9851 completed before a 9852 following 9853 local/generic store 9854 atomic/atomicrmw 9855 with an equal or 9856 wider sync scope 9857 and memory ordering 9858 stronger than 9859 unordered (this is 9860 termed the 9861 release-fence-paired-atomic). 9862 This satisfies the 9863 requirements of 9864 release. 9865 - Must happen before 9866 the following 9867 buffer_gl0_inv. 9868 - Ensures that the 9869 acquire-fence-paired 9870 atomic has completed 9871 before invalidating 9872 the 9873 cache. Therefore 9874 any following 9875 locations read must 9876 be no older than 9877 the value read by 9878 the 9879 acquire-fence-paired-atomic. 9880 9881 3. buffer_gl0_inv 9882 9883 - If CU wavefront execution 9884 mode, omit. 9885 - Ensures that 9886 following 9887 loads will not see 9888 stale data. 9889 9890 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 9891 - system vmcnt(0) & vscnt(0) 9892 9893 - If OpenCL and 9894 address space is 9895 not generic, omit 9896 lgkmcnt(0). 9897 - If OpenCL and 9898 address space is 9899 local, omit 9900 vmcnt(0) and vscnt(0). 9901 - However, since LLVM 9902 currently has no 9903 address space on 9904 the fence need to 9905 conservatively 9906 always generate 9907 (see comment for 9908 previous fence). 9909 - Could be split into 9910 separate s_waitcnt 9911 vmcnt(0), s_waitcnt 9912 vscnt(0) and s_waitcnt 9913 lgkmcnt(0) to allow 9914 them to be 9915 independently moved 9916 according to the 9917 following rules. 9918 - s_waitcnt vmcnt(0) 9919 must happen after 9920 any preceding 9921 global/generic 9922 load/load 9923 atomic/ 9924 atomicrmw-with-return-value. 9925 - s_waitcnt vscnt(0) 9926 must happen after 9927 any preceding 9928 global/generic 9929 store/store atomic/ 9930 atomicrmw-no-return-value. 9931 - s_waitcnt lgkmcnt(0) 9932 must happen after 9933 any preceding 9934 local/generic 9935 load/store/load 9936 atomic/store 9937 atomic/atomicrmw. 9938 - Must happen before 9939 the following 9940 buffer_gl*_inv. 9941 - Ensures that the 9942 preceding 9943 global/local/generic 9944 load 9945 atomic/atomicrmw 9946 with an equal or 9947 wider sync scope 9948 and memory ordering 9949 stronger than 9950 unordered (this is 9951 termed the 9952 acquire-fence-paired-atomic) 9953 has completed 9954 before invalidating 9955 the caches. This 9956 satisfies the 9957 requirements of 9958 acquire. 9959 - Ensures that all 9960 previous memory 9961 operations have 9962 completed before a 9963 following 9964 global/local/generic 9965 store 9966 atomic/atomicrmw 9967 with an equal or 9968 wider sync scope 9969 and memory ordering 9970 stronger than 9971 unordered (this is 9972 termed the 9973 release-fence-paired-atomic). 9974 This satisfies the 9975 requirements of 9976 release. 9977 9978 2. buffer_gl0_inv; 9979 buffer_gl1_inv 9980 9981 - Must happen before 9982 any following 9983 global/generic 9984 load/load 9985 atomic/store/store 9986 atomic/atomicrmw. 9987 - Ensures that 9988 following loads 9989 will not see stale 9990 global data. This 9991 satisfies the 9992 requirements of 9993 acquire. 9994 9995 **Sequential Consistent Atomic** 9996 ------------------------------------------------------------------------------------ 9997 load atomic seq_cst - singlethread - global *Same as corresponding 9998 - wavefront - local load atomic acquire, 9999 - generic except must generated 10000 all instructions even 10001 for OpenCL.* 10002 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10003 - generic vmcnt(0) & vscnt(0) 10004 10005 - If CU wavefront execution 10006 mode, omit vmcnt(0) and 10007 vscnt(0). 10008 - Could be split into 10009 separate s_waitcnt 10010 vmcnt(0), s_waitcnt 10011 vscnt(0), and s_waitcnt 10012 lgkmcnt(0) to allow 10013 them to be 10014 independently moved 10015 according to the 10016 following rules. 10017 - s_waitcnt lgkmcnt(0) must 10018 happen after 10019 preceding 10020 local/generic load 10021 atomic/store 10022 atomic/atomicrmw 10023 with memory 10024 ordering of seq_cst 10025 and with equal or 10026 wider sync scope. 10027 (Note that seq_cst 10028 fences have their 10029 own s_waitcnt 10030 lgkmcnt(0) and so do 10031 not need to be 10032 considered.) 10033 - s_waitcnt vmcnt(0) 10034 must happen after 10035 preceding 10036 global/generic load 10037 atomic/ 10038 atomicrmw-with-return-value 10039 with memory 10040 ordering of seq_cst 10041 and with equal or 10042 wider sync scope. 10043 (Note that seq_cst 10044 fences have their 10045 own s_waitcnt 10046 vmcnt(0) and so do 10047 not need to be 10048 considered.) 10049 - s_waitcnt vscnt(0) 10050 Must happen after 10051 preceding 10052 global/generic store 10053 atomic/ 10054 atomicrmw-no-return-value 10055 with memory 10056 ordering of seq_cst 10057 and with equal or 10058 wider sync scope. 10059 (Note that seq_cst 10060 fences have their 10061 own s_waitcnt 10062 vscnt(0) and so do 10063 not need to be 10064 considered.) 10065 - Ensures any 10066 preceding 10067 sequential 10068 consistent global/local 10069 memory instructions 10070 have completed 10071 before executing 10072 this sequentially 10073 consistent 10074 instruction. This 10075 prevents reordering 10076 a seq_cst store 10077 followed by a 10078 seq_cst load. (Note 10079 that seq_cst is 10080 stronger than 10081 acquire/release as 10082 the reordering of 10083 load acquire 10084 followed by a store 10085 release is 10086 prevented by the 10087 s_waitcnt of 10088 the release, but 10089 there is nothing 10090 preventing a store 10091 release followed by 10092 load acquire from 10093 completing out of 10094 order. The s_waitcnt 10095 could be placed after 10096 seq_store or before 10097 the seq_load. We 10098 choose the load to 10099 make the s_waitcnt be 10100 as late as possible 10101 so that the store 10102 may have already 10103 completed.) 10104 10105 2. *Following 10106 instructions same as 10107 corresponding load 10108 atomic acquire, 10109 except must generated 10110 all instructions even 10111 for OpenCL.* 10112 load atomic seq_cst - workgroup - local 10113 10114 1. s_waitcnt vmcnt(0) & vscnt(0) 10115 10116 - If CU wavefront execution 10117 mode, omit. 10118 - Could be split into 10119 separate s_waitcnt 10120 vmcnt(0) and s_waitcnt 10121 vscnt(0) to allow 10122 them to be 10123 independently moved 10124 according to the 10125 following rules. 10126 - s_waitcnt vmcnt(0) 10127 Must happen after 10128 preceding 10129 global/generic load 10130 atomic/ 10131 atomicrmw-with-return-value 10132 with memory 10133 ordering of seq_cst 10134 and with equal or 10135 wider sync scope. 10136 (Note that seq_cst 10137 fences have their 10138 own s_waitcnt 10139 vmcnt(0) and so do 10140 not need to be 10141 considered.) 10142 - s_waitcnt vscnt(0) 10143 Must happen after 10144 preceding 10145 global/generic store 10146 atomic/ 10147 atomicrmw-no-return-value 10148 with memory 10149 ordering of seq_cst 10150 and with equal or 10151 wider sync scope. 10152 (Note that seq_cst 10153 fences have their 10154 own s_waitcnt 10155 vscnt(0) and so do 10156 not need to be 10157 considered.) 10158 - Ensures any 10159 preceding 10160 sequential 10161 consistent global 10162 memory instructions 10163 have completed 10164 before executing 10165 this sequentially 10166 consistent 10167 instruction. This 10168 prevents reordering 10169 a seq_cst store 10170 followed by a 10171 seq_cst load. (Note 10172 that seq_cst is 10173 stronger than 10174 acquire/release as 10175 the reordering of 10176 load acquire 10177 followed by a store 10178 release is 10179 prevented by the 10180 s_waitcnt of 10181 the release, but 10182 there is nothing 10183 preventing a store 10184 release followed by 10185 load acquire from 10186 completing out of 10187 order. The s_waitcnt 10188 could be placed after 10189 seq_store or before 10190 the seq_load. We 10191 choose the load to 10192 make the s_waitcnt be 10193 as late as possible 10194 so that the store 10195 may have already 10196 completed.) 10197 10198 2. *Following 10199 instructions same as 10200 corresponding load 10201 atomic acquire, 10202 except must generated 10203 all instructions even 10204 for OpenCL.* 10205 10206 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10207 - system - generic vmcnt(0) & vscnt(0) 10208 10209 - Could be split into 10210 separate s_waitcnt 10211 vmcnt(0), s_waitcnt 10212 vscnt(0) and s_waitcnt 10213 lgkmcnt(0) to allow 10214 them to be 10215 independently moved 10216 according to the 10217 following rules. 10218 - s_waitcnt lgkmcnt(0) 10219 must happen after 10220 preceding 10221 local load 10222 atomic/store 10223 atomic/atomicrmw 10224 with memory 10225 ordering of seq_cst 10226 and with equal or 10227 wider sync scope. 10228 (Note that seq_cst 10229 fences have their 10230 own s_waitcnt 10231 lgkmcnt(0) and so do 10232 not need to be 10233 considered.) 10234 - s_waitcnt vmcnt(0) 10235 must happen after 10236 preceding 10237 global/generic load 10238 atomic/ 10239 atomicrmw-with-return-value 10240 with memory 10241 ordering of seq_cst 10242 and with equal or 10243 wider sync scope. 10244 (Note that seq_cst 10245 fences have their 10246 own s_waitcnt 10247 vmcnt(0) and so do 10248 not need to be 10249 considered.) 10250 - s_waitcnt vscnt(0) 10251 Must happen after 10252 preceding 10253 global/generic store 10254 atomic/ 10255 atomicrmw-no-return-value 10256 with memory 10257 ordering of seq_cst 10258 and with equal or 10259 wider sync scope. 10260 (Note that seq_cst 10261 fences have their 10262 own s_waitcnt 10263 vscnt(0) and so do 10264 not need to be 10265 considered.) 10266 - Ensures any 10267 preceding 10268 sequential 10269 consistent global 10270 memory instructions 10271 have completed 10272 before executing 10273 this sequentially 10274 consistent 10275 instruction. This 10276 prevents reordering 10277 a seq_cst store 10278 followed by a 10279 seq_cst load. (Note 10280 that seq_cst is 10281 stronger than 10282 acquire/release as 10283 the reordering of 10284 load acquire 10285 followed by a store 10286 release is 10287 prevented by the 10288 s_waitcnt of 10289 the release, but 10290 there is nothing 10291 preventing a store 10292 release followed by 10293 load acquire from 10294 completing out of 10295 order. The s_waitcnt 10296 could be placed after 10297 seq_store or before 10298 the seq_load. We 10299 choose the load to 10300 make the s_waitcnt be 10301 as late as possible 10302 so that the store 10303 may have already 10304 completed.) 10305 10306 2. *Following 10307 instructions same as 10308 corresponding load 10309 atomic acquire, 10310 except must generated 10311 all instructions even 10312 for OpenCL.* 10313 store atomic seq_cst - singlethread - global *Same as corresponding 10314 - wavefront - local store atomic release, 10315 - workgroup - generic except must generated 10316 - agent all instructions even 10317 - system for OpenCL.* 10318 atomicrmw seq_cst - singlethread - global *Same as corresponding 10319 - wavefront - local atomicrmw acq_rel, 10320 - workgroup - generic except must generated 10321 - agent all instructions even 10322 - system for OpenCL.* 10323 fence seq_cst - singlethread *none* *Same as corresponding 10324 - wavefront fence acq_rel, 10325 - workgroup except must generated 10326 - agent all instructions even 10327 - system for OpenCL.* 10328 ============ ============ ============== ========== ================================ 10329 10330Trap Handler ABI 10331~~~~~~~~~~~~~~~~ 10332 10333For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10334runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10335supports the ``s_trap`` instruction. For usage see: 10336 10337- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10338- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10339- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10340 10341 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10342 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10343 10344 =================== =============== =============== ======================================= 10345 Usage Code Sequence Trap Handler Description 10346 Inputs 10347 =================== =============== =============== ======================================= 10348 reserved ``s_trap 0x00`` Reserved by hardware. 10349 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10350 ``queue_ptr`` intrinsic (not implemented). 10351 ``VGPR0``: 10352 ``arg`` 10353 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10354 ``queue_ptr`` the trap instruction. The associated 10355 queue is signalled to put it into the 10356 error state. When the queue is put in 10357 the error state, the waves executing 10358 dispatches on the queue will be 10359 terminated. 10360 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10361 as a no-operation. The trap handler 10362 is entered and immediately returns to 10363 continue execution of the wavefront. 10364 - If the debugger is enabled, causes 10365 the debug trap to be reported by the 10366 debugger and the wavefront is put in 10367 the halt state with the PC at the 10368 instruction. The debugger must 10369 increment the PC and resume the wave. 10370 reserved ``s_trap 0x04`` Reserved. 10371 reserved ``s_trap 0x05`` Reserved. 10372 reserved ``s_trap 0x06`` Reserved. 10373 reserved ``s_trap 0x07`` Reserved. 10374 reserved ``s_trap 0x08`` Reserved. 10375 reserved ``s_trap 0xfe`` Reserved. 10376 reserved ``s_trap 0xff`` Reserved. 10377 =================== =============== =============== ======================================= 10378 10379.. 10380 10381 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10382 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10383 10384 =================== =============== =============== ======================================= 10385 Usage Code Sequence Trap Handler Description 10386 Inputs 10387 =================== =============== =============== ======================================= 10388 reserved ``s_trap 0x00`` Reserved by hardware. 10389 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10390 breakpoints. Causes wave to be halted 10391 with the PC at the trap instruction. 10392 The debugger is responsible to resume 10393 the wave, including the instruction 10394 that the breakpoint overwrote. 10395 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10396 ``queue_ptr`` the trap instruction. The associated 10397 queue is signalled to put it into the 10398 error state. When the queue is put in 10399 the error state, the waves executing 10400 dispatches on the queue will be 10401 terminated. 10402 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10403 as a no-operation. The trap handler 10404 is entered and immediately returns to 10405 continue execution of the wavefront. 10406 - If the debugger is enabled, causes 10407 the debug trap to be reported by the 10408 debugger and the wavefront is put in 10409 the halt state with the PC at the 10410 instruction. The debugger must 10411 increment the PC and resume the wave. 10412 reserved ``s_trap 0x04`` Reserved. 10413 reserved ``s_trap 0x05`` Reserved. 10414 reserved ``s_trap 0x06`` Reserved. 10415 reserved ``s_trap 0x07`` Reserved. 10416 reserved ``s_trap 0x08`` Reserved. 10417 reserved ``s_trap 0xfe`` Reserved. 10418 reserved ``s_trap 0xff`` Reserved. 10419 =================== =============== =============== ======================================= 10420 10421.. 10422 10423 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10424 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10425 10426 =================== =============== ================ ================= ======================================= 10427 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10428 =================== =============== ================ ================= ======================================= 10429 reserved ``s_trap 0x00`` Reserved by hardware. 10430 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10431 breakpoints. Causes wave to be halted 10432 with the PC at the trap instruction. 10433 The debugger is responsible to resume 10434 the wave, including the instruction 10435 that the breakpoint overwrote. 10436 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10437 ``queue_ptr`` the trap instruction. The associated 10438 queue is signalled to put it into the 10439 error state. When the queue is put in 10440 the error state, the waves executing 10441 dispatches on the queue will be 10442 terminated. 10443 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10444 as a no-operation. The trap handler 10445 is entered and immediately returns to 10446 continue execution of the wavefront. 10447 - If the debugger is enabled, causes 10448 the debug trap to be reported by the 10449 debugger and the wavefront is put in 10450 the halt state with the PC at the 10451 instruction. The debugger must 10452 increment the PC and resume the wave. 10453 reserved ``s_trap 0x04`` Reserved. 10454 reserved ``s_trap 0x05`` Reserved. 10455 reserved ``s_trap 0x06`` Reserved. 10456 reserved ``s_trap 0x07`` Reserved. 10457 reserved ``s_trap 0x08`` Reserved. 10458 reserved ``s_trap 0xfe`` Reserved. 10459 reserved ``s_trap 0xff`` Reserved. 10460 =================== =============== ================ ================= ======================================= 10461 10462.. _amdgpu-amdhsa-function-call-convention: 10463 10464Call Convention 10465~~~~~~~~~~~~~~~ 10466 10467.. note:: 10468 10469 This section is currently incomplete and has inaccuracies. It is WIP that will 10470 be updated as information is determined. 10471 10472See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10473addresses. Unswizzled addresses are normal linear addresses. 10474 10475.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10476 10477Kernel Functions 10478++++++++++++++++ 10479 10480This section describes the call convention ABI for the outer kernel function. 10481 10482See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10483convention. 10484 10485The following is not part of the AMDGPU kernel calling convention but describes 10486how the AMDGPU implements function calls: 10487 104881. Clang decides the kernarg layout to match the *HSA Programmer's Language 10489 Reference* [HSA]_. 10490 10491 - All structs are passed directly. 10492 - Lambda values are passed *TBA*. 10493 10494 .. TODO:: 10495 10496 - Does this really follow HSA rules? Or are structs >16 bytes passed 10497 by-value struct? 10498 - What is ABI for lambda values? 10499 105004. The kernel performs certain setup in its prolog, as described in 10501 :ref:`amdgpu-amdhsa-kernel-prolog`. 10502 10503.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10504 10505Non-Kernel Functions 10506++++++++++++++++++++ 10507 10508This section describes the call convention ABI for functions other than the 10509outer kernel function. 10510 10511If a kernel has function calls then scratch is always allocated and used for 10512the call stack which grows from low address to high address using the swizzled 10513scratch address space. 10514 10515On entry to a function: 10516 105171. SGPR0-3 contain a V# with the following properties (see 10518 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10519 10520 * Base address pointing to the beginning of the wavefront scratch backing 10521 memory. 10522 * Swizzled with dword element size and stride of wavefront size elements. 10523 105242. The FLAT_SCRATCH register pair is setup. See 10525 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 105263. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10527 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 105284. The EXEC register is set to the lanes active on entry to the function. 105295. MODE register: *TBD* 105306. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10531 below. 105327. SGPR30-31 return address (RA). The code address that the function must 10533 return to when it completes. The value is undefined if the function is *no 10534 return*. 105358. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10536 offset relative to the beginning of the wavefront scratch backing memory. 10537 10538 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10539 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10540 manner. 10541 10542 The unswizzled SP value can be converted into the swizzled SP value by: 10543 10544 | swizzled SP = unswizzled SP / wavefront size 10545 10546 This may be used to obtain the private address space address of stack 10547 objects and to convert this address to a flat address by adding the flat 10548 scratch aperture base address. 10549 10550 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10551 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10552 10553 .. note:: 10554 10555 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10556 OpenCL language which has the largest base type defined as 16 bytes. 10557 10558 On entry, the swizzled SP value is the address of the first function 10559 argument passed on the stack. Other stack passed arguments are positive 10560 offsets from the entry swizzled SP value. 10561 10562 The function may use positive offsets beyond the last stack passed argument 10563 for stack allocated local variables and register spill slots. If necessary, 10564 the function may align these to greater alignment than 16 bytes. After these 10565 the function may dynamically allocate space for such things as runtime sized 10566 ``alloca`` local allocations. 10567 10568 If the function calls another function, it will place any stack allocated 10569 arguments after the last local allocation and adjust SGPR32 to the address 10570 after the last local allocation. 10571 105729. All other registers are unspecified. 1057310. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10574 to the function. 10575 10576On exit from a function: 10577 105781. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10579 described below. Any registers used are considered clobbered registers. 105802. The following registers are preserved and have the same value as on entry: 10581 10582 * FLAT_SCRATCH 10583 * EXEC 10584 * GFX6-GFX8: M0 10585 * All SGPR registers except the clobbered registers of SGPR4-31. 10586 * VGPR40-47 10587 * VGPR56-63 10588 * VGPR72-79 10589 * VGPR88-95 10590 * VGPR104-111 10591 * VGPR120-127 10592 * VGPR136-143 10593 * VGPR152-159 10594 * VGPR168-175 10595 * VGPR184-191 10596 * VGPR200-207 10597 * VGPR216-223 10598 * VGPR232-239 10599 * VGPR248-255 10600 10601 .. note:: 10602 10603 Except the argument registers, the VGPRs clobbered and the preserved 10604 registers are intermixed at regular intervals in order to keep a 10605 similar ratio independent of the number of allocated VGPRs. 10606 10607 * Lanes of all VGPRs that are inactive at the call site. 10608 10609 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10610 optimization may mark some of clobbered SGPR and VGPR registers as 10611 preserved if it can be determined that the called function does not change 10612 their value. 10613 106142. The PC is set to the RA provided on entry. 106153. MODE register: *TBD*. 106164. All other registers are clobbered. 106175. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10618 function is available to the caller. 10619 10620.. TODO:: 10621 10622 - On gfx908 are all ACC registers clobbered? 10623 10624 - How are function results returned? The address of structured types is passed 10625 by reference, but what about other types? 10626 10627The function input arguments are made up of the formal arguments explicitly 10628declared by the source language function plus the implicit input arguments used 10629by the implementation. 10630 10631The source language input arguments are: 10632 106331. Any source language implicit ``this`` or ``self`` argument comes first as a 10634 pointer type. 106352. Followed by the function formal arguments in left to right source order. 10636 10637The source language result arguments are: 10638 106391. The function result argument. 10640 10641The source language input or result struct type arguments that are less than or 10642equal to 16 bytes, are decomposed recursively into their base type fields, and 10643each field is passed as if a separate argument. For input arguments, if the 10644called function requires the struct to be in memory, for example because its 10645address is taken, then the function body is responsible for allocating a stack 10646location and copying the field arguments into it. Clang terms this *direct 10647struct*. 10648 10649The source language input struct type arguments that are greater than 16 bytes, 10650are passed by reference. The caller is responsible for allocating a stack 10651location to make a copy of the struct value and pass the address as the input 10652argument. The called function is responsible to perform the dereference when 10653accessing the input argument. Clang terms this *by-value struct*. 10654 10655A source language result struct type argument that is greater than 16 bytes, is 10656returned by reference. The caller is responsible for allocating a stack location 10657to hold the result value and passes the address as the last input argument 10658(before the implicit input arguments). In this case there are no result 10659arguments. The called function is responsible to perform the dereference when 10660storing the result value. Clang terms this *structured return (sret)*. 10661 10662*TODO: correct the ``sret`` definition.* 10663 10664.. TODO:: 10665 10666 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10667 pass as non-decomposed struct as stack argument? Or something else? Is the 10668 memory location in the caller stack frame, or a stack memory argument and so 10669 no address is passed as the caller can directly write to the argument stack 10670 location? But then the stack location is still live after return. If an 10671 argument stack location is it the first stack argument or the last one? 10672 10673Lambda argument types are treated as struct types with an implementation defined 10674set of fields. 10675 10676.. TODO:: 10677 10678 Need to specify the ABI for lambda types for AMDGPU. 10679 10680For AMDGPU backend all source language arguments (including the decomposed 10681struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10682they are passed in SGPRs. 10683 10684The AMDGPU backend walks the function call graph from the leaves to determine 10685which implicit input arguments are used, propagating to each caller of the 10686function. The used implicit arguments are appended to the function arguments 10687after the source language arguments in the following order: 10688 10689.. TODO:: 10690 10691 Is recursion or external functions supported? 10692 106931. Work-Item ID (1 VGPR) 10694 10695 The X, Y and Z work-item ID are packed into a single VGRP with the following 10696 layout. Only fields actually used by the function are set. The other bits 10697 are undefined. 10698 10699 The values come from the initial kernel execution state. See 10700 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10701 10702 .. table:: Work-item implicit argument layout 10703 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10704 10705 ======= ======= ============== 10706 Bits Size Field Name 10707 ======= ======= ============== 10708 9:0 10 bits X Work-Item ID 10709 19:10 10 bits Y Work-Item ID 10710 29:20 10 bits Z Work-Item ID 10711 31:30 2 bits Unused 10712 ======= ======= ============== 10713 107142. Dispatch Ptr (2 SGPRs) 10715 10716 The value comes from the initial kernel execution state. See 10717 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10718 107193. Queue Ptr (2 SGPRs) 10720 10721 The value comes from the initial kernel execution state. See 10722 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10723 107244. Kernarg Segment Ptr (2 SGPRs) 10725 10726 The value comes from the initial kernel execution state. See 10727 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10728 107295. Dispatch id (2 SGPRs) 10730 10731 The value comes from the initial kernel execution state. See 10732 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10733 107346. Work-Group ID X (1 SGPR) 10735 10736 The value comes from the initial kernel execution state. See 10737 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10738 107397. Work-Group ID Y (1 SGPR) 10740 10741 The value comes from the initial kernel execution state. See 10742 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10743 107448. Work-Group ID Z (1 SGPR) 10745 10746 The value comes from the initial kernel execution state. See 10747 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10748 107499. Implicit Argument Ptr (2 SGPRs) 10750 10751 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10752 global address space pointer to the first kernarg implicit argument. 10753 10754The input and result arguments are assigned in order in the following manner: 10755 10756.. note:: 10757 10758 There are likely some errors and omissions in the following description that 10759 need correction. 10760 10761 .. TODO:: 10762 10763 Check the Clang source code to decipher how function arguments and return 10764 results are handled. Also see the AMDGPU specific values used. 10765 10766* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 10767 VGPR31. 10768 10769 If there are more arguments than will fit in these registers, the remaining 10770 arguments are allocated on the stack in order on naturally aligned 10771 addresses. 10772 10773 .. TODO:: 10774 10775 How are overly aligned structures allocated on the stack? 10776 10777* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 10778 SGPR29. 10779 10780 If there are more arguments than will fit in these registers, the remaining 10781 arguments are allocated on the stack in order on naturally aligned 10782 addresses. 10783 10784Note that decomposed struct type arguments may have some fields passed in 10785registers and some in memory. 10786 10787.. TODO:: 10788 10789 So, a struct which can pass some fields as decomposed register arguments, will 10790 pass the rest as decomposed stack elements? But an argument that will not start 10791 in registers will not be decomposed and will be passed as a non-decomposed 10792 stack value? 10793 10794The following is not part of the AMDGPU function calling convention but 10795describes how the AMDGPU implements function calls: 10796 107971. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 10798 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 10799 are used, or for the reasons defined in ``SIFrameLowering``. 108002. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 10801 to access the incoming stack arguments in the function. The BP is needed 10802 only when the function requires the runtime stack alignment. 10803 108043. Allocating SGPR arguments on the stack are not supported. 10805 108064. No CFI is currently generated. See 10807 :ref:`amdgpu-dwarf-call-frame-information`. 10808 10809 .. note:: 10810 10811 CFI will be generated that defines the CFA as the unswizzled address 10812 relative to the wave scratch base in the unswizzled private address space 10813 of the lowest address stack allocated local variable. 10814 10815 ``DW_AT_frame_base`` will be defined as the swizzled address in the 10816 swizzled private address space by dividing the CFA by the wavefront size 10817 (since CFA is always at least dword aligned which matches the scratch 10818 swizzle element size). 10819 10820 If no dynamic stack alignment was performed, the stack allocated arguments 10821 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 10822 local variables and register spill slots are accessed as positive offsets 10823 relative to ``DW_AT_frame_base``. 10824 108255. Function argument passing is implemented by copying the input physical 10826 registers to virtual registers on entry. The register allocator can spill if 10827 necessary. These are copied back to physical registers at call sites. The 10828 net effect is that each function call can have these values in entirely 10829 distinct locations. The IPRA can help avoid shuffling argument registers. 108306. Call sites are implemented by setting up the arguments at positive offsets 10831 from SP. Then SP is incremented to account for the known frame size before 10832 the call and decremented after the call. 10833 10834 .. note:: 10835 10836 The CFI will reflect the changed calculation needed to compute the CFA 10837 from SP. 10838 108397. 4 byte spill slots are used in the stack frame. One slot is allocated for an 10840 emergency spill slot. Buffer instructions are used for stack accesses and 10841 not the ``flat_scratch`` instruction. 10842 10843 .. TODO:: 10844 10845 Explain when the emergency spill slot is used. 10846 10847.. TODO:: 10848 10849 Possible broken issues: 10850 10851 - Stack arguments must be aligned to required alignment. 10852 - Stack is aligned to max(16, max formal argument alignment) 10853 - Direct argument < 64 bits should check register budget. 10854 - Register budget calculation should respect ``inreg`` for SGPR. 10855 - SGPR overflow is not handled. 10856 - struct with 1 member unpeeling is not checking size of member. 10857 - ``sret`` is after ``this`` pointer. 10858 - Caller is not implementing stack realignment: need an extra pointer. 10859 - Should say AMDGPU passes FP rather than SP. 10860 - Should CFI define CFA as address of locals or arguments. Difference is 10861 apparent when have implemented dynamic alignment. 10862 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 10863 highest address of stack frame and use negative offset for locals. Would 10864 allow SP to be the same as FP and could support signal-handler-like as now 10865 have a real SP for the top of the stack. 10866 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 10867 arguments? 10868 10869AMDPAL 10870------ 10871 10872This section provides code conventions used when the target triple OS is 10873``amdpal`` (see :ref:`amdgpu-target-triples`). 10874 10875.. _amdgpu-amdpal-code-object-metadata-section: 10876 10877Code Object Metadata 10878~~~~~~~~~~~~~~~~~~~~ 10879 10880.. note:: 10881 10882 The metadata is currently in development and is subject to major 10883 changes. Only the current version is supported. *When this document 10884 was generated the version was 2.6.* 10885 10886Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 10887record (see :ref:`amdgpu-note-records-v3-v4`). 10888 10889The metadata is represented as Message Pack formatted binary data (see 10890[MsgPack]_). The top level is a Message Pack map that includes the keys 10891defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 10892and referenced tables. 10893 10894Additional information can be added to the maps. To avoid conflicts, any 10895key names should be prefixed by "*vendor-name*." where ``vendor-name`` 10896can be the name of the vendor and specific vendor tool that generates the 10897information. The prefix is abbreviated to simply "." when it appears 10898within a map that has been added by the same *vendor-name*. 10899 10900 .. table:: AMDPAL Code Object Metadata Map 10901 :name: amdgpu-amdpal-code-object-metadata-map-table 10902 10903 =================== ============== ========= ====================================================================== 10904 String Key Value Type Required? Description 10905 =================== ============== ========= ====================================================================== 10906 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 10907 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 10908 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 10909 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 10910 definition of the keys included in that map. 10911 =================== ============== ========= ====================================================================== 10912 10913.. 10914 10915 .. table:: AMDPAL Code Object Pipeline Metadata Map 10916 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 10917 10918 ====================================== ============== ========= =================================================== 10919 String Key Value Type Required? Description 10920 ====================================== ============== ========= =================================================== 10921 ".name" string Source name of the pipeline. 10922 ".type" string Pipeline type, e.g. VsPs. Values include: 10923 10924 - "VsPs" 10925 - "Gs" 10926 - "Cs" 10927 - "Ngg" 10928 - "Tess" 10929 - "GsTess" 10930 - "NggTess" 10931 10932 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 10933 2 integers 64 bits is the "stable" portion of the hash, used 10934 for e.g. shader replacement lookup. Upper 64 bits 10935 is the "unique" portion of the hash, used for 10936 e.g. pipeline cache lookup. The value is 10937 implementation defined, and can not be relied on 10938 between different builds of the compiler. 10939 ".shaders" map Per-API shader metadata. See 10940 :ref:`amdgpu-amdpal-code-object-shader-map-table` 10941 for the definition of the keys included in that 10942 map. 10943 ".hardware_stages" map Per-hardware stage metadata. See 10944 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 10945 for the definition of the keys included in that 10946 map. 10947 ".shader_functions" map Per-shader function metadata. See 10948 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 10949 for the definition of the keys included in that 10950 map. 10951 ".registers" map Required Hardware register configuration. See 10952 :ref:`amdgpu-amdpal-code-object-register-map-table` 10953 for the definition of the keys included in that 10954 map. 10955 ".user_data_limit" integer Number of user data entries accessed by this 10956 pipeline. 10957 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 10958 NoUserDataSpilling. 10959 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 10960 viewport array index feature. Pipelines which use 10961 this feature can render into all 16 viewports, 10962 whereas pipelines which do not use it are 10963 restricted to viewport #0. 10964 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 10965 handling data-passing between the ES and GS 10966 shader stages. This can be zero if the data is 10967 passed using off-chip buffers. This value should 10968 be used to program all user-SGPRs which have been 10969 marked with "UserDataMapping::EsGsLdsSize" 10970 (typically only the GS and VS HW stages will ever 10971 have a user-SGPR so marked). 10972 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 10973 (maximum number of threads in a subgroup). 10974 ".num_interpolants" integer Graphics only. Number of PS interpolants. 10975 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 10976 ".api" string Name of the client graphics API. 10977 ".api_create_info" binary Graphics API shader create info binary blob. Can 10978 be defined by the driver using the compiler if 10979 they want to be able to correlate API-specific 10980 information used during creation at a later time. 10981 ====================================== ============== ========= =================================================== 10982 10983.. 10984 10985 .. table:: AMDPAL Code Object Shader Map 10986 :name: amdgpu-amdpal-code-object-shader-map-table 10987 10988 10989 +-------------+--------------+-------------------------------------------------------------------+ 10990 |String Key |Value Type |Description | 10991 +=============+==============+===================================================================+ 10992 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 10993 |- ".vertex" | |for the definition of the keys included in that map. | 10994 |- ".hull" | | | 10995 |- ".domain" | | | 10996 |- ".geometry"| | | 10997 |- ".pixel" | | | 10998 +-------------+--------------+-------------------------------------------------------------------+ 10999 11000.. 11001 11002 .. table:: AMDPAL Code Object API Shader Metadata Map 11003 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 11004 11005 ==================== ============== ========= ===================================================================== 11006 String Key Value Type Required? Description 11007 ==================== ============== ========= ===================================================================== 11008 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 11009 2 integers is implementation defined, and can not be relied on between 11010 different builds of the compiler. 11011 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 11012 string include: 11013 11014 - ".ls" 11015 - ".hs" 11016 - ".es" 11017 - ".gs" 11018 - ".vs" 11019 - ".ps" 11020 - ".cs" 11021 11022 ==================== ============== ========= ===================================================================== 11023 11024.. 11025 11026 .. table:: AMDPAL Code Object Hardware Stage Map 11027 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 11028 11029 +-------------+--------------+-----------------------------------------------------------------------+ 11030 |String Key |Value Type |Description | 11031 +=============+==============+=======================================================================+ 11032 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 11033 |- ".hs" | |for the definition of the keys included in that map. | 11034 |- ".es" | | | 11035 |- ".gs" | | | 11036 |- ".vs" | | | 11037 |- ".ps" | | | 11038 |- ".cs" | | | 11039 +-------------+--------------+-----------------------------------------------------------------------+ 11040 11041.. 11042 11043 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 11044 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 11045 11046 ========================== ============== ========= =============================================================== 11047 String Key Value Type Required? Description 11048 ========================== ============== ========= =============================================================== 11049 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 11050 ".scratch_memory_size" integer Scratch memory size in bytes. 11051 ".lds_size" integer Local Data Share size in bytes. 11052 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 11053 ".vgpr_count" integer Number of VGPRs used. 11054 ".sgpr_count" integer Number of SGPRs used. 11055 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 11056 directive to instruct the compiler to limit the VGPR usage to 11057 be less than or equal to the specified value (only set if 11058 different from HW default). 11059 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 11060 default). 11061 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 11062 3 integers 11063 ".wavefront_size" integer Wavefront size (only set if different from HW default). 11064 ".uses_uavs" boolean The shader reads or writes UAVs. 11065 ".uses_rovs" boolean The shader reads or writes ROVs. 11066 ".writes_uavs" boolean The shader writes to one or more UAVs. 11067 ".writes_depth" boolean The shader writes out a depth value. 11068 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 11069 memory or GDS. 11070 ".uses_prim_id" boolean The shader uses PrimID. 11071 ========================== ============== ========= =============================================================== 11072 11073.. 11074 11075 .. table:: AMDPAL Code Object Shader Function Map 11076 :name: amdgpu-amdpal-code-object-shader-function-map-table 11077 11078 =============== ============== ==================================================================== 11079 String Key Value Type Description 11080 =============== ============== ==================================================================== 11081 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 11082 entry address. The value is the function's metadata. See 11083 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 11084 =============== ============== ==================================================================== 11085 11086.. 11087 11088 .. table:: AMDPAL Code Object Shader Function Metadata Map 11089 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 11090 11091 ============================= ============== ================================================================= 11092 String Key Value Type Description 11093 ============================= ============== ================================================================= 11094 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 11095 2 integers is implementation defined, and can not be relied on between 11096 different builds of the compiler. 11097 ".scratch_memory_size" sequence of Size in bytes of scratch memory used by the shader. 11098 2 integers 11099 ".lds_size" sequence of Size in bytes of LDS memory. 11100 2 integers 11101 ".vgpr_count" integer Number of VGPRs used by the shader. 11102 ".sgpr_count" integer Number of SGPRs used by the shader. 11103 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 11104 ".shader_subtype" string Shader subtype/kind. Values include: 11105 11106 - "Unknown" 11107 11108 ============================= ============== ================================================================= 11109 11110.. 11111 11112 .. table:: AMDPAL Code Object Register Map 11113 :name: amdgpu-amdpal-code-object-register-map-table 11114 11115 ========================== ============== ==================================================================== 11116 32-bit Integer Key Value Type Description 11117 ========================== ============== ==================================================================== 11118 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 11119 a GRBM register (i.e., driver accessible GPU register number, not 11120 shader GPR register number). The driver is required to program each 11121 specified register to the corresponding specified value when 11122 executing this pipeline. Typically, the ``reg offsets`` are the 11123 ``uint16_t`` offsets to each register as defined by the hardware 11124 chip headers. The register is set to the provided value. However, a 11125 ``reg offset`` that specifies a user data register (e.g., 11126 COMPUTE_USER_DATA_0) needs special treatment. See 11127 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 11128 information. 11129 ========================== ============== ==================================================================== 11130 11131.. _amdgpu-amdpal-code-object-user-data-section: 11132 11133User Data 11134+++++++++ 11135 11136Each hardware stage has a set of 32-bit physical SPI *user data registers* 11137(either 16 or 32 based on graphics IP and the stage) which can be 11138written from a command buffer and then loaded into SGPRs when waves are 11139launched via a subsequent dispatch or draw operation. This is the way 11140most arguments are passed from the application/runtime to a hardware 11141shader. 11142 11143PAL abstracts this functionality by exposing a set of 128 *user data 11144entries* per pipeline a client can use to pass arguments from a command 11145buffer to one or more shaders in that pipeline. The ELF code object must 11146specify a mapping from virtualized *user data entries* to physical *user 11147data registers*, and PAL is responsible for implementing that mapping, 11148including spilling overflow *user data entries* to memory if needed. 11149 11150Since the *user data registers* are GRBM-accessible SPI registers, this 11151mapping is actually embedded in the ``.registers`` metadata entry. For 11152most registers, the value in that map is a literal 32-bit value that 11153should be written to the register by the driver. However, when the 11154register is a *user data register* (any USER_DATA register e.g., 11155SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 11156the driver to write either a *user data entry* value or one of several 11157driver-internal values to the register. This encoding is described in 11158the following table: 11159 11160.. note:: 11161 11162 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 11163 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 11164 always be programmed to the address of the GlobalTable, and *user data 11165 register* 1 must always be programmed to the address of the PerShaderTable. 11166 11167.. 11168 11169 .. table:: AMDPAL User Data Mapping 11170 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 11171 11172 ========== ================= =============================================================================== 11173 Value Name Description 11174 ========== ================= =============================================================================== 11175 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 11176 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 11177 always point to *user data register* 0). 11178 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 11179 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 11180 for more detail (should always point to *user data register* 1). 11181 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 11182 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 11183 more detail. 11184 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 11185 reference the draw index in the vertex shader. Only supported by the first 11186 stage in a graphics pipeline. 11187 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 11188 a graphics pipeline. 11189 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 11190 graphics pipeline. 11191 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 11192 a buffer containing the grid dimensions for a Compute dispatch operation. The 11193 high half of the address is stored in the next sequential user-SGPR. Only 11194 supported by compute pipelines. 11195 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 11196 space used for the ES/GS pseudo-ring-buffer for passing data between shader 11197 stages. 11198 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 11199 pipeline instancing. 11200 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 11201 can only appear for one shader stage per pipeline. 11202 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 11203 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 11204 only appear for one shader stage per pipeline. 11205 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 11206 only appear for one shader stage per pipeline (PS). These replace color targets 11207 and are completely separate from any UAVs used by the shader. This is optional, 11208 and only used by the PS when UAV exports are used to replace color-target 11209 exports to optimize specific shaders. 11210 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 11211 some NGG pipelines to perform culling. This value contains the address of the 11212 first of two consecutive registers which provide the full GPU address. 11213 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 11214 ========== ================= =============================================================================== 11215 11216.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 11217 11218Per-Shader Table 11219################ 11220 11221Low 32 bits of the GPU address for an optional buffer in the ``.data`` 11222section of the ELF. The high 32 bits of the address match the high 32 bits 11223of the shader's program counter. 11224 11225The buffer can be anything the shader compiler needs it for, and 11226allows each shader to have its own region of the ``.data`` section. 11227Typically, this could be a table of buffer SRD's and the data pointed to 11228by the buffer SRD's, but it could be a flat-address region of memory as 11229well. Its layout and usage are defined by the shader compiler. 11230 11231Each shader's table in the ``.data`` section is referenced by the symbol 11232``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 11233hardware shader stage the data is for. E.g., 11234``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 11235 11236.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 11237 11238Spill Table 11239########### 11240 11241It is possible for a hardware shader to need access to more *user data 11242entries* than there are slots available in user data registers for one 11243or more hardware shader stages. In that case, the PAL runtime expects 11244the necessary *user data entries* to be spilled to GPU memory and use 11245one user data register to point to the spilled user data memory. The 11246value of the *user data entry* must then represent the location where 11247a shader expects to read the low 32-bits of the table's GPU virtual 11248address. The *spill table* itself represents a set of 32-bit values 11249managed by the PAL runtime in GPU-accessible memory that can be made 11250indirectly accessible to a hardware shader. 11251 11252Unspecified OS 11253-------------- 11254 11255This section provides code conventions used when the target triple OS is 11256empty (see :ref:`amdgpu-target-triples`). 11257 11258Trap Handler ABI 11259~~~~~~~~~~~~~~~~ 11260 11261For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11262not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11263instructions are handled as follows: 11264 11265 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11266 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11267 11268 =============== =============== =========================================== 11269 Usage Code Sequence Description 11270 =============== =============== =========================================== 11271 llvm.trap s_endpgm Causes wavefront to be terminated. 11272 llvm.debugtrap *none* Compiler warning given that there is no 11273 trap handler installed. 11274 =============== =============== =========================================== 11275 11276Source Languages 11277================ 11278 11279.. _amdgpu-opencl: 11280 11281OpenCL 11282------ 11283 11284When the language is OpenCL the following differences occur: 11285 112861. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 112872. The AMDGPU backend appends additional arguments to the kernel's explicit 11288 arguments for the AMDHSA OS (see 11289 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 112903. Additional metadata is generated 11291 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11292 11293 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11294 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11295 11296 ======== ==== ========= =========================================== 11297 Position Byte Byte Description 11298 Size Alignment 11299 ======== ==== ========= =========================================== 11300 1 8 8 OpenCL Global Offset X 11301 2 8 8 OpenCL Global Offset Y 11302 3 8 8 OpenCL Global Offset Z 11303 4 8 8 OpenCL address of printf buffer 11304 5 8 8 OpenCL address of virtual queue used by 11305 enqueue_kernel. 11306 6 8 8 OpenCL address of AqlWrap struct used by 11307 enqueue_kernel. 11308 7 8 8 Pointer argument used for Multi-gird 11309 synchronization. 11310 ======== ==== ========= =========================================== 11311 11312.. _amdgpu-hcc: 11313 11314HCC 11315--- 11316 11317When the language is HCC the following differences occur: 11318 113191. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11320 11321.. _amdgpu-assembler: 11322 11323Assembler 11324--------- 11325 11326AMDGPU backend has LLVM-MC based assembler which is currently in development. 11327It supports AMDGCN GFX6-GFX10. 11328 11329This section describes general syntax for instructions and operands. 11330 11331Instructions 11332~~~~~~~~~~~~ 11333 11334An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11335 11336 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11337 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11338 11339:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11340:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11341 11342The order of operands and modifiers is fixed. 11343Most modifiers are optional and may be omitted. 11344 11345Links to detailed instruction syntax description may be found in the following 11346table. Note that features under development are not included 11347in this description. 11348 11349 =================================== ======================================= 11350 Core ISA ISA Extensions 11351 =================================== ======================================= 11352 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11353 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11354 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11355 11356 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11357 11358 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11359 11360 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11361 11362 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11363 11364 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11365 11366 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11367 11368 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11369 =================================== ======================================= 11370 11371For more information about instructions, their semantics and supported 11372combinations of operands, refer to one of instruction set architecture manuals 11373[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_, 11374[AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11375 11376Operands 11377~~~~~~~~ 11378 11379Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11380 11381Modifiers 11382~~~~~~~~~ 11383 11384Detailed description of modifiers may be found 11385:doc:`here<AMDGPUModifierSyntax>`. 11386 11387Instruction Examples 11388~~~~~~~~~~~~~~~~~~~~ 11389 11390DS 11391++ 11392 11393.. code-block:: nasm 11394 11395 ds_add_u32 v2, v4 offset:16 11396 ds_write_src2_b64 v2 offset0:4 offset1:8 11397 ds_cmpst_f32 v2, v4, v6 11398 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11399 11400For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11401Manual. 11402 11403FLAT 11404++++ 11405 11406.. code-block:: nasm 11407 11408 flat_load_dword v1, v[3:4] 11409 flat_store_dwordx3 v[3:4], v[5:7] 11410 flat_atomic_swap v1, v[3:4], v5 glc 11411 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11412 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11413 11414For full list of supported instructions, refer to "FLAT instructions" in ISA 11415Manual. 11416 11417MUBUF 11418+++++ 11419 11420.. code-block:: nasm 11421 11422 buffer_load_dword v1, off, s[4:7], s1 11423 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11424 buffer_store_format_xy v[1:2], off, s[4:7], s1 11425 buffer_wbinvl1 11426 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11427 11428For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11429Manual. 11430 11431SMRD/SMEM 11432+++++++++ 11433 11434.. code-block:: nasm 11435 11436 s_load_dword s1, s[2:3], 0xfc 11437 s_load_dwordx8 s[8:15], s[2:3], s4 11438 s_load_dwordx16 s[88:103], s[2:3], s4 11439 s_dcache_inv_vol 11440 s_memtime s[4:5] 11441 11442For full list of supported instructions, refer to "Scalar Memory Operations" in 11443ISA Manual. 11444 11445SOP1 11446++++ 11447 11448.. code-block:: nasm 11449 11450 s_mov_b32 s1, s2 11451 s_mov_b64 s[0:1], 0x80000000 11452 s_cmov_b32 s1, 200 11453 s_wqm_b64 s[2:3], s[4:5] 11454 s_bcnt0_i32_b64 s1, s[2:3] 11455 s_swappc_b64 s[2:3], s[4:5] 11456 s_cbranch_join s[4:5] 11457 11458For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11459Manual. 11460 11461SOP2 11462++++ 11463 11464.. code-block:: nasm 11465 11466 s_add_u32 s1, s2, s3 11467 s_and_b64 s[2:3], s[4:5], s[6:7] 11468 s_cselect_b32 s1, s2, s3 11469 s_andn2_b32 s2, s4, s6 11470 s_lshr_b64 s[2:3], s[4:5], s6 11471 s_ashr_i32 s2, s4, s6 11472 s_bfm_b64 s[2:3], s4, s6 11473 s_bfe_i64 s[2:3], s[4:5], s6 11474 s_cbranch_g_fork s[4:5], s[6:7] 11475 11476For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11477Manual. 11478 11479SOPC 11480++++ 11481 11482.. code-block:: nasm 11483 11484 s_cmp_eq_i32 s1, s2 11485 s_bitcmp1_b32 s1, s2 11486 s_bitcmp0_b64 s[2:3], s4 11487 s_setvskip s3, s5 11488 11489For full list of supported instructions, refer to "SOPC Instructions" in ISA 11490Manual. 11491 11492SOPP 11493++++ 11494 11495.. code-block:: nasm 11496 11497 s_barrier 11498 s_nop 2 11499 s_endpgm 11500 s_waitcnt 0 ; Wait for all counters to be 0 11501 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11502 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11503 s_sethalt 9 11504 s_sleep 10 11505 s_sendmsg 0x1 11506 s_sendmsg sendmsg(MSG_INTERRUPT) 11507 s_trap 1 11508 11509For full list of supported instructions, refer to "SOPP Instructions" in ISA 11510Manual. 11511 11512Unless otherwise mentioned, little verification is performed on the operands 11513of SOPP Instructions, so it is up to the programmer to be familiar with the 11514range or acceptable values. 11515 11516VALU 11517++++ 11518 11519For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11520the assembler will automatically use optimal encoding based on its operands. To 11521force specific encoding, one can add a suffix to the opcode of the instruction: 11522 11523* _e32 for 32-bit VOP1/VOP2/VOPC 11524* _e64 for 64-bit VOP3 11525* _dpp for VOP_DPP 11526* _sdwa for VOP_SDWA 11527 11528VOP1/VOP2/VOP3/VOPC examples: 11529 11530.. code-block:: nasm 11531 11532 v_mov_b32 v1, v2 11533 v_mov_b32_e32 v1, v2 11534 v_nop 11535 v_cvt_f64_i32_e32 v[1:2], v2 11536 v_floor_f32_e32 v1, v2 11537 v_bfrev_b32_e32 v1, v2 11538 v_add_f32_e32 v1, v2, v3 11539 v_mul_i32_i24_e64 v1, v2, 3 11540 v_mul_i32_i24_e32 v1, -3, v3 11541 v_mul_i32_i24_e32 v1, -100, v3 11542 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11543 v_max_f16_e32 v1, v2, v3 11544 11545VOP_DPP examples: 11546 11547.. code-block:: nasm 11548 11549 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11550 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11551 v_mov_b32 v0, v0 wave_shl:1 11552 v_mov_b32 v0, v0 row_mirror 11553 v_mov_b32 v0, v0 row_bcast:31 11554 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11555 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11556 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11557 11558VOP_SDWA examples: 11559 11560.. code-block:: nasm 11561 11562 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11563 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11564 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11565 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11566 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11567 11568For full list of supported instructions, refer to "Vector ALU instructions". 11569 11570.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11571 11572Code Object V2 Predefined Symbols 11573~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11574 11575.. warning:: 11576 Code object V2 is not the default code object version emitted by 11577 this version of LLVM. 11578 11579The AMDGPU assembler defines and updates some symbols automatically. These 11580symbols do not affect code generation. 11581 11582.option.machine_version_major 11583+++++++++++++++++++++++++++++ 11584 11585Set to the GFX major generation number of the target being assembled for. For 11586example, when assembling for a "GFX9" target this will be set to the integer 11587value "9". The possible GFX major generation numbers are presented in 11588:ref:`amdgpu-processors`. 11589 11590.option.machine_version_minor 11591+++++++++++++++++++++++++++++ 11592 11593Set to the GFX minor generation number of the target being assembled for. For 11594example, when assembling for a "GFX810" target this will be set to the integer 11595value "1". The possible GFX minor generation numbers are presented in 11596:ref:`amdgpu-processors`. 11597 11598.option.machine_version_stepping 11599++++++++++++++++++++++++++++++++ 11600 11601Set to the GFX stepping generation number of the target being assembled for. 11602For example, when assembling for a "GFX704" target this will be set to the 11603integer value "4". The possible GFX stepping generation numbers are presented 11604in :ref:`amdgpu-processors`. 11605 11606.kernel.vgpr_count 11607++++++++++++++++++ 11608 11609Set to zero each time a 11610:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11611encountered. At each instruction, if the current value of this symbol is less 11612than or equal to the maximum VGPR number explicitly referenced within that 11613instruction then the symbol value is updated to equal that VGPR number plus 11614one. 11615 11616.kernel.sgpr_count 11617++++++++++++++++++ 11618 11619Set to zero each time a 11620:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11621encountered. At each instruction, if the current value of this symbol is less 11622than or equal to the maximum VGPR number explicitly referenced within that 11623instruction then the symbol value is updated to equal that SGPR number plus 11624one. 11625 11626.. _amdgpu-amdhsa-assembler-directives-v2: 11627 11628Code Object V2 Directives 11629~~~~~~~~~~~~~~~~~~~~~~~~~ 11630 11631.. warning:: 11632 Code object V2 is not the default code object version emitted by 11633 this version of LLVM. 11634 11635AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11636one can specify them with assembler directives. 11637 11638.hsa_code_object_version major, minor 11639+++++++++++++++++++++++++++++++++++++ 11640 11641*major* and *minor* are integers that specify the version of the HSA code 11642object that will be generated by the assembler. 11643 11644.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11645+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11646 11647 11648*major*, *minor*, and *stepping* are all integers that describe the instruction 11649set architecture (ISA) version of the assembly program. 11650 11651*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11652"AMD" and *arch* should always be equal to "AMDGPU". 11653 11654By default, the assembler will derive the ISA version, *vendor*, and *arch* 11655from the value of the -mcpu option that is passed to the assembler. 11656 11657.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11658 11659.amdgpu_hsa_kernel (name) 11660+++++++++++++++++++++++++ 11661 11662This directives specifies that the symbol with given name is a kernel entry 11663point (label) and the object should contain corresponding symbol of type 11664STT_AMDGPU_HSA_KERNEL. 11665 11666.amd_kernel_code_t 11667++++++++++++++++++ 11668 11669This directive marks the beginning of a list of key / value pairs that are used 11670to specify the amd_kernel_code_t object that will be emitted by the assembler. 11671The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11672amd_kernel_code_t values that are unspecified a default value will be used. The 11673default value for all keys is 0, with the following exceptions: 11674 11675- *amd_code_version_major* defaults to 1. 11676- *amd_kernel_code_version_minor* defaults to 2. 11677- *amd_machine_kind* defaults to 1. 11678- *amd_machine_version_major*, *machine_version_minor*, and 11679 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11680 that is passed to the assembler. 11681- *kernel_code_entry_byte_offset* defaults to 256. 11682- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11683 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11684 Note that wavefront size is specified as a power of two, so a value of **n** 11685 means a size of 2^ **n**. 11686- *call_convention* defaults to -1. 11687- *kernarg_segment_alignment*, *group_segment_alignment*, and 11688 *private_segment_alignment* default to 4. Note that alignments are specified 11689 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11690- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11691 GFX90A onwards. 11692- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11693 GFX10 onwards. 11694- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11695 11696The *.amd_kernel_code_t* directive must be placed immediately after the 11697function label and before any instructions. 11698 11699For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11700comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11701 11702.. _amdgpu-amdhsa-assembler-example-v2: 11703 11704Code Object V2 Example Source Code 11705~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11706 11707.. warning:: 11708 Code Object V2 is not the default code object version emitted by 11709 this version of LLVM. 11710 11711Here is an example of a minimal assembly source file, defining one HSA kernel: 11712 11713.. code:: 11714 :number-lines: 11715 11716 .hsa_code_object_version 1,0 11717 .hsa_code_object_isa 11718 11719 .hsatext 11720 .globl hello_world 11721 .p2align 8 11722 .amdgpu_hsa_kernel hello_world 11723 11724 hello_world: 11725 11726 .amd_kernel_code_t 11727 enable_sgpr_kernarg_segment_ptr = 1 11728 is_ptr64 = 1 11729 compute_pgm_rsrc1_vgprs = 0 11730 compute_pgm_rsrc1_sgprs = 0 11731 compute_pgm_rsrc2_user_sgpr = 2 11732 compute_pgm_rsrc1_wgp_mode = 0 11733 compute_pgm_rsrc1_mem_ordered = 0 11734 compute_pgm_rsrc1_fwd_progress = 1 11735 .end_amd_kernel_code_t 11736 11737 s_load_dwordx2 s[0:1], s[0:1] 0x0 11738 v_mov_b32 v0, 3.14159 11739 s_waitcnt lgkmcnt(0) 11740 v_mov_b32 v1, s0 11741 v_mov_b32 v2, s1 11742 flat_store_dword v[1:2], v0 11743 s_endpgm 11744 .Lfunc_end0: 11745 .size hello_world, .Lfunc_end0-hello_world 11746 11747.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11748 11749Code Object V3 to V4 Predefined Symbols 11750~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11751 11752The AMDGPU assembler defines and updates some symbols automatically. These 11753symbols do not affect code generation. 11754 11755.amdgcn.gfx_generation_number 11756+++++++++++++++++++++++++++++ 11757 11758Set to the GFX major generation number of the target being assembled for. For 11759example, when assembling for a "GFX9" target this will be set to the integer 11760value "9". The possible GFX major generation numbers are presented in 11761:ref:`amdgpu-processors`. 11762 11763.amdgcn.gfx_generation_minor 11764++++++++++++++++++++++++++++ 11765 11766Set to the GFX minor generation number of the target being assembled for. For 11767example, when assembling for a "GFX810" target this will be set to the integer 11768value "1". The possible GFX minor generation numbers are presented in 11769:ref:`amdgpu-processors`. 11770 11771.amdgcn.gfx_generation_stepping 11772+++++++++++++++++++++++++++++++ 11773 11774Set to the GFX stepping generation number of the target being assembled for. 11775For example, when assembling for a "GFX704" target this will be set to the 11776integer value "4". The possible GFX stepping generation numbers are presented 11777in :ref:`amdgpu-processors`. 11778 11779.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 11780 11781.amdgcn.next_free_vgpr 11782++++++++++++++++++++++ 11783 11784Set to zero before assembly begins. At each instruction, if the current value 11785of this symbol is less than or equal to the maximum VGPR number explicitly 11786referenced within that instruction then the symbol value is updated to equal 11787that VGPR number plus one. 11788 11789May be used to set the `.amdhsa_next_free_vgpr` directive in 11790:ref:`amdhsa-kernel-directives-table`. 11791 11792May be set at any time, e.g. manually set to zero at the start of each kernel. 11793 11794.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 11795 11796.amdgcn.next_free_sgpr 11797++++++++++++++++++++++ 11798 11799Set to zero before assembly begins. At each instruction, if the current value 11800of this symbol is less than or equal the maximum SGPR number explicitly 11801referenced within that instruction then the symbol value is updated to equal 11802that SGPR number plus one. 11803 11804May be used to set the `.amdhsa_next_free_spgr` directive in 11805:ref:`amdhsa-kernel-directives-table`. 11806 11807May be set at any time, e.g. manually set to zero at the start of each kernel. 11808 11809.. _amdgpu-amdhsa-assembler-directives-v3-v4: 11810 11811Code Object V3 to V4 Directives 11812~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11813 11814Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 11815architecture processors, and are not OS-specific. Directives which begin with 11816``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 11817``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 11818:ref:`amdgpu-processors`. 11819 11820.. _amdgpu-assembler-directive-amdgcn-target: 11821 11822.amdgcn_target <target-triple> "-" <target-id> 11823++++++++++++++++++++++++++++++++++++++++++++++ 11824 11825Optional directive which declares the ``<target-triple>-<target-id>`` supported 11826by the containing assembler source file. Used by the assembler to validate 11827command-line options such as ``-triple``, ``-mcpu``, and 11828``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 11829:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 11830 11831.. note:: 11832 11833 The target ID syntax used for code object V2 to V3 for this directive differs 11834 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 11835 11836.amdhsa_kernel <name> 11837+++++++++++++++++++++ 11838 11839Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 11840``<name>.kd``, in the current location of the current section. Only valid when 11841the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 11842instruction to execute, and does not need to be previously defined. 11843 11844Marks the beginning of a list of directives used to generate the bytes of a 11845kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 11846Directives which may appear in this list are described in 11847:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 11848be valid for the target being assembled for, and cannot be repeated. Directives 11849support the range of values specified by the field they reference in 11850:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 11851assumed to have its default value, unless it is marked as "Required", in which 11852case it is an error to omit the directive. This list of directives is 11853terminated by an ``.end_amdhsa_kernel`` directive. 11854 11855 .. table:: AMDHSA Kernel Assembler Directives 11856 :name: amdhsa-kernel-directives-table 11857 11858 ======================================================== =================== ============ =================== 11859 Directive Default Supported On Description 11860 ======================================================== =================== ============ =================== 11861 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 11862 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11863 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 11864 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11865 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 11866 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11867 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 11868 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11869 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 11870 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11871 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 11872 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11873 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 11874 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11875 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 11876 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11877 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 11878 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11879 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 11880 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11881 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 11882 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11883 Specific 11884 (wavefrontsize64) 11885 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 11886 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11887 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 11888 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11889 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 11890 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11891 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 11892 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11893 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 11894 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11895 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 11896 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11897 Possible values are defined in 11898 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 11899 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 11900 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 11901 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11902 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 11903 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11904 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11905 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 11906 Used to calculate ACCUM_OFFSET in 11907 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11908 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 11909 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11910 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11911 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 11912 scratch memory. Used to calculate 11913 GRANULATED_WAVEFRONT_SGPR_COUNT in 11914 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11915 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 11916 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11917 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11918 (xnack) 11919 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 11920 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11921 Possible values are defined in 11922 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11923 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 11924 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11925 Possible values are defined in 11926 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11927 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 11928 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11929 Possible values are defined in 11930 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11931 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 11932 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11933 Possible values are defined in 11934 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11935 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 11936 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11937 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 11938 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11939 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 11940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11941 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 11942 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11943 Specific 11944 (tgsplit) 11945 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 11946 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11947 Specific 11948 (cumode) 11949 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 11950 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11951 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 11952 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11953 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 11954 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11955 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 11956 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11957 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 11958 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11959 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 11960 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11961 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 11962 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11963 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 11964 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11965 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 11966 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11967 ======================================================== =================== ============ =================== 11968 11969.amdgpu_metadata 11970++++++++++++++++ 11971 11972Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 11973note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 11974 11975The contents must be in the [YAML]_ markup format, with the same structure and 11976semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 11977:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 11978 11979This directive is terminated by an ``.end_amdgpu_metadata`` directive. 11980 11981.. _amdgpu-amdhsa-assembler-example-v3-v4: 11982 11983Code Object V3 to V4 Example Source Code 11984~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11985 11986Here is an example of a minimal assembly source file, defining one HSA kernel: 11987 11988.. code:: 11989 :number-lines: 11990 11991 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 11992 11993 .text 11994 .globl hello_world 11995 .p2align 8 11996 .type hello_world,@function 11997 hello_world: 11998 s_load_dwordx2 s[0:1], s[0:1] 0x0 11999 v_mov_b32 v0, 3.14159 12000 s_waitcnt lgkmcnt(0) 12001 v_mov_b32 v1, s0 12002 v_mov_b32 v2, s1 12003 flat_store_dword v[1:2], v0 12004 s_endpgm 12005 .Lfunc_end0: 12006 .size hello_world, .Lfunc_end0-hello_world 12007 12008 .rodata 12009 .p2align 6 12010 .amdhsa_kernel hello_world 12011 .amdhsa_user_sgpr_kernarg_segment_ptr 1 12012 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12013 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12014 .end_amdhsa_kernel 12015 12016 .amdgpu_metadata 12017 --- 12018 amdhsa.version: 12019 - 1 12020 - 0 12021 amdhsa.kernels: 12022 - .name: hello_world 12023 .symbol: hello_world.kd 12024 .kernarg_segment_size: 48 12025 .group_segment_fixed_size: 0 12026 .private_segment_fixed_size: 0 12027 .kernarg_segment_align: 4 12028 .wavefront_size: 64 12029 .sgpr_count: 2 12030 .vgpr_count: 3 12031 .max_flat_workgroup_size: 256 12032 ... 12033 .end_amdgpu_metadata 12034 12035If an assembly source file contains multiple kernels and/or functions, the 12036:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 12037:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 12038the ``.set <symbol>, <expression>`` directive. For example, in the case of two 12039kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 12040to group the function with the kernel that calls it and reset the symbols 12041between the two connected components: 12042 12043.. code:: 12044 :number-lines: 12045 12046 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12047 12048 // gpr tracking symbols are implicitly set to zero 12049 12050 .text 12051 .globl kern0 12052 .p2align 8 12053 .type kern0,@function 12054 kern0: 12055 // ... 12056 s_endpgm 12057 .Lkern0_end: 12058 .size kern0, .Lkern0_end-kern0 12059 12060 .rodata 12061 .p2align 6 12062 .amdhsa_kernel kern0 12063 // ... 12064 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12065 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12066 .end_amdhsa_kernel 12067 12068 // reset symbols to begin tracking usage in func1 and kern1 12069 .set .amdgcn.next_free_vgpr, 0 12070 .set .amdgcn.next_free_sgpr, 0 12071 12072 .text 12073 .hidden func1 12074 .global func1 12075 .p2align 2 12076 .type func1,@function 12077 func1: 12078 // ... 12079 s_setpc_b64 s[30:31] 12080 .Lfunc1_end: 12081 .size func1, .Lfunc1_end-func1 12082 12083 .globl kern1 12084 .p2align 8 12085 .type kern1,@function 12086 kern1: 12087 // ... 12088 s_getpc_b64 s[4:5] 12089 s_add_u32 s4, s4, func1@rel32@lo+4 12090 s_addc_u32 s5, s5, func1@rel32@lo+4 12091 s_swappc_b64 s[30:31], s[4:5] 12092 // ... 12093 s_endpgm 12094 .Lkern1_end: 12095 .size kern1, .Lkern1_end-kern1 12096 12097 .rodata 12098 .p2align 6 12099 .amdhsa_kernel kern1 12100 // ... 12101 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12102 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12103 .end_amdhsa_kernel 12104 12105These symbols cannot identify connected components in order to automatically 12106track the usage for each kernel. However, in some cases careful organization of 12107the kernels and functions in the source file means there is minimal additional 12108effort required to accurately calculate GPR usage. 12109 12110Additional Documentation 12111======================== 12112 12113.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 12114.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 12115.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 12116.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 12117.. [AMD-GCN-GFX10-RDNA1] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 12118.. [AMD-GCN-GFX10-RDNA2] `AMD "RDNA 2" Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 12119.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 12120.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 12121.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 12122.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 12123.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 12124.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12125.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12126.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12127.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12128.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12129.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12130.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12131.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12132.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12133.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12134.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12135