1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 42to specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as: 73 74 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 75 loader on Linux. See *AMD ROCm Platform Release Notes* 76 [AMD-ROCm-Release-Notes]_ for supported hardware and 77 software. 78 - AMD's PAL runtime using the *pal-amdhsa* loader on 79 Windows. 80 81 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 82 runtime using the *pal-amdpal* loader on Windows and Linux 83 Pro. 84 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 85 3D runtime using the *mesa-mesa3d* loader on Linux. 86 ============== ============================================================ 87 88 .. table:: AMDGPU Environments 89 :name: amdgpu-environment-table 90 91 ============ ============================================================== 92 Environment Description 93 ============ ============================================================== 94 *<empty>* Default. 95 ============ ============================================================== 96 97.. _amdgpu-processors: 98 99Processors 100---------- 101 102Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 103specify the AMDGPU processor together with optional target features. See 104:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 105specific information. 106 107Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 108 109* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 110 111 112 .. table:: AMDGPU Processors 113 :name: amdgpu-processor-table 114 115 =========== =============== ============ ===== ================= =============== =============== ====================== 116 Processor Alternative Target dGPU/ Target Target OS Support Example 117 Processor Triple APU Features Properties *(see* Products 118 Architecture Supported `amdgpu-os`_ 119 *and 120 corresponding 121 runtime release 122 notes for 123 current 124 information and 125 level of 126 support)* 127 =========== =============== ============ ===== ================= =============== =============== ====================== 128 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 129 ----------------------------------------------------------------------------------------------------------------------- 130 ``r600`` ``r600`` dGPU - Does not 131 support 132 generic 133 address 134 space 135 ``r630`` ``r600`` dGPU - Does not 136 support 137 generic 138 address 139 space 140 ``rs880`` ``r600`` dGPU - Does not 141 support 142 generic 143 address 144 space 145 ``rv670`` ``r600`` dGPU - Does not 146 support 147 generic 148 address 149 space 150 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 151 ----------------------------------------------------------------------------------------------------------------------- 152 ``rv710`` ``r600`` dGPU - Does not 153 support 154 generic 155 address 156 space 157 ``rv730`` ``r600`` dGPU - Does not 158 support 159 generic 160 address 161 space 162 ``rv770`` ``r600`` dGPU - Does not 163 support 164 generic 165 address 166 space 167 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 168 ----------------------------------------------------------------------------------------------------------------------- 169 ``cedar`` ``r600`` dGPU - Does not 170 support 171 generic 172 address 173 space 174 ``cypress`` ``r600`` dGPU - Does not 175 support 176 generic 177 address 178 space 179 ``juniper`` ``r600`` dGPU - Does not 180 support 181 generic 182 address 183 space 184 ``redwood`` ``r600`` dGPU - Does not 185 support 186 generic 187 address 188 space 189 ``sumo`` ``r600`` dGPU - Does not 190 support 191 generic 192 address 193 space 194 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 195 ----------------------------------------------------------------------------------------------------------------------- 196 ``barts`` ``r600`` dGPU - Does not 197 support 198 generic 199 address 200 space 201 ``caicos`` ``r600`` dGPU - Does not 202 support 203 generic 204 address 205 space 206 ``cayman`` ``r600`` dGPU - Does not 207 support 208 generic 209 address 210 space 211 ``turks`` ``r600`` dGPU - Does not 212 support 213 generic 214 address 215 space 216 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 217 ----------------------------------------------------------------------------------------------------------------------- 218 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 219 support 220 generic 221 address 222 space 223 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 224 - ``verde`` support 225 generic 226 address 227 space 228 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 229 - ``oland`` support 230 generic 231 address 232 space 233 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 234 ----------------------------------------------------------------------------------------------------------------------- 235 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 236 flat - *pal-amdhsa* - A6 Pro-7050B 237 scratch - *pal-amdpal* - A8-7100 238 - A8 Pro-7150B 239 - A10-7300 240 - A10 Pro-7350B 241 - FX-7500 242 - A8-7200P 243 - A10-7400P 244 - FX-7600P 245 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 246 flat - *pal-amdhsa* - FirePro W9100 247 scratch - *pal-amdpal* - FirePro S9150 248 - FirePro S9170 249 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 250 flat - *pal-amdhsa* - Radeon R9 290x 251 scratch - *pal-amdpal* - Radeon R390 252 - Radeon R390x 253 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 254 - ``mullins`` flat - *pal-amdpal* - E1-2200 255 scratch - E1-2500 256 - E2-3000 257 - E2-3800 258 - A4-5000 259 - A4-5100 260 - A6-5200 261 - A4 Pro-3340B 262 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 263 flat - *pal-amdpal* - Radeon HD 8770 264 scratch - R7 260 265 - R7 260X 266 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 267 flat - *pal-amdpal* 268 scratch .. TODO:: 269 270 Add product 271 names. 272 273 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 274 ----------------------------------------------------------------------------------------------------------------------- 275 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 276 flat - *pal-amdhsa* - Pro A6-8500B 277 scratch - *pal-amdpal* - A8-8600P 278 - Pro A8-8600B 279 - FX-8800P 280 - Pro A12-8800B 281 - A10-8700P 282 - Pro A10-8700B 283 - A10-8780P 284 - A10-9600P 285 - A10-9630P 286 - A12-9700P 287 - A12-9730P 288 - FX-9800P 289 - FX-9830P 290 - E2-9010 291 - A6-9210 292 - A9-9410 293 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 294 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 295 scratch - *pal-amdpal* - Radeon R9 385 296 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 297 - *pal-amdhsa* - Radeon R9 Fury 298 - *pal-amdpal* - Radeon R9 FuryX 299 - Radeon Pro Duo 300 - FirePro S9300x2 301 - Radeon Instinct MI8 302 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 303 flat - *pal-amdhsa* - Radeon RX 480 304 scratch - *pal-amdpal* - Radeon Instinct MI6 305 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 306 flat - *pal-amdhsa* 307 scratch - *pal-amdpal* 308 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 309 flat - *pal-amdhsa* - FirePro S7100 310 scratch - *pal-amdpal* - FirePro W7100 311 - Mobile FirePro 312 M7170 313 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 314 flat - *pal-amdhsa* 315 scratch - *pal-amdpal* .. TODO:: 316 317 Add product 318 names. 319 320 **GCN GFX9 (Vega)** [AMD-GCN-GFX9]_ 321 ----------------------------------------------------------------------------------------------------------------------- 322 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 323 flat - *pal-amdhsa* Frontier Edition 324 scratch - *pal-amdpal* - Radeon RX Vega 56 325 - Radeon RX Vega 64 326 - Radeon RX Vega 64 327 Liquid 328 - Radeon Instinct MI25 329 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 330 flat - *pal-amdhsa* - Ryzen 5 2400G 331 scratch - *pal-amdpal* 332 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 333 - *pal-amdhsa* 334 - *pal-amdpal* .. TODO:: 335 336 Add product 337 names. 338 339 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 340 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 341 scratch - *pal-amdpal* - Radeon VII 342 - Radeon Pro VII 343 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* *TBA* 344 - xnack - Absolute 345 flat .. TODO:: 346 scratch 347 Add product 348 names. 349 350 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 351 flat 352 scratch .. TODO:: 353 354 Add product 355 names. 356 357 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 358 - tgsplit flat 359 - xnack scratch .. TODO:: 360 - Packed 361 work-item Add product 362 IDs names. 363 364 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 365 flat - Ryzen 7 4700GE 366 scratch - Ryzen 5 4600G 367 - Ryzen 5 4600GE 368 - Ryzen 3 4300G 369 - Ryzen 3 4300GE 370 - Ryzen Pro 4000G 371 - Ryzen 7 Pro 4700G 372 - Ryzen 7 Pro 4750GE 373 - Ryzen 5 Pro 4650G 374 - Ryzen 5 Pro 4650GE 375 - Ryzen 3 Pro 4350G 376 - Ryzen 3 Pro 4350GE 377 378 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 379 ----------------------------------------------------------------------------------------------------------------------- 380 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 381 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 382 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 383 - Radeon Pro 5600M 384 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* *TBA* 385 - wavefrontsize64 - Absolute - *pal-amdhsa* 386 - xnack flat - *pal-amdpal* 387 scratch .. TODO:: 388 389 Add product 390 names. 391 392 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 393 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 394 - xnack scratch - *pal-amdpal* 395 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 396 ----------------------------------------------------------------------------------------------------------------------- 397 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 398 - wavefrontsize64 flat - *pal-amdhsa* 399 scratch - *pal-amdpal* .. TODO:: 400 401 Add product 402 names. 403 404 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 405 - wavefrontsize64 flat - *pal-amdhsa* 406 scratch - *pal-amdpal* .. TODO:: 407 408 Add product 409 names. 410 411 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 412 - wavefrontsize64 flat - *pal-amdhsa* 413 scratch - *pal-amdpal* .. TODO:: 414 415 Add product 416 names. 417 418 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 419 - wavefrontsize64 flat 420 scratch .. TODO:: 421 422 Add product 423 names. 424 425 =========== =============== ============ ===== ================= =============== =============== ====================== 426 427.. _amdgpu-target-features: 428 429Target Features 430--------------- 431 432Target features control how code is generated to support certain 433processor specific features. Not all target features are supported by 434all processors. The runtime must ensure that the features supported by 435the device used to execute the code match the features enabled when 436generating the code. A mismatch of features may result in incorrect 437execution, or a reduction in performance. 438 439The target features supported by each processor is listed in 440:ref:`amdgpu-processor-table`. 441 442Target features are controlled by exactly one of the following Clang 443options: 444 445``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 446 447 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 448 optional components of the target ID. If omitted, the target feature has the 449 ``any`` value. See :ref:`amdgpu-target-id`. 450 451``-m[no-]<target-feature>`` 452 453 Target features not specified by the target ID are specified using a 454 separate option. These target features can have an ``on`` or ``off`` 455 value. ``on`` is specified by omitting the ``no-`` prefix, and 456 ``off`` is specified by including the ``no-`` prefix. The default 457 if not specified is ``off``. 458 459For example: 460 461``-mcpu=gfx908:xnack+`` 462 Enable the ``xnack`` feature. 463``-mcpu=gfx908:xnack-`` 464 Disable the ``xnack`` feature. 465``-mcumode`` 466 Enable the ``cumode`` feature. 467``-mno-cumode`` 468 Disable the ``cumode`` feature. 469 470 .. table:: AMDGPU Target Features 471 :name: amdgpu-target-features-table 472 473 =============== ============================ ================================================== 474 Target Feature Clang Option to Control Description 475 Name 476 =============== ============================ ================================================== 477 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 478 when generating code for kernels. When disabled 479 native WGP wavefront execution mode is used, 480 when enabled CU wavefront execution mode is used 481 (see :ref:`amdgpu-amdhsa-memory-model`). 482 483 sramecc - ``-mcpu`` If specified, generate code that can only be 484 - ``--offload-arch`` loaded and executed in a process that has a 485 matching setting for SRAMECC. 486 487 If not specified for code object V2 to V3, generate 488 code that can be loaded and executed in a process 489 with SRAMECC enabled. 490 491 If not specified for code object V4, generate 492 code that can be loaded and executed in a process 493 with either setting of SRAMECC. 494 495 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 496 work-groups are launched in threadgroup split mode. 497 When enabled the waves of a work-group may be 498 launched in different CUs. 499 500 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 501 generating code for kernels. When disabled 502 native wavefront size 32 is used, when enabled 503 wavefront size 64 is used. 504 505 xnack - ``-mcpu`` If specified, generate code that can only be 506 - ``--offload-arch`` loaded and executed in a process that has a 507 matching setting for XNACK replay. 508 509 If not specified for code object V2 to V3, generate 510 code that can be loaded and executed in a process 511 with XNACK replay enabled. 512 513 If not specified for code object V4, generate 514 code that can be loaded and executed in a process 515 with either setting of XNACK replay. 516 517 XNACK replay can be used for demand paging and 518 page migration. If enabled in the device, then if 519 a page fault occurs the code may execute 520 incorrectly unless generated with XNACK replay 521 enabled, or generated for code object V4 without 522 specifying XNACK replay. Executing code that was 523 generated with XNACK replay enabled, or generated 524 for code object V4 without specifying XNACK replay, 525 on a device that does not have XNACK replay 526 enabled will execute correctly but may be less 527 performant than code generated for XNACK replay 528 disabled. 529 =============== ============================ ================================================== 530 531.. _amdgpu-target-id: 532 533Target ID 534--------- 535 536AMDGPU supports target IDs. See `Clang Offload Bundler 537<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 538description. The AMDGPU target specific information is: 539 540**processor** 541 Is an AMDGPU processor or alternative processor name specified in 542 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 543 the primary processor and alternative processor names. The canonical form 544 target ID only allow the primary processor name. 545 546**target-feature** 547 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 548 is supported by the processor. The target features supported by each processor 549 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 550 a target ID are marked as being controlled by ``-mcpu`` and 551 ``--offload-arch``. Each target feature must appear at most once in a target 552 ID. The non-canonical form target ID allows the target features to be 553 specified in any order. The canonical form target ID requires the target 554 features to be specified in alphabetic order. 555 556.. _amdgpu-target-id-v2-v3: 557 558Code Object V2 to V3 Target ID 559~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 560 561The target ID syntax for code object V2 to V3 is the same as defined in `Clang 562Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 563when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 564directive and the bundle entry ID. In those cases it has the following BNF 565syntax: 566 567.. code:: 568 569 <target-id> ::== <processor> ( "+" <target-feature> )* 570 571Where a target feature is omitted if *Off* and present if *On* or *Any*. 572 573.. note:: 574 575 The code object V2 to V3 cannot represent *Any* and treats it the same as 576 *On*. 577 578.. _amdgpu-embedding-bundled-objects: 579 580Embedding Bundled Code Objects 581------------------------------ 582 583AMDGPU supports the HIP and OpenMP languages that perform code object embedding 584as described in `Clang Offload Bundler 585<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 586 587.. note:: 588 589 The target ID syntax used for code object V2 to V3 for a bundle entry ID 590 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 591 592.. _amdgpu-address-spaces: 593 594Address Spaces 595-------------- 596 597The AMDGPU architecture supports a number of memory address spaces. The address 598space names use the OpenCL standard names, with some additions. 599 600The AMDGPU address spaces correspond to target architecture specific LLVM 601address space numbers used in LLVM IR. 602 603The AMDGPU address spaces are described in 604:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 605supported for the ``amdgcn`` target. 606 607 .. table:: AMDGPU Address Spaces 608 :name: amdgpu-address-spaces-table 609 610 ================================= =============== =========== ================ ======= ============================ 611 .. 64-Bit Process Address Space 612 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 613 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 614 Space Number Name Name Size 615 ================================= =============== =========== ================ ======= ============================ 616 Generic 0 flat flat 64 0x0000000000000000 617 Global 1 global global 64 0x0000000000000000 618 Region 2 N/A GDS 32 *not implemented for AMDHSA* 619 Local 3 group LDS 32 0xFFFFFFFF 620 Constant 4 constant *same as global* 64 0x0000000000000000 621 Private 5 private scratch 32 0xFFFFFFFF 622 Constant 32-bit 6 *TODO* 0x00000000 623 Buffer Fat Pointer (experimental) 7 *TODO* 624 ================================= =============== =========== ================ ======= ============================ 625 626**Generic** 627 The generic address space is supported unless the *Target Properties* column 628 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 629 space*. 630 631 The generic address space uses the hardware flat address support for two fixed 632 ranges of virtual addresses (the private and local apertures), that are 633 outside the range of addressable global memory, to map from a flat address to 634 a private or local address. This uses FLAT instructions that can take a flat 635 address and access global, private (scratch), and group (LDS) memory depending 636 on if the address is within one of the aperture ranges. 637 638 Flat access to scratch requires hardware aperture setup and setup in the 639 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 640 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 641 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 642 643 To convert between a private or group address space address (termed a segment 644 address) and a flat address the base address of the corresponding aperture 645 can be used. For GFX7-GFX8 these are available in the 646 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 647 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 648 GFX9-GFX10 the aperture base addresses are directly available as inline 649 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 650 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 651 aligned to 2^32 which makes it easier to convert from flat to segment or 652 segment to flat. 653 654 A global address space address has the same value when used as a flat address 655 so no conversion is needed. 656 657**Global and Constant** 658 The global and constant address spaces both use global virtual addresses, 659 which are the same virtual address space used by the CPU. However, some 660 virtual addresses may only be accessible to the CPU, some only accessible 661 by the GPU, and some by both. 662 663 Using the constant address space indicates that the data will not change 664 during the execution of the kernel. This allows scalar read instructions to 665 be used. As the constant address space could only be modified on the host 666 side, a generic pointer loaded from the constant address space is safe to be 667 assumed as a global pointer since only the device global memory is visible 668 and managed on the host side. The vector and scalar L1 caches are invalidated 669 of volatile data before each kernel dispatch execution to allow constant 670 memory to change values between kernel dispatches. 671 672**Region** 673 The region address space uses the hardware Global Data Store (GDS). All 674 wavefronts executing on the same device will access the same memory for any 675 given region address. However, the same region address accessed by wavefronts 676 executing on different devices will access different memory. It is higher 677 performance than global memory. It is allocated by the runtime. The data 678 store (DS) instructions can be used to access it. 679 680**Local** 681 The local address space uses the hardware Local Data Store (LDS) which is 682 automatically allocated when the hardware creates the wavefronts of a 683 work-group, and freed when all the wavefronts of a work-group have 684 terminated. All wavefronts belonging to the same work-group will access the 685 same memory for any given local address. However, the same local address 686 accessed by wavefronts belonging to different work-groups will access 687 different memory. It is higher performance than global memory. The data store 688 (DS) instructions can be used to access it. 689 690**Private** 691 The private address space uses the hardware scratch memory support which 692 automatically allocates memory when it creates a wavefront and frees it when 693 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 694 given private address will be different to the memory accessed by another lane 695 of the same or different wavefront for the same private address. 696 697 If a kernel dispatch uses scratch, then the hardware allocates memory from a 698 pool of backing memory allocated by the runtime for each wavefront. The lanes 699 of the wavefront access this using dword (4 byte) interleaving. The mapping 700 used from private address to backing memory address is: 701 702 ``wavefront-scratch-base + 703 ((private-address / 4) * wavefront-size * 4) + 704 (wavefront-lane-id * 4) + (private-address % 4)`` 705 706 If each lane of a wavefront accesses the same private address, the 707 interleaving results in adjacent dwords being accessed and hence requires 708 fewer cache lines to be fetched. 709 710 There are different ways that the wavefront scratch base address is 711 determined by a wavefront (see 712 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 713 714 Scratch memory can be accessed in an interleaved manner using buffer 715 instructions with the scratch buffer descriptor and per wavefront scratch 716 offset, by the scratch instructions, or by flat instructions. Multi-dword 717 access is not supported except by flat and scratch instructions in 718 GFX9-GFX10. 719 720**Constant 32-bit** 721 *TODO* 722 723**Buffer Fat Pointer** 724 The buffer fat pointer is an experimental address space that is currently 725 unsupported in the backend. It exposes a non-integral pointer that is in 726 the future intended to support the modelling of 128-bit buffer descriptors 727 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 728 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 729 model the buffer descriptors used heavily in graphics workloads targeting 730 the backend. 731 732.. _amdgpu-memory-scopes: 733 734Memory Scopes 735------------- 736 737This section provides LLVM memory synchronization scopes supported by the AMDGPU 738backend memory model when the target triple OS is ``amdhsa`` (see 739:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 740 741The memory model supported is based on the HSA memory model [HSA]_ which is 742based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 743relation is transitive over the synchronizes-with relation independent of scope 744and synchronizes-with allows the memory scope instances to be inclusive (see 745table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 746 747This is different to the OpenCL [OpenCL]_ memory model which does not have scope 748inclusion and requires the memory scopes to exactly match. However, this 749is conservatively correct for OpenCL. 750 751 .. table:: AMDHSA LLVM Sync Scopes 752 :name: amdgpu-amdhsa-llvm-sync-scopes-table 753 754 ======================= =================================================== 755 LLVM Sync Scope Description 756 ======================= =================================================== 757 *none* The default: ``system``. 758 759 Synchronizes with, and participates in modification 760 and seq_cst total orderings with, other operations 761 (except image operations) for all address spaces 762 (except private, or generic that accesses private) 763 provided the other operation's sync scope is: 764 765 - ``system``. 766 - ``agent`` and executed by a thread on the same 767 agent. 768 - ``workgroup`` and executed by a thread in the 769 same work-group. 770 - ``wavefront`` and executed by a thread in the 771 same wavefront. 772 773 ``agent`` Synchronizes with, and participates in modification 774 and seq_cst total orderings with, other operations 775 (except image operations) for all address spaces 776 (except private, or generic that accesses private) 777 provided the other operation's sync scope is: 778 779 - ``system`` or ``agent`` and executed by a thread 780 on the same agent. 781 - ``workgroup`` and executed by a thread in the 782 same work-group. 783 - ``wavefront`` and executed by a thread in the 784 same wavefront. 785 786 ``workgroup`` Synchronizes with, and participates in modification 787 and seq_cst total orderings with, other operations 788 (except image operations) for all address spaces 789 (except private, or generic that accesses private) 790 provided the other operation's sync scope is: 791 792 - ``system``, ``agent`` or ``workgroup`` and 793 executed by a thread in the same work-group. 794 - ``wavefront`` and executed by a thread in the 795 same wavefront. 796 797 ``wavefront`` Synchronizes with, and participates in modification 798 and seq_cst total orderings with, other operations 799 (except image operations) for all address spaces 800 (except private, or generic that accesses private) 801 provided the other operation's sync scope is: 802 803 - ``system``, ``agent``, ``workgroup`` or 804 ``wavefront`` and executed by a thread in the 805 same wavefront. 806 807 ``singlethread`` Only synchronizes with and participates in 808 modification and seq_cst total orderings with, 809 other operations (except image operations) running 810 in the same thread for all address spaces (for 811 example, in signal handlers). 812 813 ``one-as`` Same as ``system`` but only synchronizes with other 814 operations within the same address space. 815 816 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 817 operations within the same address space. 818 819 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 820 other operations within the same address space. 821 822 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 823 other operations within the same address space. 824 825 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 826 other operations within the same address space. 827 ======================= =================================================== 828 829LLVM IR Intrinsics 830------------------ 831 832The AMDGPU backend implements the following LLVM IR intrinsics. 833 834*This section is WIP.* 835 836.. TODO:: 837 838 List AMDGPU intrinsics. 839 840LLVM IR Attributes 841------------------ 842 843The AMDGPU backend supports the following LLVM IR attributes. 844 845 .. table:: AMDGPU LLVM IR Attributes 846 :name: amdgpu-llvm-ir-attributes-table 847 848 ======================================= ========================================================== 849 LLVM Attribute Description 850 ======================================= ========================================================== 851 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 852 will be specified when the kernel is dispatched. Generated 853 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 854 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 855 argument block size for the implicit arguments. This 856 varies by OS and language (for OpenCL see 857 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 858 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 859 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 860 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 861 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 862 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 863 execution unit. Generated by the ``amdgpu_waves_per_eu`` 864 CLANG attribute [CLANG-ATTR]_. 865 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 866 mode register to be set on entry. Overrides the default for 867 the calling convention. 868 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 869 the mode register to be set on entry. Overrides the default 870 for the calling convention. 871 ======================================= ========================================================== 872 873.. _amdgpu-elf-code-object: 874 875ELF Code Object 876=============== 877 878The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 879can be linked by ``lld`` to produce a standard ELF shared code object which can 880be loaded and executed on an AMDGPU target. 881 882.. _amdgpu-elf-header: 883 884Header 885------ 886 887The AMDGPU backend uses the following ELF header: 888 889 .. table:: AMDGPU ELF Header 890 :name: amdgpu-elf-header-table 891 892 ========================== =============================== 893 Field Value 894 ========================== =============================== 895 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 896 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 897 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 898 - ``ELFOSABI_AMDGPU_HSA`` 899 - ``ELFOSABI_AMDGPU_PAL`` 900 - ``ELFOSABI_AMDGPU_MESA3D`` 901 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 902 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 903 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 904 - ``ELFABIVERSION_AMDGPU_PAL`` 905 - ``ELFABIVERSION_AMDGPU_MESA3D`` 906 ``e_type`` - ``ET_REL`` 907 - ``ET_DYN`` 908 ``e_machine`` ``EM_AMDGPU`` 909 ``e_entry`` 0 910 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 911 :ref:`amdgpu-elf-header-e_flags-table-v3`, 912 and :ref:`amdgpu-elf-header-e_flags-table-v4` 913 ========================== =============================== 914 915.. 916 917 .. table:: AMDGPU ELF Header Enumeration Values 918 :name: amdgpu-elf-header-enumeration-values-table 919 920 =============================== ===== 921 Name Value 922 =============================== ===== 923 ``EM_AMDGPU`` 224 924 ``ELFOSABI_NONE`` 0 925 ``ELFOSABI_AMDGPU_HSA`` 64 926 ``ELFOSABI_AMDGPU_PAL`` 65 927 ``ELFOSABI_AMDGPU_MESA3D`` 66 928 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 929 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 930 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 931 ``ELFABIVERSION_AMDGPU_PAL`` 0 932 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 933 =============================== ===== 934 935``e_ident[EI_CLASS]`` 936 The ELF class is: 937 938 * ``ELFCLASS32`` for ``r600`` architecture. 939 940 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 941 process address space applications. 942 943``e_ident[EI_DATA]`` 944 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 945 946``e_ident[EI_OSABI]`` 947 One of the following AMDGPU target architecture specific OS ABIs 948 (see :ref:`amdgpu-os`): 949 950 * ``ELFOSABI_NONE`` for *unknown* OS. 951 952 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 953 954 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 955 956 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 957 958``e_ident[EI_ABIVERSION]`` 959 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 960 object conforms: 961 962 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 963 runtime ABI for code object V2. Specify using the Clang option 964 ``-mcode-object-version=2``. 965 966 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 967 runtime ABI for code object V3. Specify using the Clang option 968 ``-mcode-object-version=3``. This is the default code object 969 version if not specified. 970 971 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 972 runtime ABI for code object V4. Specify using the Clang option 973 ``-mcode-object-version=4``. 974 975 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 976 runtime ABI. 977 978 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 979 3D runtime ABI. 980 981``e_type`` 982 Can be one of the following values: 983 984 985 ``ET_REL`` 986 The type produced by the AMDGPU backend compiler as it is relocatable code 987 object. 988 989 ``ET_DYN`` 990 The type produced by the linker as it is a shared code object. 991 992 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 993 994``e_machine`` 995 The value ``EM_AMDGPU`` is used for the machine for all processors supported 996 by the ``r600`` and ``amdgcn`` architectures (see 997 :ref:`amdgpu-processor-table`). The specific processor is specified in the 998 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 999 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1000 ``e_flags`` for code object V3 to V4 (see 1001 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1002 :ref:`amdgpu-elf-header-e_flags-table-v4`). 1003 1004``e_entry`` 1005 The entry point is 0 as the entry points for individual kernels must be 1006 selected in order to invoke them through AQL packets. 1007 1008``e_flags`` 1009 The AMDGPU backend uses the following ELF header flags: 1010 1011 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1012 :name: amdgpu-elf-header-e_flags-v2-table 1013 1014 ===================================== ===== ============================= 1015 Name Value Description 1016 ===================================== ===== ============================= 1017 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1018 target feature is 1019 enabled for all code 1020 contained in the code object. 1021 If the processor 1022 does not support the 1023 ``xnack`` target 1024 feature then must 1025 be 0. 1026 See 1027 :ref:`amdgpu-target-features`. 1028 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1029 handler is enabled for all 1030 code contained in the code 1031 object. If the processor 1032 does not support a trap 1033 handler then must be 0. 1034 See 1035 :ref:`amdgpu-target-features`. 1036 ===================================== ===== ============================= 1037 1038 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1039 :name: amdgpu-elf-header-e_flags-table-v3 1040 1041 ================================= ===== ============================= 1042 Name Value Description 1043 ================================= ===== ============================= 1044 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1045 mask for 1046 ``EF_AMDGPU_MACH_xxx`` values 1047 defined in 1048 :ref:`amdgpu-ef-amdgpu-mach-table`. 1049 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1050 target feature is 1051 enabled for all code 1052 contained in the code object. 1053 If the processor 1054 does not support the 1055 ``xnack`` target 1056 feature then must 1057 be 0. 1058 See 1059 :ref:`amdgpu-target-features`. 1060 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1061 target feature is 1062 enabled for all code 1063 contained in the code object. 1064 If the processor 1065 does not support the 1066 ``sramecc`` target 1067 feature then must 1068 be 0. 1069 See 1070 :ref:`amdgpu-target-features`. 1071 ================================= ===== ============================= 1072 1073 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1074 :name: amdgpu-elf-header-e_flags-table-v4 1075 1076 ============================================ ===== =================================== 1077 Name Value Description 1078 ============================================ ===== =================================== 1079 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1080 mask for 1081 ``EF_AMDGPU_MACH_xxx`` values 1082 defined in 1083 :ref:`amdgpu-ef-amdgpu-mach-table`. 1084 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1085 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1086 values. 1087 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1088 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1089 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1090 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1091 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1092 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1093 values. 1094 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1095 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1096 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1097 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1098 ============================================ ===== =================================== 1099 1100 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1101 :name: amdgpu-ef-amdgpu-mach-table 1102 1103 ==================================== ========== ============================= 1104 Name Value Description (see 1105 :ref:`amdgpu-processor-table`) 1106 ==================================== ========== ============================= 1107 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1108 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1109 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1110 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1111 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1112 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1113 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1114 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1115 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1116 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1117 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1118 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1119 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1120 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1121 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1122 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1123 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1124 *reserved* 0x011 - Reserved for ``r600`` 1125 0x01f architecture processors. 1126 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1127 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1128 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1129 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1130 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1131 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1132 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1133 *reserved* 0x027 Reserved. 1134 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1135 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1136 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1137 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1138 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1139 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1140 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1141 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1142 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1143 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1144 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1145 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1146 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1147 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1148 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1149 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1150 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1151 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1152 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1153 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1154 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1155 *reserved* 0x03d Reserved. 1156 *reserved* 0x03e Reserved. 1157 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1158 *reserved* 0x040 Reserved. 1159 *reserved* 0x041 Reserved. 1160 ==================================== ========== ============================= 1161 1162Sections 1163-------- 1164 1165An AMDGPU target ELF code object has the standard ELF sections which include: 1166 1167 .. table:: AMDGPU ELF Sections 1168 :name: amdgpu-elf-sections-table 1169 1170 ================== ================ ================================= 1171 Name Type Attributes 1172 ================== ================ ================================= 1173 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1174 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1175 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1176 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1177 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1178 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1179 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1180 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1181 ``.note`` ``SHT_NOTE`` *none* 1182 ``.rela``\ *name* ``SHT_RELA`` *none* 1183 ``.rela.dyn`` ``SHT_RELA`` *none* 1184 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1185 ``.shstrtab`` ``SHT_STRTAB`` *none* 1186 ``.strtab`` ``SHT_STRTAB`` *none* 1187 ``.symtab`` ``SHT_SYMTAB`` *none* 1188 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1189 ================== ================ ================================= 1190 1191These sections have their standard meanings (see [ELF]_) and are only generated 1192if needed. 1193 1194``.debug``\ *\** 1195 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1196 information on the DWARF produced by the AMDGPU backend. 1197 1198``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1199 The standard sections used by a dynamic loader. 1200 1201``.note`` 1202 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1203 backend. 1204 1205``.rela``\ *name*, ``.rela.dyn`` 1206 For relocatable code objects, *name* is the name of the section that the 1207 relocation records apply. For example, ``.rela.text`` is the section name for 1208 relocation records associated with the ``.text`` section. 1209 1210 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1211 records from each of the relocatable code object's ``.rela``\ *name* sections. 1212 1213 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1214 the AMDGPU backend. 1215 1216``.text`` 1217 The executable machine code for the kernels and functions they call. Generated 1218 as position independent code. See :ref:`amdgpu-code-conventions` for 1219 information on conventions used in the isa generation. 1220 1221.. _amdgpu-note-records: 1222 1223Note Records 1224------------ 1225 1226The AMDGPU backend code object contains ELF note records in the ``.note`` 1227section. The set of generated notes and their semantics depend on the code 1228object version; see :ref:`amdgpu-note-records-v2` and 1229:ref:`amdgpu-note-records-v3-v4`. 1230 1231As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1232must be generated after the ``name`` field to ensure the ``desc`` field is 4 1233byte aligned. In addition, minimal zero-byte padding must be generated to 1234ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1235field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1236alignment. 1237 1238.. _amdgpu-note-records-v2: 1239 1240Code Object V2 Note Records 1241~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1242 1243.. warning:: 1244 Code object V2 is not the default code object version emitted by 1245 this version of LLVM. 1246 1247The AMDGPU backend code object uses the following ELF note record in the 1248``.note`` section when compiling for code object V2. 1249 1250The note record vendor field is "AMD". 1251 1252Additional note records may be present, but any which are not documented here 1253are deprecated and should not be used. 1254 1255 .. table:: AMDGPU Code Object V2 ELF Note Records 1256 :name: amdgpu-elf-note-records-v2-table 1257 1258 ===== ===================================== ====================================== 1259 Name Type Description 1260 ===== ===================================== ====================================== 1261 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1262 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1263 Finalizer and not the LLVM compiler. 1264 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1265 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1266 YAML [YAML]_ textual format. 1267 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1268 ===== ===================================== ====================================== 1269 1270.. 1271 1272 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1273 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1274 1275 ===================================== ===== 1276 Name Value 1277 ===================================== ===== 1278 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1279 ``NT_AMD_HSA_HSAIL`` 2 1280 ``NT_AMD_HSA_ISA_VERSION`` 3 1281 *reserved* 4-9 1282 ``NT_AMD_HSA_METADATA`` 10 1283 ``NT_AMD_HSA_ISA_NAME`` 11 1284 ===================================== ===== 1285 1286``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1287 Specifies the code object version number. The description field has the 1288 following layout: 1289 1290 .. code:: 1291 1292 struct amdgpu_hsa_note_code_object_version_s { 1293 uint32_t major_version; 1294 uint32_t minor_version; 1295 }; 1296 1297 The ``major_version`` has a value less than or equal to 2. 1298 1299``NT_AMD_HSA_HSAIL`` 1300 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1301 field has the following layout: 1302 1303 .. code:: 1304 1305 struct amdgpu_hsa_note_hsail_s { 1306 uint32_t hsail_major_version; 1307 uint32_t hsail_minor_version; 1308 uint8_t profile; 1309 uint8_t machine_model; 1310 uint8_t default_float_round; 1311 }; 1312 1313``NT_AMD_HSA_ISA_VERSION`` 1314 Specifies the target ISA version. The description field has the following layout: 1315 1316 .. code:: 1317 1318 struct amdgpu_hsa_note_isa_s { 1319 uint16_t vendor_name_size; 1320 uint16_t architecture_name_size; 1321 uint32_t major; 1322 uint32_t minor; 1323 uint32_t stepping; 1324 char vendor_and_architecture_name[1]; 1325 }; 1326 1327 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1328 vendor and architecture names respectively, including the NUL character. 1329 1330 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1331 vendor, immediately followed by the NUL terminated string for the 1332 architecture. 1333 1334 This note record is used by the HSA runtime loader. 1335 1336 Code object V2 only supports a limited number of processors and has fixed 1337 settings for target features. See 1338 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1339 processors and the corresponding target ID. In the table the note record ISA 1340 name is a concatenation of the vendor name, architecture name, major, minor, 1341 and stepping separated by a ":". 1342 1343 The target ID column shows the processor name and fixed target features used 1344 by the LLVM compiler. The LLVM compiler does not generate a 1345 ``NT_AMD_HSA_HSAIL`` note record. 1346 1347 A code object generated by the Finalizer also uses code object V2 and always 1348 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1349 ``sramecc`` target feature is as shown in 1350 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1351 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1352 bit. 1353 1354``NT_AMD_HSA_ISA_NAME`` 1355 Specifies the target ISA name as a non-NUL terminated string. 1356 1357 This note record is not used by the HSA runtime loader. 1358 1359 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1360 V2's limited support of processors and fixed settings for target features. 1361 1362 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1363 from the string to the corresponding target ID. If the ``xnack`` target 1364 feature is supported and enabled, the string produced by the LLVM compiler 1365 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1366 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1367 1368``NT_AMD_HSA_METADATA`` 1369 Specifies extensible metadata associated with the code objects executed on HSA 1370 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1371 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1372 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1373 metadata string. 1374 1375 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1376 :name: amdgpu-elf-note-record-supported_processors-v2-table 1377 1378 ==================== ========================== 1379 Note Record ISA Name Target ID 1380 ==================== ========================== 1381 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1382 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1383 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1384 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1385 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1386 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1387 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1388 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1389 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1390 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1391 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1392 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1393 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1394 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1395 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1396 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1397 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1398 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1399 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1400 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1401 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1402 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1403 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1404 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1405 ==================== ========================== 1406 1407.. _amdgpu-note-records-v3-v4: 1408 1409Code Object V3 to V4 Note Records 1410~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1411 1412The AMDGPU backend code object uses the following ELF note record in the 1413``.note`` section when compiling for code object V3 to V4. 1414 1415The note record vendor field is "AMDGPU". 1416 1417Additional note records may be present, but any which are not documented here 1418are deprecated and should not be used. 1419 1420 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1421 :name: amdgpu-elf-note-records-table-v3-v4 1422 1423 ======== ============================== ====================================== 1424 Name Type Description 1425 ======== ============================== ====================================== 1426 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1427 binary format. 1428 ======== ============================== ====================================== 1429 1430.. 1431 1432 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1433 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1434 1435 ============================== ===== 1436 Name Value 1437 ============================== ===== 1438 *reserved* 0-31 1439 ``NT_AMDGPU_METADATA`` 32 1440 ============================== ===== 1441 1442``NT_AMDGPU_METADATA`` 1443 Specifies extensible metadata associated with an AMDGPU code object. It is 1444 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1445 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1446 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1447 ``amdhsa`` OS. 1448 1449.. _amdgpu-symbols: 1450 1451Symbols 1452------- 1453 1454Symbols include the following: 1455 1456 .. table:: AMDGPU ELF Symbols 1457 :name: amdgpu-elf-symbols-table 1458 1459 ===================== ================== ================ ================== 1460 Name Type Section Description 1461 ===================== ================== ================ ================== 1462 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1463 - ``.rodata`` 1464 - ``.bss`` 1465 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1466 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1467 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1468 ===================== ================== ================ ================== 1469 1470Global variable 1471 Global variables both used and defined by the compilation unit. 1472 1473 If the symbol is defined in the compilation unit then it is allocated in the 1474 appropriate section according to if it has initialized data or is readonly. 1475 1476 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1477 will resolve relocations using the definition provided by another code object 1478 or explicitly defined by the runtime. 1479 1480 If the symbol resides in local/group memory (LDS) then its section is the 1481 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1482 ``st_value`` field describes alignment requirements as it does for common 1483 symbols. 1484 1485 .. TODO:: 1486 1487 Add description of linked shared object symbols. Seems undefined symbols 1488 are marked as STT_NOTYPE. 1489 1490Kernel descriptor 1491 Every HSA kernel has an associated kernel descriptor. It is the address of the 1492 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1493 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1494 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1495 1496Kernel entry point 1497 Every HSA kernel also has a symbol for its machine code entry point. 1498 1499.. _amdgpu-relocation-records: 1500 1501Relocation Records 1502------------------ 1503 1504AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1505relocatable fields are: 1506 1507``word32`` 1508 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1509 alignment. These values use the same byte order as other word values in the 1510 AMDGPU architecture. 1511 1512``word64`` 1513 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1514 alignment. These values use the same byte order as other word values in the 1515 AMDGPU architecture. 1516 1517Following notations are used for specifying relocation calculations: 1518 1519**A** 1520 Represents the addend used to compute the value of the relocatable field. 1521 1522**G** 1523 Represents the offset into the global offset table at which the relocation 1524 entry's symbol will reside during execution. 1525 1526**GOT** 1527 Represents the address of the global offset table. 1528 1529**P** 1530 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1531 of the storage unit being relocated (computed using ``r_offset``). 1532 1533**S** 1534 Represents the value of the symbol whose index resides in the relocation 1535 entry. Relocations not using this must specify a symbol index of 1536 ``STN_UNDEF``. 1537 1538**B** 1539 Represents the base address of a loaded executable or shared object which is 1540 the difference between the ELF address and the actual load address. 1541 Relocations using this are only valid in executable or shared objects. 1542 1543The following relocation types are supported: 1544 1545 .. table:: AMDGPU ELF Relocation Records 1546 :name: amdgpu-elf-relocation-records-table 1547 1548 ========================== ======= ===== ========== ============================== 1549 Relocation Type Kind Value Field Calculation 1550 ========================== ======= ===== ========== ============================== 1551 ``R_AMDGPU_NONE`` 0 *none* *none* 1552 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1553 Dynamic 1554 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1555 Dynamic 1556 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1557 Dynamic 1558 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1559 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1560 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1561 Dynamic 1562 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1563 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1564 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1565 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1566 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1567 *reserved* 12 1568 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1569 ========================== ======= ===== ========== ============================== 1570 1571``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1572the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1573 1574There is no current OS loader support for 32-bit programs and so 1575``R_AMDGPU_ABS32`` is not used. 1576 1577.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1578 1579Loaded Code Object Path Uniform Resource Identifier (URI) 1580--------------------------------------------------------- 1581 1582The AMD GPU code object loader represents the path of the ELF shared object from 1583which the code object was loaded as a textual Unifom Resource Identifier (URI). 1584Note that the code object is the in memory loaded relocated form of the ELF 1585shared object. Multiple code objects may be loaded at different memory 1586addresses in the same process from the same ELF shared object. 1587 1588The loaded code object path URI syntax is defined by the following BNF syntax: 1589 1590.. code:: 1591 1592 code_object_uri ::== file_uri | memory_uri 1593 file_uri ::== "file://" file_path [ range_specifier ] 1594 memory_uri ::== "memory://" process_id range_specifier 1595 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1596 file_path ::== URI_ENCODED_OS_FILE_PATH 1597 process_id ::== DECIMAL_NUMBER 1598 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1599 1600**number** 1601 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1602 and octal values by "0". 1603 1604**file_path** 1605 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1606 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1607 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1608 the path are separated by "/". 1609 1610**offset** 1611 Is a 0-based byte offset to the start of the code object. For a file URI, it 1612 is from the start of the file specified by the ``file_path``, and if omitted 1613 defaults to 0. For a memory URI, it is the memory address and is required. 1614 1615**size** 1616 Is the number of bytes in the code object. For a file URI, if omitted it 1617 defaults to the size of the file. It is required for a memory URI. 1618 1619**process_id** 1620 Is the identity of the process owning the memory. For Linux it is the C 1621 unsigned integral decimal literal for the process ID (PID). 1622 1623For example: 1624 1625.. code:: 1626 1627 file:///dir1/dir2/file1 1628 file:///dir3/dir4/file2#offset=0x2000&size=3000 1629 memory://1234#offset=0x20000&size=3000 1630 1631.. _amdgpu-dwarf-debug-information: 1632 1633DWARF Debug Information 1634======================= 1635 1636.. warning:: 1637 1638 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1639 is not currently fully implemented and is subject to change. 1640 1641AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1642:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1643object executable code and data to the source language constructs. It can be 1644used by tools such as debuggers and profilers. It uses features defined in 1645:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1646DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1647 1648This section defines the AMDGPU target architecture specific DWARF mappings. 1649 1650.. _amdgpu-dwarf-register-identifier: 1651 1652Register Identifier 1653------------------- 1654 1655This section defines the AMDGPU target architecture register numbers used in 1656DWARF operation expressions (see DWARF Version 5 section 2.5 and 1657:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1658instructions (see DWARF Version 5 section 6.4 and 1659:ref:`amdgpu-dwarf-call-frame-information`). 1660 1661A single code object can contain code for kernels that have different wavefront 1662sizes. The vector registers and some scalar registers are based on the wavefront 1663size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1664simplifies the consumer of the DWARF so that each register has a fixed size, 1665rather than being dynamic according to the wavefront size mode. Similarly, 1666distinct DWARF registers are defined for those registers that vary in size 1667according to the process address size. This allows a consumer to treat a 1668specific AMDGPU processor as a single architecture regardless of how it is 1669configured at run time. The compiler explicitly specifies the DWARF registers 1670that match the mode in which the code it is generating will be executed. 1671 1672DWARF registers are encoded as numbers, which are mapped to architecture 1673registers. The mapping for AMDGPU is defined in 1674:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1675mapping. 1676 1677.. table:: AMDGPU DWARF Register Mapping 1678 :name: amdgpu-dwarf-register-mapping-table 1679 1680 ============== ================= ======== ================================== 1681 DWARF Register AMDGPU Register Bit Size Description 1682 ============== ================= ======== ================================== 1683 0 PC_32 32 Program Counter (PC) when 1684 executing in a 32-bit process 1685 address space. Used in the CFI to 1686 describe the PC of the calling 1687 frame. 1688 1 EXEC_MASK_32 32 Execution Mask Register when 1689 executing in wavefront 32 mode. 1690 2-15 *Reserved* *Reserved for highly accessed 1691 registers using DWARF shortcut.* 1692 16 PC_64 64 Program Counter (PC) when 1693 executing in a 64-bit process 1694 address space. Used in the CFI to 1695 describe the PC of the calling 1696 frame. 1697 17 EXEC_MASK_64 64 Execution Mask Register when 1698 executing in wavefront 64 mode. 1699 18-31 *Reserved* *Reserved for highly accessed 1700 registers using DWARF shortcut.* 1701 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1702 Registers. 1703 96-127 *Reserved* *Reserved for frequently accessed 1704 registers using DWARF 1-byte ULEB.* 1705 128 STATUS 32 Status Register. 1706 129-511 *Reserved* *Reserved for future Scalar 1707 Architectural Registers.* 1708 512 VCC_32 32 Vector Condition Code Register 1709 when executing in wavefront 32 1710 mode. 1711 513-1023 *Reserved* *Reserved for future Vector 1712 Architectural Registers when 1713 executing in wavefront 32 mode.* 1714 768 VCC_64 64 Vector Condition Code Register 1715 when executing in wavefront 64 1716 mode. 1717 769-1023 *Reserved* *Reserved for future Vector 1718 Architectural Registers when 1719 executing in wavefront 64 mode.* 1720 1024-1087 *Reserved* *Reserved for padding.* 1721 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1722 1130-1535 *Reserved* *Reserved for future Scalar 1723 General Purpose Registers.* 1724 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1725 when executing in wavefront 32 1726 mode. 1727 1792-2047 *Reserved* *Reserved for future Vector 1728 General Purpose Registers when 1729 executing in wavefront 32 mode.* 1730 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1731 when executing in wavefront 32 1732 mode. 1733 2304-2559 *Reserved* *Reserved for future Vector 1734 Accumulation Registers when 1735 executing in wavefront 32 mode.* 1736 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1737 when executing in wavefront 64 1738 mode. 1739 2816-3071 *Reserved* *Reserved for future Vector 1740 General Purpose Registers when 1741 executing in wavefront 64 mode.* 1742 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1743 when executing in wavefront 64 1744 mode. 1745 3328-3583 *Reserved* *Reserved for future Vector 1746 Accumulation Registers when 1747 executing in wavefront 64 mode.* 1748 ============== ================= ======== ================================== 1749 1750The vector registers are represented as the full size for the wavefront. They 1751are organized as consecutive dwords (32-bits), one per lane, with the dword at 1752the least significant bit position corresponding to lane 0 and so forth. DWARF 1753location expressions involving the ``DW_OP_LLVM_offset`` and 1754``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1755register corresponding to the lane that is executing the current thread of 1756execution in languages that are implemented using a SIMD or SIMT execution 1757model. 1758 1759If the wavefront size is 32 lanes then the wavefront 32 mode register 1760definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1761mode register definitions are used. Some AMDGPU targets support executing in 1762both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1763to the wavefront mode of the generated code will be used. 1764 1765If code is generated to execute in a 32-bit process address space, then the 176632-bit process address space register definitions are used. If code is generated 1767to execute in a 64-bit process address space, then the 64-bit process address 1768space register definitions are used. The ``amdgcn`` target only supports the 176964-bit process address space. 1770 1771.. _amdgpu-dwarf-address-class-identifier: 1772 1773Address Class Identifier 1774------------------------ 1775 1776The DWARF address class represents the source language memory space. See DWARF 1777Version 5 section 2.12 which is updated by the *DWARF Extensions For 1778Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1779 1780The DWARF address class mapping used for AMDGPU is defined in 1781:ref:`amdgpu-dwarf-address-class-mapping-table`. 1782 1783.. table:: AMDGPU DWARF Address Class Mapping 1784 :name: amdgpu-dwarf-address-class-mapping-table 1785 1786 ========================= ====== ================= 1787 DWARF AMDGPU 1788 -------------------------------- ----------------- 1789 Address Class Name Value Address Space 1790 ========================= ====== ================= 1791 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1792 ``DW_ADDR_LLVM_global`` 0x0001 Global 1793 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1794 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1795 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1796 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1797 ========================= ====== ================= 1798 1799The DWARF address class values defined in the *DWARF Extensions For 1800Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1801 1802In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1803available for use for the AMD extension for access to the hardware GDS memory 1804which is scratchpad memory allocated per device. 1805 1806For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1807address class of ``DW_ADDR_none`` is used. 1808 1809See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1810mapping of DWARF address classes to DWARF address spaces, including address size 1811and NULL value. 1812 1813.. _amdgpu-dwarf-address-space-identifier: 1814 1815Address Space Identifier 1816------------------------ 1817 1818DWARF address spaces correspond to target architecture specific linear 1819addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1820For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1821 1822The DWARF address space mapping used for AMDGPU is defined in 1823:ref:`amdgpu-dwarf-address-space-mapping-table`. 1824 1825.. table:: AMDGPU DWARF Address Space Mapping 1826 :name: amdgpu-dwarf-address-space-mapping-table 1827 1828 ======================================= ===== ======= ======== ================= ======================= 1829 DWARF AMDGPU Notes 1830 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1831 Address Space Name Value Address Bit Size Address Space 1832 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1833 .. 64-bit 32-bit 1834 process process 1835 address address 1836 space space 1837 ======================================= ===== ======= ======== ================= ======================= 1838 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1839 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1840 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1841 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1842 *Reserved* 0x04 1843 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1844 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1845 ======================================= ===== ======= ======== ================= ======================= 1846 1847See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1848including address size and NULL value. 1849 1850The ``DW_ASPACE_none`` address space is the default target architecture address 1851space used in DWARF operations that do not specify an address space. It 1852therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1853related operations can refer to addresses in the program code. 1854 1855The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1856specify the flat address space. If the address corresponds to an address in the 1857local address space, then it corresponds to the wavefront that is executing the 1858focused thread of execution. If the address corresponds to an address in the 1859private address space, then it corresponds to the lane that is executing the 1860focused thread of execution for languages that are implemented using a SIMD or 1861SIMT execution model. 1862 1863.. note:: 1864 1865 CUDA-like languages such as HIP that do not have address spaces in the 1866 language type system, but do allow variables to be allocated in different 1867 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1868 address space in the DWARF expression operations as the default address space 1869 is the global address space. 1870 1871The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1872specify the local address space corresponding to the wavefront that is executing 1873the focused thread of execution. 1874 1875The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1876to specify the private address space corresponding to the lane that is executing 1877the focused thread of execution for languages that are implemented using a SIMD 1878or SIMT execution model. 1879 1880The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1881to specify the unswizzled private address space corresponding to the wavefront 1882that is executing the focused thread of execution. The wavefront view of private 1883memory is the per wavefront unswizzled backing memory layout defined in 1884:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1885location for the backing memory of the wavefront (namely the address is not 1886offset by ``wavefront-scratch-base``). The following formula can be used to 1887convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1888``DW_ASPACE_AMDGPU_private_wave`` address: 1889 1890:: 1891 1892 private-address-wavefront = 1893 ((private-address-lane / 4) * wavefront-size * 4) + 1894 (wavefront-lane-id * 4) + (private-address-lane % 4) 1895 1896If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1897of the dwords for each lane starting with lane 0 is required, then this 1898simplifies to: 1899 1900:: 1901 1902 private-address-wavefront = 1903 private-address-lane * wavefront-size 1904 1905A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1906complete spilled vector register back into a complete vector register in the 1907CFI. The frame pointer can be a private lane address which is dword aligned, 1908which can be shifted to multiply by the wavefront size, and then used to form a 1909private wavefront address that gives a location for a contiguous set of dwords, 1910one per lane, where the vector register dwords are spilled. The compiler knows 1911the wavefront size since it generates the code. Note that the type of the 1912address may have to be converted as the size of a 1913``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1914``DW_ASPACE_AMDGPU_private_wave`` address. 1915 1916.. _amdgpu-dwarf-lane-identifier: 1917 1918Lane identifier 1919--------------- 1920 1921DWARF lane identifies specify a target architecture lane position for hardware 1922that executes in a SIMD or SIMT manner, and on which a source language maps its 1923threads of execution onto those lanes. The DWARF lane identifier is pushed by 1924the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1925section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1926section :ref:`amdgpu-dwarf-operation-expressions`. 1927 1928For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1929wavefront. It is numbered from 0 to the wavefront size minus 1. 1930 1931Operation Expressions 1932--------------------- 1933 1934DWARF expressions are used to compute program values and the locations of 1935program objects. See DWARF Version 5 section 2.5 and 1936:ref:`amdgpu-dwarf-operation-expressions`. 1937 1938DWARF location descriptions describe how to access storage which includes memory 1939and registers. When accessing storage on AMDGPU, bytes are ordered with least 1940significant bytes first, and bits are ordered within bytes with least 1941significant bits first. 1942 1943For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1944unwinding vector registers that are spilled under the execution mask to memory: 1945the zero-single location description is the vector register, and the one-single 1946location description is the spilled memory location description. The 1947``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1948memory location description. 1949 1950In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1951``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1952controlled by the execution mask. An undefined location description together 1953with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1954to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1955 1956Debugger Information Entry Attributes 1957------------------------------------- 1958 1959This section describes how certain debugger information entry attributes are 1960used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1961by *DWARF Extensions For Heterogeneous Debugging* section 1962:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1963 1964.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1965 1966``DW_AT_LLVM_lane_pc`` 1967~~~~~~~~~~~~~~~~~~~~~~ 1968 1969For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1970location of the separate lanes of a SIMT thread. 1971 1972If the lane is an active lane then this will be the same as the current program 1973location. 1974 1975If the lane is inactive, but was active on entry to the subprogram, then this is 1976the program location in the subprogram at which execution of the lane is 1977conceptual positioned. 1978 1979If the lane was not active on entry to the subprogram, then this will be the 1980undefined location. A client debugger can check if the lane is part of a valid 1981work-group by checking that the lane is in the range of the associated 1982work-group within the grid, accounting for partial work-groups. If it is not, 1983then the debugger can omit any information for the lane. Otherwise, the debugger 1984may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1985calling subprogram until it finds a non-undefined location. Conceptually the 1986lane only has the call frames that it has a non-undefined 1987``DW_AT_LLVM_lane_pc``. 1988 1989The following example illustrates how the AMDGPU backend can generate a DWARF 1990location list expression for the nested ``IF/THEN/ELSE`` structures of the 1991following subprogram pseudo code for a target with 64 lanes per wavefront. 1992 1993.. code:: 1994 :number-lines: 1995 1996 SUBPROGRAM X 1997 BEGIN 1998 a; 1999 IF (c1) THEN 2000 b; 2001 IF (c2) THEN 2002 c; 2003 ELSE 2004 d; 2005 ENDIF 2006 e; 2007 ELSE 2008 f; 2009 ENDIF 2010 g; 2011 END 2012 2013The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2014execution mask (``EXEC``) to linearize the control flow. The condition is 2015evaluated to make a mask of the lanes for which the condition evaluates to true. 2016First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2017logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2018``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2019the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2020region the ``EXEC`` mask is restored to the value it had at the beginning of the 2021region. This is shown below. Other approaches are possible, but the basic 2022concept is the same. 2023 2024.. code:: 2025 :number-lines: 2026 2027 $lex_start: 2028 a; 2029 %1 = EXEC 2030 %2 = c1 2031 $lex_1_start: 2032 EXEC = %1 & %2 2033 $if_1_then: 2034 b; 2035 %3 = EXEC 2036 %4 = c2 2037 $lex_1_1_start: 2038 EXEC = %3 & %4 2039 $lex_1_1_then: 2040 c; 2041 EXEC = ~EXEC & %3 2042 $lex_1_1_else: 2043 d; 2044 EXEC = %3 2045 $lex_1_1_end: 2046 e; 2047 EXEC = ~EXEC & %1 2048 $lex_1_else: 2049 f; 2050 EXEC = %1 2051 $lex_1_end: 2052 g; 2053 $lex_end: 2054 2055To create the DWARF location list expression that defines the location 2056description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2057pseudo instruction can be used to annotate the linearized control flow. This can 2058be done by defining an artificial variable for the lane PC. The DWARF location 2059list expression created for it is used as the value of the 2060``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2061 2062A DWARF procedure is defined for each well nested structured control flow region 2063which provides the conceptual lane program location for a lane if it is not 2064active (namely it is divergent). The DWARF operation expression for each region 2065conceptually inherits the value of the immediately enclosing region and modifies 2066it according to the semantics of the region. 2067 2068For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2069the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2070region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2071region since the ``THEN`` region has completed. 2072 2073The lane PC artificial variable is assigned at each region transition. It uses 2074the immediately enclosing region's DWARF procedure to compute the program 2075location for each lane assuming they are divergent, and then modifies the result 2076by inserting the current program location for each lane that the ``EXEC`` mask 2077indicates is active. 2078 2079By having separate DWARF procedures for each region, they can be reused to 2080define the value for any nested region. This reduces the total size of the DWARF 2081operation expressions. 2082 2083The following provides an example using pseudo LLVM MIR. 2084 2085.. code:: 2086 :number-lines: 2087 2088 $lex_start: 2089 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2090 DW_AT_name = "__uint64"; 2091 DW_AT_byte_size = 8; 2092 DW_AT_encoding = DW_ATE_unsigned; 2093 ]; 2094 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2095 DW_AT_name = "__active_lane_pc"; 2096 DW_AT_location = [ 2097 DW_OP_regx PC; 2098 DW_OP_LLVM_extend 64, 64; 2099 DW_OP_regval_type EXEC, %uint_64; 2100 DW_OP_LLVM_select_bit_piece 64, 64; 2101 ]; 2102 ]; 2103 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2104 DW_AT_name = "__divergent_lane_pc"; 2105 DW_AT_location = [ 2106 DW_OP_LLVM_undefined; 2107 DW_OP_LLVM_extend 64, 64; 2108 ]; 2109 ]; 2110 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2111 DW_OP_call_ref %__divergent_lane_pc; 2112 DW_OP_call_ref %__active_lane_pc; 2113 ]; 2114 a; 2115 %1 = EXEC; 2116 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2117 %2 = c1; 2118 $lex_1_start: 2119 EXEC = %1 & %2; 2120 $lex_1_then: 2121 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2122 DW_AT_name = "__divergent_lane_pc_1_then"; 2123 DW_AT_location = DIExpression[ 2124 DW_OP_call_ref %__divergent_lane_pc; 2125 DW_OP_addrx &lex_1_start; 2126 DW_OP_stack_value; 2127 DW_OP_LLVM_extend 64, 64; 2128 DW_OP_call_ref %__lex_1_save_exec; 2129 DW_OP_deref_type 64, %__uint_64; 2130 DW_OP_LLVM_select_bit_piece 64, 64; 2131 ]; 2132 ]; 2133 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2134 DW_OP_call_ref %__divergent_lane_pc_1_then; 2135 DW_OP_call_ref %__active_lane_pc; 2136 ]; 2137 b; 2138 %3 = EXEC; 2139 DBG_VALUE %3, %__lex_1_1_save_exec; 2140 %4 = c2; 2141 $lex_1_1_start: 2142 EXEC = %3 & %4; 2143 $lex_1_1_then: 2144 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2145 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2146 DW_AT_location = DIExpression[ 2147 DW_OP_call_ref %__divergent_lane_pc_1_then; 2148 DW_OP_addrx &lex_1_1_start; 2149 DW_OP_stack_value; 2150 DW_OP_LLVM_extend 64, 64; 2151 DW_OP_call_ref %__lex_1_1_save_exec; 2152 DW_OP_deref_type 64, %__uint_64; 2153 DW_OP_LLVM_select_bit_piece 64, 64; 2154 ]; 2155 ]; 2156 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2157 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2158 DW_OP_call_ref %__active_lane_pc; 2159 ]; 2160 c; 2161 EXEC = ~EXEC & %3; 2162 $lex_1_1_else: 2163 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2164 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2165 DW_AT_location = DIExpression[ 2166 DW_OP_call_ref %__divergent_lane_pc_1_then; 2167 DW_OP_addrx &lex_1_1_end; 2168 DW_OP_stack_value; 2169 DW_OP_LLVM_extend 64, 64; 2170 DW_OP_call_ref %__lex_1_1_save_exec; 2171 DW_OP_deref_type 64, %__uint_64; 2172 DW_OP_LLVM_select_bit_piece 64, 64; 2173 ]; 2174 ]; 2175 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2176 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2177 DW_OP_call_ref %__active_lane_pc; 2178 ]; 2179 d; 2180 EXEC = %3; 2181 $lex_1_1_end: 2182 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2183 DW_OP_call_ref %__divergent_lane_pc; 2184 DW_OP_call_ref %__active_lane_pc; 2185 ]; 2186 e; 2187 EXEC = ~EXEC & %1; 2188 $lex_1_else: 2189 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2190 DW_AT_name = "__divergent_lane_pc_1_else"; 2191 DW_AT_location = DIExpression[ 2192 DW_OP_call_ref %__divergent_lane_pc; 2193 DW_OP_addrx &lex_1_end; 2194 DW_OP_stack_value; 2195 DW_OP_LLVM_extend 64, 64; 2196 DW_OP_call_ref %__lex_1_save_exec; 2197 DW_OP_deref_type 64, %__uint_64; 2198 DW_OP_LLVM_select_bit_piece 64, 64; 2199 ]; 2200 ]; 2201 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2202 DW_OP_call_ref %__divergent_lane_pc_1_else; 2203 DW_OP_call_ref %__active_lane_pc; 2204 ]; 2205 f; 2206 EXEC = %1; 2207 $lex_1_end: 2208 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2209 DW_OP_call_ref %__divergent_lane_pc; 2210 DW_OP_call_ref %__active_lane_pc; 2211 ]; 2212 g; 2213 $lex_end: 2214 2215The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2216that are active, with the current program location. 2217 2218Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2219the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2220instruction, location list entries will be created that describe where the 2221artificial variables are allocated at any given program location. The compiler 2222may allocate them to registers or spill them to memory. 2223 2224The DWARF procedures for each region use the values of the saved execution mask 2225artificial variables to only update the lanes that are active on entry to the 2226region. All other lanes retain the value of the enclosing region where they were 2227last active. If they were not active on entry to the subprogram, then will have 2228the undefined location description. 2229 2230Other structured control flow regions can be handled similarly. For example, 2231loops would set the divergent program location for the region at the end of the 2232loop. Any lanes active will be in the loop, and any lanes not active must have 2233exited the loop. 2234 2235An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2236``IF/THEN/ELSE`` regions. 2237 2238The DWARF procedures can use the active lane artificial variable described in 2239:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2240``EXEC`` mask in order to support whole or quad wavefront mode. 2241 2242.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2243 2244``DW_AT_LLVM_active_lane`` 2245~~~~~~~~~~~~~~~~~~~~~~~~~~ 2246 2247The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2248entry is used to specify the lanes that are conceptually active for a SIMT 2249thread. 2250 2251The execution mask may be modified to implement whole or quad wavefront mode 2252operations. For example, all lanes may need to temporarily be made active to 2253execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2254update it to enable the necessary lanes, perform the operations, and then 2255restore the ``EXEC`` mask from the saved value. While executing the whole 2256wavefront region, the conceptual execution mask is the saved value, not the 2257``EXEC`` value. 2258 2259This is handled by defining an artificial variable for the active lane mask. The 2260active lane mask artificial variable would be the actual ``EXEC`` mask for 2261normal regions, and the saved execution mask for regions where the mask is 2262temporarily updated. The location list expression created for this artificial 2263variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2264attribute. 2265 2266``DW_AT_LLVM_augmentation`` 2267~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2268 2269For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2270debugger information entry has the following value for the augmentation string: 2271 2272:: 2273 2274 [amdgpu:v0.0] 2275 2276The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2277extensions used in the DWARF of the compilation unit. The version number 2278conforms to [SEMVER]_. 2279 2280Call Frame Information 2281---------------------- 2282 2283DWARF Call Frame Information (CFI) describes how a consumer can virtually 2284*unwind* call frames in a running process or core dump. See DWARF Version 5 2285section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2286 2287For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2288 22891. ``augmentation`` string contains the following null-terminated UTF-8 string: 2290 2291 :: 2292 2293 [amd:v0.0] 2294 2295 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2296 extensions used in this CIE or to the FDEs that use it. The version number 2297 conforms to [SEMVER]_. 2298 22992. ``address_size`` for the ``Global`` address space is defined in 2300 :ref:`amdgpu-dwarf-address-space-identifier`. 2301 23023. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2303 23044. ``code_alignment_factor`` is 4 bytes. 2305 2306 .. TODO:: 2307 2308 Add to :ref:`amdgpu-processor-table` table. 2309 23105. ``data_alignment_factor`` is 4 bytes. 2311 2312 .. TODO:: 2313 2314 Add to :ref:`amdgpu-processor-table` table. 2315 23166. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2317 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2318 23197. ``initial_instructions`` Since a subprogram X with fewer registers can be 2320 called from subprogram Y that has more allocated, X will not change any of 2321 the extra registers as it cannot access them. Therefore, the default rule 2322 for all columns is ``same value``. 2323 2324For AMDGPU the register number follows the numbering defined in 2325:ref:`amdgpu-dwarf-register-identifier`. 2326 2327For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2328the return address to get the address of a byte within the call site 2329instructions. See DWARF Version 5 section 6.4.4. 2330 2331Accelerated Access 2332------------------ 2333 2334See DWARF Version 5 section 6.1. 2335 2336Lookup By Name Section Header 2337~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2338 2339See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2340 2341For AMDGPU the lookup by name section header table: 2342 2343``augmentation_string_size`` (uword) 2344 2345 Set to the length of the ``augmentation_string`` value which is always a 2346 multiple of 4. 2347 2348``augmentation_string`` (sequence of UTF-8 characters) 2349 2350 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2351 2352 :: 2353 2354 [amdgpu:v0.0] 2355 2356 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2357 extensions used in the DWARF of this index. The version number conforms to 2358 [SEMVER]_. 2359 2360 .. note:: 2361 2362 This is different to the DWARF Version 5 definition that requires the first 2363 4 characters to be the vendor ID. But this is consistent with the other 2364 augmentation strings and does allow multiple vendor contributions. However, 2365 backwards compatibility may be more desirable. 2366 2367Lookup By Address Section Header 2368~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2369 2370See DWARF Version 5 section 6.1.2. 2371 2372For AMDGPU the lookup by address section header table: 2373 2374``address_size`` (ubyte) 2375 2376 Match the address size for the ``Global`` address space defined in 2377 :ref:`amdgpu-dwarf-address-space-identifier`. 2378 2379``segment_selector_size`` (ubyte) 2380 2381 AMDGPU does not use a segment selector so this is 0. The entries in the 2382 ``.debug_aranges`` do not have a segment selector. 2383 2384Line Number Information 2385----------------------- 2386 2387See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2388 2389AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2390The instruction set must be obtained from the ELF file header ``e_flags`` field 2391in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2392<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2393 2394.. TODO:: 2395 2396 Should the ``isa`` state machine register be used to indicate if the code is 2397 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2398 2399For AMDGPU the line number program header fields have the following values (see 2400DWARF Version 5 section 6.2.4): 2401 2402``address_size`` (ubyte) 2403 Matches the address size for the ``Global`` address space defined in 2404 :ref:`amdgpu-dwarf-address-space-identifier`. 2405 2406``segment_selector_size`` (ubyte) 2407 AMDGPU does not use a segment selector so this is 0. 2408 2409``minimum_instruction_length`` (ubyte) 2410 For GFX9-GFX10 this is 4. 2411 2412``maximum_operations_per_instruction`` (ubyte) 2413 For GFX9-GFX10 this is 1. 2414 2415Source text for online-compiled programs (for example, those compiled by the 2416OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2417See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2418Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2419<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2420 2421The Clang option used to control source embedding in AMDGPU is defined in 2422:ref:`amdgpu-clang-debug-options-table`. 2423 2424 .. table:: AMDGPU Clang Debug Options 2425 :name: amdgpu-clang-debug-options-table 2426 2427 ==================== ================================================== 2428 Debug Flag Description 2429 ==================== ================================================== 2430 -g[no-]embed-source Enable/disable embedding source text in DWARF 2431 debug sections. Useful for environments where 2432 source cannot be written to disk, such as 2433 when performing online compilation. 2434 ==================== ================================================== 2435 2436For example: 2437 2438``-gembed-source`` 2439 Enable the embedded source. 2440 2441``-gno-embed-source`` 2442 Disable the embedded source. 2443 244432-Bit and 64-Bit DWARF Formats 2445------------------------------- 2446 2447See DWARF Version 5 section 7.4 and 2448:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2449 2450For AMDGPU: 2451 2452* For the ``amdgcn`` target architecture only the 64-bit process address space 2453 is supported. 2454 2455* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2456 the 32-bit DWARF format. 2457 2458Unit Headers 2459------------ 2460 2461For AMDGPU the following values apply for each of the unit headers described in 2462DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2463 2464``address_size`` (ubyte) 2465 Matches the address size for the ``Global`` address space defined in 2466 :ref:`amdgpu-dwarf-address-space-identifier`. 2467 2468.. _amdgpu-code-conventions: 2469 2470Code Conventions 2471================ 2472 2473This section provides code conventions used for each supported target triple OS 2474(see :ref:`amdgpu-target-triples`). 2475 2476AMDHSA 2477------ 2478 2479This section provides code conventions used when the target triple OS is 2480``amdhsa`` (see :ref:`amdgpu-target-triples`). 2481 2482.. _amdgpu-amdhsa-code-object-metadata: 2483 2484Code Object Metadata 2485~~~~~~~~~~~~~~~~~~~~ 2486 2487The code object metadata specifies extensible metadata associated with the code 2488objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2489encoding and semantics of this metadata depends on the code object version; see 2490:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2491:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2492:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2493 2494Code object metadata is specified in a note record (see 2495:ref:`amdgpu-note-records`) and is required when the target triple OS is 2496``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2497information necessary to support the HSA compatible runtime kernel queries. For 2498example, the segment sizes needed in a dispatch packet. In addition, a 2499high-level language runtime may require other information to be included. For 2500example, the AMD OpenCL runtime records kernel argument information. 2501 2502.. _amdgpu-amdhsa-code-object-metadata-v2: 2503 2504Code Object V2 Metadata 2505+++++++++++++++++++++++ 2506 2507.. warning:: 2508 Code object V2 is not the default code object version emitted by this version 2509 of LLVM. 2510 2511Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2512(see :ref:`amdgpu-note-records-v2`). 2513 2514The metadata is specified as a YAML formatted string (see [YAML]_ and 2515:doc:`YamlIO`). 2516 2517.. TODO:: 2518 2519 Is the string null terminated? It probably should not if YAML allows it to 2520 contain null characters, otherwise it should be. 2521 2522The metadata is represented as a single YAML document comprised of the mapping 2523defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2524referenced tables. 2525 2526For boolean values, the string values of ``false`` and ``true`` are used for 2527false and true respectively. 2528 2529Additional information can be added to the mappings. To avoid conflicts, any 2530non-AMD key names should be prefixed by "*vendor-name*.". 2531 2532 .. table:: AMDHSA Code Object V2 Metadata Map 2533 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2534 2535 ========== ============== ========= ======================================= 2536 String Key Value Type Required? Description 2537 ========== ============== ========= ======================================= 2538 "Version" sequence of Required - The first integer is the major 2539 2 integers version. Currently 1. 2540 - The second integer is the minor 2541 version. Currently 0. 2542 "Printf" sequence of Each string is encoded information 2543 strings about a printf function call. The 2544 encoded information is organized as 2545 fields separated by colon (':'): 2546 2547 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2548 2549 where: 2550 2551 ``ID`` 2552 A 32-bit integer as a unique id for 2553 each printf function call 2554 2555 ``N`` 2556 A 32-bit integer equal to the number 2557 of arguments of printf function call 2558 minus 1 2559 2560 ``S[i]`` (where i = 0, 1, ... , N-1) 2561 32-bit integers for the size in bytes 2562 of the i-th FormatString argument of 2563 the printf function call 2564 2565 FormatString 2566 The format string passed to the 2567 printf function call. 2568 "Kernels" sequence of Required Sequence of the mappings for each 2569 mapping kernel in the code object. See 2570 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2571 for the definition of the mapping. 2572 ========== ============== ========= ======================================= 2573 2574.. 2575 2576 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2577 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2578 2579 ================= ============== ========= ================================ 2580 String Key Value Type Required? Description 2581 ================= ============== ========= ================================ 2582 "Name" string Required Source name of the kernel. 2583 "SymbolName" string Required Name of the kernel 2584 descriptor ELF symbol. 2585 "Language" string Source language of the kernel. 2586 Values include: 2587 2588 - "OpenCL C" 2589 - "OpenCL C++" 2590 - "HCC" 2591 - "OpenMP" 2592 2593 "LanguageVersion" sequence of - The first integer is the major 2594 2 integers version. 2595 - The second integer is the 2596 minor version. 2597 "Attrs" mapping Mapping of kernel attributes. 2598 See 2599 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2600 for the mapping definition. 2601 "Args" sequence of Sequence of mappings of the 2602 mapping kernel arguments. See 2603 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2604 for the definition of the mapping. 2605 "CodeProps" mapping Mapping of properties related to 2606 the kernel code. See 2607 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2608 for the mapping definition. 2609 ================= ============== ========= ================================ 2610 2611.. 2612 2613 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2614 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2615 2616 =================== ============== ========= ============================== 2617 String Key Value Type Required? Description 2618 =================== ============== ========= ============================== 2619 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2620 3 integers must be >=1 and the dispatch 2621 work-group size X, Y, Z must 2622 correspond to the specified 2623 values. Defaults to 0, 0, 0. 2624 2625 Corresponds to the OpenCL 2626 ``reqd_work_group_size`` 2627 attribute. 2628 "WorkGroupSizeHint" sequence of The dispatch work-group size 2629 3 integers X, Y, Z is likely to be the 2630 specified values. 2631 2632 Corresponds to the OpenCL 2633 ``work_group_size_hint`` 2634 attribute. 2635 "VecTypeHint" string The name of a scalar or vector 2636 type. 2637 2638 Corresponds to the OpenCL 2639 ``vec_type_hint`` attribute. 2640 2641 "RuntimeHandle" string The external symbol name 2642 associated with a kernel. 2643 OpenCL runtime allocates a 2644 global buffer for the symbol 2645 and saves the kernel's address 2646 to it, which is used for 2647 device side enqueueing. Only 2648 available for device side 2649 enqueued kernels. 2650 =================== ============== ========= ============================== 2651 2652.. 2653 2654 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2655 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2656 2657 ================= ============== ========= ================================ 2658 String Key Value Type Required? Description 2659 ================= ============== ========= ================================ 2660 "Name" string Kernel argument name. 2661 "TypeName" string Kernel argument type name. 2662 "Size" integer Required Kernel argument size in bytes. 2663 "Align" integer Required Kernel argument alignment in 2664 bytes. Must be a power of two. 2665 "ValueKind" string Required Kernel argument kind that 2666 specifies how to set up the 2667 corresponding argument. 2668 Values include: 2669 2670 "ByValue" 2671 The argument is copied 2672 directly into the kernarg. 2673 2674 "GlobalBuffer" 2675 A global address space pointer 2676 to the buffer data is passed 2677 in the kernarg. 2678 2679 "DynamicSharedPointer" 2680 A group address space pointer 2681 to dynamically allocated LDS 2682 is passed in the kernarg. 2683 2684 "Sampler" 2685 A global address space 2686 pointer to a S# is passed in 2687 the kernarg. 2688 2689 "Image" 2690 A global address space 2691 pointer to a T# is passed in 2692 the kernarg. 2693 2694 "Pipe" 2695 A global address space pointer 2696 to an OpenCL pipe is passed in 2697 the kernarg. 2698 2699 "Queue" 2700 A global address space pointer 2701 to an OpenCL device enqueue 2702 queue is passed in the 2703 kernarg. 2704 2705 "HiddenGlobalOffsetX" 2706 The OpenCL grid dispatch 2707 global offset for the X 2708 dimension is passed in the 2709 kernarg. 2710 2711 "HiddenGlobalOffsetY" 2712 The OpenCL grid dispatch 2713 global offset for the Y 2714 dimension is passed in the 2715 kernarg. 2716 2717 "HiddenGlobalOffsetZ" 2718 The OpenCL grid dispatch 2719 global offset for the Z 2720 dimension is passed in the 2721 kernarg. 2722 2723 "HiddenNone" 2724 An argument that is not used 2725 by the kernel. Space needs to 2726 be left for it, but it does 2727 not need to be set up. 2728 2729 "HiddenPrintfBuffer" 2730 A global address space pointer 2731 to the runtime printf buffer 2732 is passed in kernarg. 2733 2734 "HiddenHostcallBuffer" 2735 A global address space pointer 2736 to the runtime hostcall buffer 2737 is passed in kernarg. 2738 2739 "HiddenDefaultQueue" 2740 A global address space pointer 2741 to the OpenCL device enqueue 2742 queue that should be used by 2743 the kernel by default is 2744 passed in the kernarg. 2745 2746 "HiddenCompletionAction" 2747 A global address space pointer 2748 to help link enqueued kernels into 2749 the ancestor tree for determining 2750 when the parent kernel has finished. 2751 2752 "HiddenMultiGridSyncArg" 2753 A global address space pointer for 2754 multi-grid synchronization is 2755 passed in the kernarg. 2756 2757 "ValueType" string Unused and deprecated. This should no longer 2758 be emitted, but is accepted for compatibility. 2759 2760 2761 "PointeeAlign" integer Alignment in bytes of pointee 2762 type for pointer type kernel 2763 argument. Must be a power 2764 of 2. Only present if 2765 "ValueKind" is 2766 "DynamicSharedPointer". 2767 "AddrSpaceQual" string Kernel argument address space 2768 qualifier. Only present if 2769 "ValueKind" is "GlobalBuffer" or 2770 "DynamicSharedPointer". Values 2771 are: 2772 2773 - "Private" 2774 - "Global" 2775 - "Constant" 2776 - "Local" 2777 - "Generic" 2778 - "Region" 2779 2780 .. TODO:: 2781 2782 Is GlobalBuffer only Global 2783 or Constant? Is 2784 DynamicSharedPointer always 2785 Local? Can HCC allow Generic? 2786 How can Private or Region 2787 ever happen? 2788 2789 "AccQual" string Kernel argument access 2790 qualifier. Only present if 2791 "ValueKind" is "Image" or 2792 "Pipe". Values 2793 are: 2794 2795 - "ReadOnly" 2796 - "WriteOnly" 2797 - "ReadWrite" 2798 2799 .. TODO:: 2800 2801 Does this apply to 2802 GlobalBuffer? 2803 2804 "ActualAccQual" string The actual memory accesses 2805 performed by the kernel on the 2806 kernel argument. Only present if 2807 "ValueKind" is "GlobalBuffer", 2808 "Image", or "Pipe". This may be 2809 more restrictive than indicated 2810 by "AccQual" to reflect what the 2811 kernel actual does. If not 2812 present then the runtime must 2813 assume what is implied by 2814 "AccQual" and "IsConst". Values 2815 are: 2816 2817 - "ReadOnly" 2818 - "WriteOnly" 2819 - "ReadWrite" 2820 2821 "IsConst" boolean Indicates if the kernel argument 2822 is const qualified. Only present 2823 if "ValueKind" is 2824 "GlobalBuffer". 2825 2826 "IsRestrict" boolean Indicates if the kernel argument 2827 is restrict qualified. Only 2828 present if "ValueKind" is 2829 "GlobalBuffer". 2830 2831 "IsVolatile" boolean Indicates if the kernel argument 2832 is volatile qualified. Only 2833 present if "ValueKind" is 2834 "GlobalBuffer". 2835 2836 "IsPipe" boolean Indicates if the kernel argument 2837 is pipe qualified. Only present 2838 if "ValueKind" is "Pipe". 2839 2840 .. TODO:: 2841 2842 Can GlobalBuffer be pipe 2843 qualified? 2844 2845 ================= ============== ========= ================================ 2846 2847.. 2848 2849 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2850 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2851 2852 ============================ ============== ========= ===================== 2853 String Key Value Type Required? Description 2854 ============================ ============== ========= ===================== 2855 "KernargSegmentSize" integer Required The size in bytes of 2856 the kernarg segment 2857 that holds the values 2858 of the arguments to 2859 the kernel. 2860 "GroupSegmentFixedSize" integer Required The amount of group 2861 segment memory 2862 required by a 2863 work-group in 2864 bytes. This does not 2865 include any 2866 dynamically allocated 2867 group segment memory 2868 that may be added 2869 when the kernel is 2870 dispatched. 2871 "PrivateSegmentFixedSize" integer Required The amount of fixed 2872 private address space 2873 memory required for a 2874 work-item in 2875 bytes. If the kernel 2876 uses a dynamic call 2877 stack then additional 2878 space must be added 2879 to this value for the 2880 call stack. 2881 "KernargSegmentAlign" integer Required The maximum byte 2882 alignment of 2883 arguments in the 2884 kernarg segment. Must 2885 be a power of 2. 2886 "WavefrontSize" integer Required Wavefront size. Must 2887 be a power of 2. 2888 "NumSGPRs" integer Required Number of scalar 2889 registers used by a 2890 wavefront for 2891 GFX6-GFX10. This 2892 includes the special 2893 SGPRs for VCC, Flat 2894 Scratch (GFX7-GFX10) 2895 and XNACK (for 2896 GFX8-GFX10). It does 2897 not include the 16 2898 SGPR added if a trap 2899 handler is 2900 enabled. It is not 2901 rounded up to the 2902 allocation 2903 granularity. 2904 "NumVGPRs" integer Required Number of vector 2905 registers used by 2906 each work-item for 2907 GFX6-GFX10 2908 "MaxFlatWorkGroupSize" integer Required Maximum flat 2909 work-group size 2910 supported by the 2911 kernel in work-items. 2912 Must be >=1 and 2913 consistent with 2914 ReqdWorkGroupSize if 2915 not 0, 0, 0. 2916 "NumSpilledSGPRs" integer Number of stores from 2917 a scalar register to 2918 a register allocator 2919 created spill 2920 location. 2921 "NumSpilledVGPRs" integer Number of stores from 2922 a vector register to 2923 a register allocator 2924 created spill 2925 location. 2926 ============================ ============== ========= ===================== 2927 2928.. _amdgpu-amdhsa-code-object-metadata-v3: 2929 2930Code Object V3 Metadata 2931+++++++++++++++++++++++ 2932 2933Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2934record (see :ref:`amdgpu-note-records-v3-v4`). 2935 2936The metadata is represented as Message Pack formatted binary data (see 2937[MsgPack]_). The top level is a Message Pack map that includes the 2938keys defined in table 2939:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2940tables. 2941 2942Additional information can be added to the maps. To avoid conflicts, 2943any key names should be prefixed by "*vendor-name*." where 2944``vendor-name`` can be the name of the vendor and specific vendor 2945tool that generates the information. The prefix is abbreviated to 2946simply "." when it appears within a map that has been added by the 2947same *vendor-name*. 2948 2949 .. table:: AMDHSA Code Object V3 Metadata Map 2950 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2951 2952 ================= ============== ========= ======================================= 2953 String Key Value Type Required? Description 2954 ================= ============== ========= ======================================= 2955 "amdhsa.version" sequence of Required - The first integer is the major 2956 2 integers version. Currently 1. 2957 - The second integer is the minor 2958 version. Currently 0. 2959 "amdhsa.printf" sequence of Each string is encoded information 2960 strings about a printf function call. The 2961 encoded information is organized as 2962 fields separated by colon (':'): 2963 2964 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2965 2966 where: 2967 2968 ``ID`` 2969 A 32-bit integer as a unique id for 2970 each printf function call 2971 2972 ``N`` 2973 A 32-bit integer equal to the number 2974 of arguments of printf function call 2975 minus 1 2976 2977 ``S[i]`` (where i = 0, 1, ... , N-1) 2978 32-bit integers for the size in bytes 2979 of the i-th FormatString argument of 2980 the printf function call 2981 2982 FormatString 2983 The format string passed to the 2984 printf function call. 2985 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2986 map kernel in the code object. See 2987 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2988 for the definition of the keys included 2989 in that map. 2990 ================= ============== ========= ======================================= 2991 2992.. 2993 2994 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2995 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2996 2997 =================================== ============== ========= ================================ 2998 String Key Value Type Required? Description 2999 =================================== ============== ========= ================================ 3000 ".name" string Required Source name of the kernel. 3001 ".symbol" string Required Name of the kernel 3002 descriptor ELF symbol. 3003 ".language" string Source language of the kernel. 3004 Values include: 3005 3006 - "OpenCL C" 3007 - "OpenCL C++" 3008 - "HCC" 3009 - "HIP" 3010 - "OpenMP" 3011 - "Assembler" 3012 3013 ".language_version" sequence of - The first integer is the major 3014 2 integers version. 3015 - The second integer is the 3016 minor version. 3017 ".args" sequence of Sequence of maps of the 3018 map kernel arguments. See 3019 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3020 for the definition of the keys 3021 included in that map. 3022 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3023 3 integers must be >=1 and the dispatch 3024 work-group size X, Y, Z must 3025 correspond to the specified 3026 values. Defaults to 0, 0, 0. 3027 3028 Corresponds to the OpenCL 3029 ``reqd_work_group_size`` 3030 attribute. 3031 ".workgroup_size_hint" sequence of The dispatch work-group size 3032 3 integers X, Y, Z is likely to be the 3033 specified values. 3034 3035 Corresponds to the OpenCL 3036 ``work_group_size_hint`` 3037 attribute. 3038 ".vec_type_hint" string The name of a scalar or vector 3039 type. 3040 3041 Corresponds to the OpenCL 3042 ``vec_type_hint`` attribute. 3043 3044 ".device_enqueue_symbol" string The external symbol name 3045 associated with a kernel. 3046 OpenCL runtime allocates a 3047 global buffer for the symbol 3048 and saves the kernel's address 3049 to it, which is used for 3050 device side enqueueing. Only 3051 available for device side 3052 enqueued kernels. 3053 ".kernarg_segment_size" integer Required The size in bytes of 3054 the kernarg segment 3055 that holds the values 3056 of the arguments to 3057 the kernel. 3058 ".group_segment_fixed_size" integer Required The amount of group 3059 segment memory 3060 required by a 3061 work-group in 3062 bytes. This does not 3063 include any 3064 dynamically allocated 3065 group segment memory 3066 that may be added 3067 when the kernel is 3068 dispatched. 3069 ".private_segment_fixed_size" integer Required The amount of fixed 3070 private address space 3071 memory required for a 3072 work-item in 3073 bytes. If the kernel 3074 uses a dynamic call 3075 stack then additional 3076 space must be added 3077 to this value for the 3078 call stack. 3079 ".kernarg_segment_align" integer Required The maximum byte 3080 alignment of 3081 arguments in the 3082 kernarg segment. Must 3083 be a power of 2. 3084 ".wavefront_size" integer Required Wavefront size. Must 3085 be a power of 2. 3086 ".sgpr_count" integer Required Number of scalar 3087 registers required by a 3088 wavefront for 3089 GFX6-GFX9. A register 3090 is required if it is 3091 used explicitly, or 3092 if a higher numbered 3093 register is used 3094 explicitly. This 3095 includes the special 3096 SGPRs for VCC, Flat 3097 Scratch (GFX7-GFX9) 3098 and XNACK (for 3099 GFX8-GFX9). It does 3100 not include the 16 3101 SGPR added if a trap 3102 handler is 3103 enabled. It is not 3104 rounded up to the 3105 allocation 3106 granularity. 3107 ".vgpr_count" integer Required Number of vector 3108 registers required by 3109 each work-item for 3110 GFX6-GFX9. A register 3111 is required if it is 3112 used explicitly, or 3113 if a higher numbered 3114 register is used 3115 explicitly. 3116 ".max_flat_workgroup_size" integer Required Maximum flat 3117 work-group size 3118 supported by the 3119 kernel in work-items. 3120 Must be >=1 and 3121 consistent with 3122 ReqdWorkGroupSize if 3123 not 0, 0, 0. 3124 ".sgpr_spill_count" integer Number of stores from 3125 a scalar register to 3126 a register allocator 3127 created spill 3128 location. 3129 ".vgpr_spill_count" integer Number of stores from 3130 a vector register to 3131 a register allocator 3132 created spill 3133 location. 3134 =================================== ============== ========= ================================ 3135 3136.. 3137 3138 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3139 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3140 3141 ====================== ============== ========= ================================ 3142 String Key Value Type Required? Description 3143 ====================== ============== ========= ================================ 3144 ".name" string Kernel argument name. 3145 ".type_name" string Kernel argument type name. 3146 ".size" integer Required Kernel argument size in bytes. 3147 ".offset" integer Required Kernel argument offset in 3148 bytes. The offset must be a 3149 multiple of the alignment 3150 required by the argument. 3151 ".value_kind" string Required Kernel argument kind that 3152 specifies how to set up the 3153 corresponding argument. 3154 Values include: 3155 3156 "by_value" 3157 The argument is copied 3158 directly into the kernarg. 3159 3160 "global_buffer" 3161 A global address space pointer 3162 to the buffer data is passed 3163 in the kernarg. 3164 3165 "dynamic_shared_pointer" 3166 A group address space pointer 3167 to dynamically allocated LDS 3168 is passed in the kernarg. 3169 3170 "sampler" 3171 A global address space 3172 pointer to a S# is passed in 3173 the kernarg. 3174 3175 "image" 3176 A global address space 3177 pointer to a T# is passed in 3178 the kernarg. 3179 3180 "pipe" 3181 A global address space pointer 3182 to an OpenCL pipe is passed in 3183 the kernarg. 3184 3185 "queue" 3186 A global address space pointer 3187 to an OpenCL device enqueue 3188 queue is passed in the 3189 kernarg. 3190 3191 "hidden_global_offset_x" 3192 The OpenCL grid dispatch 3193 global offset for the X 3194 dimension is passed in the 3195 kernarg. 3196 3197 "hidden_global_offset_y" 3198 The OpenCL grid dispatch 3199 global offset for the Y 3200 dimension is passed in the 3201 kernarg. 3202 3203 "hidden_global_offset_z" 3204 The OpenCL grid dispatch 3205 global offset for the Z 3206 dimension is passed in the 3207 kernarg. 3208 3209 "hidden_none" 3210 An argument that is not used 3211 by the kernel. Space needs to 3212 be left for it, but it does 3213 not need to be set up. 3214 3215 "hidden_printf_buffer" 3216 A global address space pointer 3217 to the runtime printf buffer 3218 is passed in kernarg. 3219 3220 "hidden_hostcall_buffer" 3221 A global address space pointer 3222 to the runtime hostcall buffer 3223 is passed in kernarg. 3224 3225 "hidden_default_queue" 3226 A global address space pointer 3227 to the OpenCL device enqueue 3228 queue that should be used by 3229 the kernel by default is 3230 passed in the kernarg. 3231 3232 "hidden_completion_action" 3233 A global address space pointer 3234 to help link enqueued kernels into 3235 the ancestor tree for determining 3236 when the parent kernel has finished. 3237 3238 "hidden_multigrid_sync_arg" 3239 A global address space pointer for 3240 multi-grid synchronization is 3241 passed in the kernarg. 3242 3243 ".value_type" string Unused and deprecated. This should no longer 3244 be emitted, but is accepted for compatibility. 3245 3246 ".pointee_align" integer Alignment in bytes of pointee 3247 type for pointer type kernel 3248 argument. Must be a power 3249 of 2. Only present if 3250 ".value_kind" is 3251 "dynamic_shared_pointer". 3252 ".address_space" string Kernel argument address space 3253 qualifier. Only present if 3254 ".value_kind" is "global_buffer" or 3255 "dynamic_shared_pointer". Values 3256 are: 3257 3258 - "private" 3259 - "global" 3260 - "constant" 3261 - "local" 3262 - "generic" 3263 - "region" 3264 3265 .. TODO:: 3266 3267 Is "global_buffer" only "global" 3268 or "constant"? Is 3269 "dynamic_shared_pointer" always 3270 "local"? Can HCC allow "generic"? 3271 How can "private" or "region" 3272 ever happen? 3273 3274 ".access" string Kernel argument access 3275 qualifier. Only present if 3276 ".value_kind" is "image" or 3277 "pipe". Values 3278 are: 3279 3280 - "read_only" 3281 - "write_only" 3282 - "read_write" 3283 3284 .. TODO:: 3285 3286 Does this apply to 3287 "global_buffer"? 3288 3289 ".actual_access" string The actual memory accesses 3290 performed by the kernel on the 3291 kernel argument. Only present if 3292 ".value_kind" is "global_buffer", 3293 "image", or "pipe". This may be 3294 more restrictive than indicated 3295 by ".access" to reflect what the 3296 kernel actual does. If not 3297 present then the runtime must 3298 assume what is implied by 3299 ".access" and ".is_const" . Values 3300 are: 3301 3302 - "read_only" 3303 - "write_only" 3304 - "read_write" 3305 3306 ".is_const" boolean Indicates if the kernel argument 3307 is const qualified. Only present 3308 if ".value_kind" is 3309 "global_buffer". 3310 3311 ".is_restrict" boolean Indicates if the kernel argument 3312 is restrict qualified. Only 3313 present if ".value_kind" is 3314 "global_buffer". 3315 3316 ".is_volatile" boolean Indicates if the kernel argument 3317 is volatile qualified. Only 3318 present if ".value_kind" is 3319 "global_buffer". 3320 3321 ".is_pipe" boolean Indicates if the kernel argument 3322 is pipe qualified. Only present 3323 if ".value_kind" is "pipe". 3324 3325 .. TODO:: 3326 3327 Can "global_buffer" be pipe 3328 qualified? 3329 3330 ====================== ============== ========= ================================ 3331 3332.. _amdgpu-amdhsa-code-object-metadata-v4: 3333 3334Code Object V4 Metadata 3335+++++++++++++++++++++++ 3336 3337.. warning:: 3338 Code object V4 is not the default code object version emitted by this version 3339 of LLVM. 3340 3341Code object V4 metadata is the same as 3342:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3343defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3344 3345 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3346 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3347 3348 ================= ============== ========= ======================================= 3349 String Key Value Type Required? Description 3350 ================= ============== ========= ======================================= 3351 "amdhsa.version" sequence of Required - The first integer is the major 3352 2 integers version. Currently 1. 3353 - The second integer is the minor 3354 version. Currently 1. 3355 "amdhsa.target" string Required The target name of the code using the syntax: 3356 3357 .. code:: 3358 3359 <target-triple> [ "-" <target-id> ] 3360 3361 A canonical target ID must be 3362 used. See :ref:`amdgpu-target-triples` 3363 and :ref:`amdgpu-target-id`. 3364 ================= ============== ========= ======================================= 3365 3366.. 3367 3368Kernel Dispatch 3369~~~~~~~~~~~~~~~ 3370 3371The HSA architected queuing language (AQL) defines a user space memory interface 3372that can be used to control the dispatch of kernels, in an agent independent 3373way. An agent can have zero or more AQL queues created for it using an HSA 3374compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3375are 64 bytes) can be placed. See the *HSA Platform System Architecture 3376Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3377 3378The packet processor of a kernel agent is responsible for detecting and 3379dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3380packet processor is implemented by the hardware command processor (CP), 3381asynchronous dispatch controller (ADC) and shader processor input controller 3382(SPI). 3383 3384An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3385the kernel mode driver to initialize and register the AQL queue with CP. 3386 3387To dispatch a kernel the following actions are performed. This can occur in the 3388CPU host program, or from an HSA kernel executing on a GPU. 3389 33901. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3391 executed is obtained. 33922. A pointer to the kernel descriptor (see 3393 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3394 It must be for a kernel that is contained in a code object that that was 3395 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3396 queue is associated. 33973. Space is allocated for the kernel arguments using the HSA compatible runtime 3398 allocator for a memory region with the kernarg property for the kernel agent 3399 that will execute the kernel. It must be at least 16-byte aligned. 34004. Kernel argument values are assigned to the kernel argument memory 3401 allocation. The layout is defined in the *HSA Programmer's Language 3402 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3403 kernel argument memory in the same way constant memory is accessed. (Note 3404 that the HSA specification allows an implementation to copy the kernel 3405 argument contents to another location that is accessed by the kernel.) 34065. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3407 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3408 for the packet. The packet must be set up, and the final write must use an 3409 atomic store release to set the packet kind to ensure the packet contents are 3410 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3411 notify the kernel agent that the AQL queue has been updated. These rules, and 3412 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3413 System Architecture Specification* [HSA]_. 34146. A kernel dispatch packet includes information about the actual dispatch, 3415 such as grid and work-group size, together with information from the code 3416 object about the kernel, such as segment sizes. The HSA compatible runtime 3417 queries on the kernel symbol can be used to obtain the code object values 3418 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 34197. CP executes micro-code and is responsible for detecting and setting up the 3420 GPU to execute the wavefronts of a kernel dispatch. 34218. CP ensures that when the a wavefront starts executing the kernel machine 3422 code, the scalar general purpose registers (SGPR) and vector general purpose 3423 registers (VGPR) are set up as required by the machine code. The required 3424 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3425 register state is defined in 3426 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 34279. The prolog of the kernel machine code (see 3428 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3429 before continuing executing the machine code that corresponds to the kernel. 343010. When the kernel dispatch has completed execution, CP signals the completion 3431 signal specified in the kernel dispatch packet if not 0. 3432 3433.. _amdgpu-amdhsa-memory-spaces: 3434 3435Memory Spaces 3436~~~~~~~~~~~~~ 3437 3438The memory space properties are: 3439 3440 .. table:: AMDHSA Memory Spaces 3441 :name: amdgpu-amdhsa-memory-spaces-table 3442 3443 ================= =========== ======== ======= ================== 3444 Memory Space Name HSA Segment Hardware Address NULL Value 3445 Name Name Size 3446 ================= =========== ======== ======= ================== 3447 Private private scratch 32 0x00000000 3448 Local group LDS 32 0xFFFFFFFF 3449 Global global global 64 0x0000000000000000 3450 Constant constant *same as 64 0x0000000000000000 3451 global* 3452 Generic flat flat 64 0x0000000000000000 3453 Region N/A GDS 32 *not implemented 3454 for AMDHSA* 3455 ================= =========== ======== ======= ================== 3456 3457The global and constant memory spaces both use global virtual addresses, which 3458are the same virtual address space used by the CPU. However, some virtual 3459addresses may only be accessible to the CPU, some only accessible by the GPU, 3460and some by both. 3461 3462Using the constant memory space indicates that the data will not change during 3463the execution of the kernel. This allows scalar read instructions to be 3464used. The vector and scalar L1 caches are invalidated of volatile data before 3465each kernel dispatch execution to allow constant memory to change values between 3466kernel dispatches. 3467 3468The local memory space uses the hardware Local Data Store (LDS) which is 3469automatically allocated when the hardware creates work-groups of wavefronts, and 3470freed when all the wavefronts of a work-group have terminated. The data store 3471(DS) instructions can be used to access it. 3472 3473The private memory space uses the hardware scratch memory support. If the kernel 3474uses scratch, then the hardware allocates memory that is accessed using 3475wavefront lane dword (4 byte) interleaving. The mapping used from private 3476address to physical address is: 3477 3478 ``wavefront-scratch-base + 3479 (private-address * wavefront-size * 4) + 3480 (wavefront-lane-id * 4)`` 3481 3482There are different ways that the wavefront scratch base address is determined 3483by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3484memory can be accessed in an interleaved manner using buffer instruction with 3485the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3486instructions, or by flat instructions. If each lane of a wavefront accesses the 3487same private address, the interleaving results in adjacent dwords being accessed 3488and hence requires fewer cache lines to be fetched. Multi-dword access is not 3489supported except by flat and scratch instructions in GFX9-GFX10. 3490 3491The generic address space uses the hardware flat address support available in 3492GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3493local apertures), that are outside the range of addressible global memory, to 3494map from a flat address to a private or local address. 3495 3496FLAT instructions can take a flat address and access global, private (scratch) 3497and group (LDS) memory depending in if the address is within one of the 3498aperture ranges. Flat access to scratch requires hardware aperture setup and 3499setup in the kernel prologue (see 3500:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3501hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3502:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3503 3504To convert between a segment address and a flat address the base address of the 3505apertures address can be used. For GFX7-GFX8 these are available in the 3506:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3507Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3508GFX9-GFX10 the aperture base addresses are directly available as inline constant 3509registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3510address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3511which makes it easier to convert from flat to segment or segment to flat. 3512 3513Image and Samplers 3514~~~~~~~~~~~~~~~~~~ 3515 3516Image and sample handles created by an HSA compatible runtime (see 3517:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3518object respectively. In order to support the HSA ``query_sampler`` operations 3519two extra dwords are used to store the HSA BRIG enumeration values for the 3520queries that are not trivially deducible from the S# representation. 3521 3522HSA Signals 3523~~~~~~~~~~~ 3524 3525HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3526are 64-bit addresses of a structure allocated in memory accessible from both the 3527CPU and GPU. The structure is defined by the runtime and subject to change 3528between releases. For example, see [AMD-ROCm-github]_. 3529 3530.. _amdgpu-amdhsa-hsa-aql-queue: 3531 3532HSA AQL Queue 3533~~~~~~~~~~~~~ 3534 3535The HSA AQL queue structure is defined by an HSA compatible runtime (see 3536:ref:`amdgpu-os`) and subject to change between releases. For example, see 3537[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3538certain language features such as the flat address aperture bases. It also 3539contains fields used by CP such as managing the allocation of scratch memory. 3540 3541.. _amdgpu-amdhsa-kernel-descriptor: 3542 3543Kernel Descriptor 3544~~~~~~~~~~~~~~~~~ 3545 3546A kernel descriptor consists of the information needed by CP to initiate the 3547execution of a kernel, including the entry point address of the machine code 3548that implements the kernel. 3549 3550Code Object V3 Kernel Descriptor 3551++++++++++++++++++++++++++++++++ 3552 3553CP microcode requires the Kernel descriptor to be allocated on 64-byte 3554alignment. 3555 3556The fields used by CP for code objects before V3 also match those specified in 3557:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3558 3559 .. table:: Code Object V3 Kernel Descriptor 3560 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3561 3562 ======= ======= =============================== ============================ 3563 Bits Size Field Name Description 3564 ======= ======= =============================== ============================ 3565 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3566 address space memory 3567 required for a work-group 3568 in bytes. This does not 3569 include any dynamically 3570 allocated local address 3571 space memory that may be 3572 added when the kernel is 3573 dispatched. 3574 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3575 private address space 3576 memory required for a 3577 work-item in bytes. 3578 Additional space may need to 3579 be added to this value if 3580 the call stack has 3581 non-inlined function calls. 3582 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3583 memory pointed to by the 3584 AQL dispatch packet. The 3585 kernarg memory is used to 3586 pass arguments to the 3587 kernel. 3588 3589 * If the kernarg pointer in 3590 the dispatch packet is NULL 3591 then there are no kernel 3592 arguments. 3593 * If the kernarg pointer in 3594 the dispatch packet is 3595 not NULL and this value 3596 is 0 then the kernarg 3597 memory size is 3598 unspecified. 3599 * If the kernarg pointer in 3600 the dispatch packet is 3601 not NULL and this value 3602 is not 0 then the value 3603 specifies the kernarg 3604 memory size in bytes. It 3605 is recommended to provide 3606 a value as it may be used 3607 by CP to optimize making 3608 the kernarg memory 3609 visible to the kernel 3610 code. 3611 3612 127:96 4 bytes Reserved, must be 0. 3613 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3614 negative) from base 3615 address of kernel 3616 descriptor to kernel's 3617 entry point instruction 3618 which must be 256 byte 3619 aligned. 3620 351:272 20 Reserved, must be 0. 3621 bytes 3622 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3623 Reserved, must be 0. 3624 GFX90A 3625 Compute Shader (CS) 3626 program settings used by 3627 CP to set up 3628 ``COMPUTE_PGM_RSRC3`` 3629 configuration 3630 register. See 3631 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3632 GFX10 3633 Compute Shader (CS) 3634 program settings used by 3635 CP to set up 3636 ``COMPUTE_PGM_RSRC3`` 3637 configuration 3638 register. See 3639 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3640 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3641 program settings used by 3642 CP to set up 3643 ``COMPUTE_PGM_RSRC1`` 3644 configuration 3645 register. See 3646 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3647 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3648 program settings used by 3649 CP to set up 3650 ``COMPUTE_PGM_RSRC2`` 3651 configuration 3652 register. See 3653 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3654 458:448 7 bits *See separate bits below.* Enable the setup of the 3655 SGPR user data registers 3656 (see 3657 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3658 3659 The total number of SGPR 3660 user data registers 3661 requested must not exceed 3662 16 and match value in 3663 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3664 Any requests beyond 16 3665 will be ignored. 3666 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3667 _BUFFER 3668 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3669 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3670 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3671 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3672 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT 3673 3674 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3675 _SIZE 3676 457:455 3 bits Reserved, must be 0. 3677 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3678 Reserved, must be 0. 3679 GFX10 3680 - If 0 execute in 3681 wavefront size 64 mode. 3682 - If 1 execute in 3683 native wavefront size 3684 32 mode. 3685 463:459 1 bit Reserved, must be 0. 3686 464 1 bit RESERVED_464 Deprecated, must be 0. 3687 467:465 3 bits Reserved, must be 0. 3688 468 1 bit RESERVED_468 Deprecated, must be 0. 3689 469:471 3 bits Reserved, must be 0. 3690 511:472 5 bytes Reserved, must be 0. 3691 512 **Total size 64 bytes.** 3692 ======= ==================================================================== 3693 3694.. 3695 3696 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3697 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3698 3699 ======= ======= =============================== =========================================================================== 3700 Bits Size Field Name Description 3701 ======= ======= =============================== =========================================================================== 3702 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3703 blocks used by each work-item; 3704 granularity is device 3705 specific: 3706 3707 GFX6-GFX9 3708 - vgprs_used 0..256 3709 - max(0, ceil(vgprs_used / 4) - 1) 3710 GFX90A 3711 - vgprs_used 0..512 3712 - vgprs_used = align(arch_vgprs, 4) 3713 + acc_vgprs 3714 - max(0, ceil(vgprs_used / 8) - 1) 3715 GFX10 (wavefront size 64) 3716 - max_vgpr 1..256 3717 - max(0, ceil(vgprs_used / 4) - 1) 3718 GFX10 (wavefront size 32) 3719 - max_vgpr 1..256 3720 - max(0, ceil(vgprs_used / 8) - 1) 3721 3722 Where vgprs_used is defined 3723 as the highest VGPR number 3724 explicitly referenced plus 3725 one. 3726 3727 Used by CP to set up 3728 ``COMPUTE_PGM_RSRC1.VGPRS``. 3729 3730 The 3731 :ref:`amdgpu-assembler` 3732 calculates this 3733 automatically for the 3734 selected processor from 3735 values provided to the 3736 `.amdhsa_kernel` directive 3737 by the 3738 `.amdhsa_next_free_vgpr` 3739 nested directive (see 3740 :ref:`amdhsa-kernel-directives-table`). 3741 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3742 blocks used by a wavefront; 3743 granularity is device 3744 specific: 3745 3746 GFX6-GFX8 3747 - sgprs_used 0..112 3748 - max(0, ceil(sgprs_used / 8) - 1) 3749 GFX9 3750 - sgprs_used 0..112 3751 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3752 GFX10 3753 Reserved, must be 0. 3754 (128 SGPRs always 3755 allocated.) 3756 3757 Where sgprs_used is 3758 defined as the highest 3759 SGPR number explicitly 3760 referenced plus one, plus 3761 a target specific number 3762 of additional special 3763 SGPRs for VCC, 3764 FLAT_SCRATCH (GFX7+) and 3765 XNACK_MASK (GFX8+), and 3766 any additional 3767 target specific 3768 limitations. It does not 3769 include the 16 SGPRs added 3770 if a trap handler is 3771 enabled. 3772 3773 The target specific 3774 limitations and special 3775 SGPR layout are defined in 3776 the hardware 3777 documentation, which can 3778 be found in the 3779 :ref:`amdgpu-processors` 3780 table. 3781 3782 Used by CP to set up 3783 ``COMPUTE_PGM_RSRC1.SGPRS``. 3784 3785 The 3786 :ref:`amdgpu-assembler` 3787 calculates this 3788 automatically for the 3789 selected processor from 3790 values provided to the 3791 `.amdhsa_kernel` directive 3792 by the 3793 `.amdhsa_next_free_sgpr` 3794 and `.amdhsa_reserve_*` 3795 nested directives (see 3796 :ref:`amdhsa-kernel-directives-table`). 3797 11:10 2 bits PRIORITY Must be 0. 3798 3799 Start executing wavefront 3800 at the specified priority. 3801 3802 CP is responsible for 3803 filling in 3804 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3805 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3806 with specified rounding 3807 mode for single (32 3808 bit) floating point 3809 precision floating point 3810 operations. 3811 3812 Floating point rounding 3813 mode values are defined in 3814 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3815 3816 Used by CP to set up 3817 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3818 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3819 with specified rounding 3820 denorm mode for half/double (16 3821 and 64-bit) floating point 3822 precision floating point 3823 operations. 3824 3825 Floating point rounding 3826 mode values are defined in 3827 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3828 3829 Used by CP to set up 3830 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3831 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3832 with specified denorm mode 3833 for single (32 3834 bit) floating point 3835 precision floating point 3836 operations. 3837 3838 Floating point denorm mode 3839 values are defined in 3840 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3841 3842 Used by CP to set up 3843 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3844 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3845 with specified denorm mode 3846 for half/double (16 3847 and 64-bit) floating point 3848 precision floating point 3849 operations. 3850 3851 Floating point denorm mode 3852 values are defined in 3853 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3854 3855 Used by CP to set up 3856 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3857 20 1 bit PRIV Must be 0. 3858 3859 Start executing wavefront 3860 in privilege trap handler 3861 mode. 3862 3863 CP is responsible for 3864 filling in 3865 ``COMPUTE_PGM_RSRC1.PRIV``. 3866 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3867 with DX10 clamp mode 3868 enabled. Used by the vector 3869 ALU to force DX10 style 3870 treatment of NaN's (when 3871 set, clamp NaN to zero, 3872 otherwise pass NaN 3873 through). 3874 3875 Used by CP to set up 3876 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3877 22 1 bit DEBUG_MODE Must be 0. 3878 3879 Start executing wavefront 3880 in single step mode. 3881 3882 CP is responsible for 3883 filling in 3884 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3885 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3886 with IEEE mode 3887 enabled. Floating point 3888 opcodes that support 3889 exception flag gathering 3890 will quiet and propagate 3891 signaling-NaN inputs per 3892 IEEE 754-2008. Min_dx10 and 3893 max_dx10 become IEEE 3894 754-2008 compliant due to 3895 signaling-NaN propagation 3896 and quieting. 3897 3898 Used by CP to set up 3899 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3900 24 1 bit BULKY Must be 0. 3901 3902 Only one work-group allowed 3903 to execute on a compute 3904 unit. 3905 3906 CP is responsible for 3907 filling in 3908 ``COMPUTE_PGM_RSRC1.BULKY``. 3909 25 1 bit CDBG_USER Must be 0. 3910 3911 Flag that can be used to 3912 control debugging code. 3913 3914 CP is responsible for 3915 filling in 3916 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3917 26 1 bit FP16_OVFL GFX6-GFX8 3918 Reserved, must be 0. 3919 GFX9-GFX10 3920 Wavefront starts execution 3921 with specified fp16 overflow 3922 mode. 3923 3924 - If 0, fp16 overflow generates 3925 +/-INF values. 3926 - If 1, fp16 overflow that is the 3927 result of an +/-INF input value 3928 or divide by 0 produces a +/-INF, 3929 otherwise clamps computed 3930 overflow to +/-MAX_FP16 as 3931 appropriate. 3932 3933 Used by CP to set up 3934 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3935 28:27 2 bits Reserved, must be 0. 3936 29 1 bit WGP_MODE GFX6-GFX9 3937 Reserved, must be 0. 3938 GFX10 3939 - If 0 execute work-groups in 3940 CU wavefront execution mode. 3941 - If 1 execute work-groups on 3942 in WGP wavefront execution mode. 3943 3944 See :ref:`amdgpu-amdhsa-memory-model`. 3945 3946 Used by CP to set up 3947 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3948 30 1 bit MEM_ORDERED GFX6-GFX9 3949 Reserved, must be 0. 3950 GFX10 3951 Controls the behavior of the 3952 s_waitcnt's vmcnt and vscnt 3953 counters. 3954 3955 - If 0 vmcnt reports completion 3956 of load and atomic with return 3957 out of order with sample 3958 instructions, and the vscnt 3959 reports the completion of 3960 store and atomic without 3961 return in order. 3962 - If 1 vmcnt reports completion 3963 of load, atomic with return 3964 and sample instructions in 3965 order, and the vscnt reports 3966 the completion of store and 3967 atomic without return in order. 3968 3969 Used by CP to set up 3970 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3971 31 1 bit FWD_PROGRESS GFX6-GFX9 3972 Reserved, must be 0. 3973 GFX10 3974 - If 0 execute SIMD wavefronts 3975 using oldest first policy. 3976 - If 1 execute SIMD wavefronts to 3977 ensure wavefronts will make some 3978 forward progress. 3979 3980 Used by CP to set up 3981 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3982 32 **Total size 4 bytes** 3983 ======= =================================================================================================================== 3984 3985.. 3986 3987 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3988 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3989 3990 ======= ======= =============================== =========================================================================== 3991 Bits Size Field Name Description 3992 ======= ======= =============================== =========================================================================== 3993 0 1 bit ENABLE_PRIVATE_SEGMENT Enable the setup of the 3994 private segment. 3995 3996 In addition, enable the 3997 setup of the SGPR 3998 wavefront scratch offset 3999 system register (see 4000 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4001 4002 Used by CP to set up 4003 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4004 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4005 user data registers 4006 requested. This number must 4007 match the number of user 4008 data registers enabled. 4009 4010 Used by CP to set up 4011 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4012 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4013 4014 This bit represents 4015 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4016 which is set by the CP if 4017 the runtime has installed a 4018 trap handler. 4019 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4020 system SGPR register for 4021 the work-group id in the X 4022 dimension (see 4023 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4024 4025 Used by CP to set up 4026 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4027 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4028 system SGPR register for 4029 the work-group id in the Y 4030 dimension (see 4031 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4032 4033 Used by CP to set up 4034 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4035 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4036 system SGPR register for 4037 the work-group id in the Z 4038 dimension (see 4039 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4040 4041 Used by CP to set up 4042 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4043 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4044 system SGPR register for 4045 work-group information (see 4046 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4047 4048 Used by CP to set up 4049 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4050 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4051 VGPR system registers used 4052 for the work-item ID. 4053 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4054 defines the values. 4055 4056 Used by CP to set up 4057 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4058 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4059 4060 Wavefront starts execution 4061 with address watch 4062 exceptions enabled which 4063 are generated when L1 has 4064 witnessed a thread access 4065 an *address of 4066 interest*. 4067 4068 CP is responsible for 4069 filling in the address 4070 watch bit in 4071 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4072 according to what the 4073 runtime requests. 4074 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4075 4076 Wavefront starts execution 4077 with memory violation 4078 exceptions exceptions 4079 enabled which are generated 4080 when a memory violation has 4081 occurred for this wavefront from 4082 L1 or LDS 4083 (write-to-read-only-memory, 4084 mis-aligned atomic, LDS 4085 address out of range, 4086 illegal address, etc.). 4087 4088 CP sets the memory 4089 violation bit in 4090 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4091 according to what the 4092 runtime requests. 4093 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4094 4095 CP uses the rounded value 4096 from the dispatch packet, 4097 not this value, as the 4098 dispatch may contain 4099 dynamically allocated group 4100 segment memory. CP writes 4101 directly to 4102 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4103 4104 Amount of group segment 4105 (LDS) to allocate for each 4106 work-group. Granularity is 4107 device specific: 4108 4109 GFX6 4110 roundup(lds-size / (64 * 4)) 4111 GFX7-GFX10 4112 roundup(lds-size / (128 * 4)) 4113 4114 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4115 _INVALID_OPERATION with specified exceptions 4116 enabled. 4117 4118 Used by CP to set up 4119 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4120 (set from bits 0..6). 4121 4122 IEEE 754 FP Invalid 4123 Operation 4124 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4125 _SOURCE input operands is a 4126 denormal number 4127 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4128 _DIVISION_BY_ZERO Zero 4129 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4130 _OVERFLOW 4131 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4132 _UNDERFLOW 4133 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4134 _INEXACT 4135 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4136 _ZERO (rcp_iflag_f32 instruction 4137 only) 4138 31 1 bit Reserved, must be 0. 4139 32 **Total size 4 bytes.** 4140 ======= =================================================================================================================== 4141 4142.. 4143 4144 .. table:: compute_pgm_rsrc3 for GFX90A 4145 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4146 4147 ======= ======= =============================== =========================================================================== 4148 Bits Size Field Name Description 4149 ======= ======= =============================== =========================================================================== 4150 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4151 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4152 63 - accum-offset = 256. 4153 6:15 10 Reserved, must be 0. 4154 bits 4155 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4156 launched in the same CU. 4157 - If 1 the waves of a work-group can be 4158 launched in different CUs. The waves 4159 cannot use S_BARRIER or LDS. 4160 17:31 15 Reserved, must be 0. 4161 bits 4162 32 **Total size 4 bytes.** 4163 ======= =================================================================================================================== 4164 4165.. 4166 4167 .. table:: compute_pgm_rsrc3 for GFX10 4168 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4169 4170 ======= ======= =============================== =========================================================================== 4171 Bits Size Field Name Description 4172 ======= ======= =============================== =========================================================================== 4173 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4174 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4175 31:4 28 Reserved, must be 0. 4176 bits 4177 32 **Total size 4 bytes.** 4178 ======= =================================================================================================================== 4179 4180.. 4181 4182 .. table:: Floating Point Rounding Mode Enumeration Values 4183 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4184 4185 ====================================== ===== ============================== 4186 Enumeration Name Value Description 4187 ====================================== ===== ============================== 4188 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4189 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4190 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4191 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4192 ====================================== ===== ============================== 4193 4194.. 4195 4196 .. table:: Floating Point Denorm Mode Enumeration Values 4197 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4198 4199 ====================================== ===== ============================== 4200 Enumeration Name Value Description 4201 ====================================== ===== ============================== 4202 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4203 Denorms 4204 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4205 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4206 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4207 ====================================== ===== ============================== 4208 4209.. 4210 4211 .. table:: System VGPR Work-Item ID Enumeration Values 4212 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4213 4214 ======================================== ===== ============================ 4215 Enumeration Name Value Description 4216 ======================================== ===== ============================ 4217 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4218 ID. 4219 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4220 dimensions ID. 4221 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4222 dimensions ID. 4223 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4224 ======================================== ===== ============================ 4225 4226.. _amdgpu-amdhsa-initial-kernel-execution-state: 4227 4228Initial Kernel Execution State 4229~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4230 4231This section defines the register state that will be set up by the packet 4232processor prior to the start of execution of every wavefront. This is limited by 4233the constraints of the hardware controllers of CP/ADC/SPI. 4234 4235The order of the SGPR registers is defined, but the compiler can specify which 4236ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4237fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4238for enabled registers are dense starting at SGPR0: the first enabled register is 4239SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4240an SGPR number. 4241 4242The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4243all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4244using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4245actually initialized. These are then immediately followed by the System SGPRs 4246that are set up by ADC/SPI and can have different values for each wavefront of 4247the grid dispatch. 4248 4249SGPR register initial state is defined in 4250:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4251 4252 .. table:: SGPR Register Set Up Order 4253 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4254 4255 ========== ========================== ====== ============================== 4256 SGPR Order Name Number Description 4257 (kernel descriptor enable of 4258 field) SGPRs 4259 ========== ========================== ====== ============================== 4260 First Private Segment Buffer 4 See 4261 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4262 _segment_buffer) 4263 then Dispatch Ptr 2 64-bit address of AQL dispatch 4264 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4265 actually executing. 4266 then Queue Ptr 2 64-bit address of amd_queue_t 4267 (enable_sgpr_queue_ptr) object for AQL queue on which 4268 the dispatch packet was 4269 queued. 4270 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4271 (enable_sgpr_kernarg segment. This is directly 4272 _segment_ptr) copied from the 4273 kernarg_address in the kernel 4274 dispatch packet. 4275 4276 Having CP load it once avoids 4277 loading it at the beginning of 4278 every wavefront. 4279 then Dispatch Id 2 64-bit Dispatch ID of the 4280 (enable_sgpr_dispatch_id) dispatch packet being 4281 executed. 4282 then Flat Scratch Init 2 See 4283 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4284 _init) 4285 then Private Segment Size 1 The 32-bit byte size of a 4286 (enable_sgpr_private single work-item's memory 4287 _segment_size) allocation. This is the 4288 value from the kernel 4289 dispatch packet Private 4290 Segment Byte Size rounded up 4291 by CP to a multiple of 4292 DWORD. 4293 4294 Having CP load it once avoids 4295 loading it at the beginning of 4296 every wavefront. 4297 4298 This is not used for 4299 GFX7-GFX8 since it is the same 4300 value as the second SGPR of 4301 Flat Scratch Init. However, it 4302 may be needed for GFX9-GFX10 which 4303 changes the meaning of the 4304 Flat Scratch Init value. 4305 then Work-Group Id X 1 32-bit work-group id in X 4306 (enable_sgpr_workgroup_id dimension of grid for 4307 _X) wavefront. 4308 then Work-Group Id Y 1 32-bit work-group id in Y 4309 (enable_sgpr_workgroup_id dimension of grid for 4310 _Y) wavefront. 4311 then Work-Group Id Z 1 32-bit work-group id in Z 4312 (enable_sgpr_workgroup_id dimension of grid for 4313 _Z) wavefront. 4314 then Work-Group Info 1 {first_wavefront, 14'b0000, 4315 (enable_sgpr_workgroup ordered_append_term[10:0], 4316 _info) threadgroup_size_in_wavefronts[5:0]} 4317 then Scratch Wavefront Offset 1 See 4318 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4319 _segment_wavefront_offset) and 4320 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4321 ========== ========================== ====== ============================== 4322 4323The order of the VGPR registers is defined, but the compiler can specify which 4324ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4325fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4326for enabled registers are dense starting at VGPR0: the first enabled register is 4327VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4328VGPR number. 4329 4330There are different methods used for the VGPR initial state: 4331 4332* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4333 specifies otherwise, a separate VGPR register is used per work-item ID. The 4334 VGPR register initial state for this method is defined in 4335 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4336* If *Target Properties* column of :ref:`amdgpu-processor-table` 4337 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4338 for all work-item IDs. The register layout for this method is defined in 4339 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4340 4341 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4342 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4343 4344 ========== ========================== ====== ============================== 4345 VGPR Order Name Number Description 4346 (kernel descriptor enable of 4347 field) VGPRs 4348 ========== ========================== ====== ============================== 4349 First Work-Item Id X 1 32-bit work-item id in X 4350 (Always initialized) dimension of work-group for 4351 wavefront lane. 4352 then Work-Item Id Y 1 32-bit work-item id in Y 4353 (enable_vgpr_workitem_id dimension of work-group for 4354 > 0) wavefront lane. 4355 then Work-Item Id Z 1 32-bit work-item id in Z 4356 (enable_vgpr_workitem_id dimension of work-group for 4357 > 1) wavefront lane. 4358 ========== ========================== ====== ============================== 4359 4360.. 4361 4362 .. table:: Register Layout for Packed Work-Item ID Method 4363 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4364 4365 ======= ======= ================ ========================================= 4366 Bits Size Field Name Description 4367 ======= ======= ================ ========================================= 4368 0:9 10 bits Work-Item Id X Work-item id in X 4369 dimension of work-group for 4370 wavefront lane. 4371 4372 Always initialized. 4373 4374 10:19 10 bits Work-Item Id Y Work-item id in Y 4375 dimension of work-group for 4376 wavefront lane. 4377 4378 Initialized if enable_vgpr_workitem_id > 4379 0, otherwise set to 0. 4380 20:29 10 bits Work-Item Id Z Work-item id in Z 4381 dimension of work-group for 4382 wavefront lane. 4383 4384 Initialized if enable_vgpr_workitem_id > 4385 1, otherwise set to 0. 4386 30:31 2 bits Reserved, set to 0. 4387 ======= ======= ================ ========================================= 4388 4389The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4390 43911. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4392 registers. 43932. Work-group Id registers X, Y, Z are set by ADC which supports any 4394 combination including none. 43953. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4396 its value cannot be included with the flat scratch init value which is per 4397 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 43984. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4399 or (X, Y, Z). 44005. Flat Scratch register pair initialization is described in 4401 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4402 4403The global segment can be accessed either using buffer instructions (GFX6 which 4404has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4405instructions (GFX9-GFX10). 4406 4407If buffer operations are used, then the compiler can generate a V# with the 4408following properties: 4409 4410* base address of 0 4411* no swizzle 4412* ATC: 1 if IOMMU present (such as APU) 4413* ptr64: 1 4414* MTYPE set to support memory coherence that matches the runtime (such as CC for 4415 APU and NC for dGPU). 4416 4417.. _amdgpu-amdhsa-kernel-prolog: 4418 4419Kernel Prolog 4420~~~~~~~~~~~~~ 4421 4422The compiler performs initialization in the kernel prologue depending on the 4423target and information about things like stack usage in the kernel and called 4424functions. Some of this initialization requires the compiler to request certain 4425User and System SGPRs be present in the 4426:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4427:ref:`amdgpu-amdhsa-kernel-descriptor`. 4428 4429.. _amdgpu-amdhsa-kernel-prolog-cfi: 4430 4431CFI 4432+++ 4433 44341. The CFI return address is undefined. 4435 44362. The CFI CFA is defined using an expression which evaluates to a location 4437 description that comprises one memory location description for the 4438 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4439 4440.. _amdgpu-amdhsa-kernel-prolog-m0: 4441 4442M0 4443++ 4444 4445GFX6-GFX8 4446 The M0 register must be initialized with a value at least the total LDS size 4447 if the kernel may access LDS via DS or flat operations. Total LDS size is 4448 available in dispatch packet. For M0, it is also possible to use maximum 4449 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4450 GFX7-GFX8). 4451GFX9-GFX10 4452 The M0 register is not used for range checking LDS accesses and so does not 4453 need to be initialized in the prolog. 4454 4455.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4456 4457Stack Pointer 4458+++++++++++++ 4459 4460If the kernel has function calls it must set up the ABI stack pointer described 4461in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4462SGPR32 to the unswizzled scratch offset of the address past the last local 4463allocation. 4464 4465.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4466 4467Frame Pointer 4468+++++++++++++ 4469 4470If the kernel needs a frame pointer for the reasons defined in 4471``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4472kernel prolog. If a frame pointer is not required then all uses of the frame 4473pointer are replaced with immediate ``0`` offsets. 4474 4475.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4476 4477Flat Scratch 4478++++++++++++ 4479 4480There are different methods used for initializing flat scratch: 4481 4482* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4483 specifies *Does not support generic address space*: 4484 4485 Flat scratch is not supported and there is no flat scratch register pair. 4486 4487* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4488 specifies *Offset flat scratch*: 4489 4490 If the kernel or any function it calls may use flat operations to access 4491 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4492 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4493 Scratch Wavefront Offset SGPR registers (see 4494 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4495 4496 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4497 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4498 being managed by SPI for the queue executing the kernel dispatch. This is 4499 the same value used in the Scratch Segment Buffer V# base address. 4500 4501 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4502 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4503 4504 The prolog must add the value of Scratch Wavefront Offset to get the 4505 wavefront's byte scratch backing memory offset from 4506 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4507 4508 The Scratch Wavefront Offset must also be used as an offset with Private 4509 segment address when using the Scratch Segment Buffer. 4510 4511 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4512 shifted by 8 before moving into FLAT_SCRATCH_HI. 4513 4514 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4515 SGPRn is the highest numbered SGPR allocated to the wavefront). 4516 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4517 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4518 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4519 aperture. 4520 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4521 work-items scratch memory usage. 4522 4523 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4524 checks that the value in the kernel dispatch packet Private Segment Byte 4525 Size is not larger and requests the runtime to increase the queue's scratch 4526 size if necessary. 4527 4528 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4529 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4530 loading it at the beginning of every wavefront. 4531 4532 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4533 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4534 in flat memory instructions. 4535 4536* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4537 specifies *Absolute flat scratch*: 4538 4539 If the kernel or any function it calls may use flat operations to access 4540 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4541 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4542 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4543 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4544 4545 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4546 memory being managed by SPI for the queue executing the kernel dispatch. 4547 4548 CP obtains this from the runtime. 4549 4550 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4551 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4552 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4553 memory instructions. 4554 4555 The Scratch Wavefront Offset must also be used as an offset with Private 4556 segment address when using the Scratch Segment Buffer (see 4557 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4558 4559.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4560 4561Private Segment Buffer 4562++++++++++++++++++++++ 4563 4564Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4565that are used as a V# to access scratch. CP uses the value provided by the 4566runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4567access the private memory space using a segment address. See 4568:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4569 4570The scratch V# is a four-aligned SGPR and always selected for the kernel as 4571follows: 4572 4573 - If it is known during instruction selection that there is stack usage, 4574 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4575 optimizations are disabled (``-O0``), if stack objects already exist (for 4576 locals, etc.), or if there are any function calls. 4577 4578 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4579 are reserved for the tentative scratch V#. These will be used if it is 4580 determined that spilling is needed. 4581 4582 - If no use is made of the tentative scratch V#, then it is unreserved, 4583 and the register count is determined ignoring it. 4584 - If use is made of the tentative scratch V#, then its register numbers 4585 are shifted to the first four-aligned SGPR index after the highest one 4586 allocated by the register allocator, and all uses are updated. The 4587 register count includes them in the shifted location. 4588 - In either case, if the processor has the SGPR allocation bug, the 4589 tentative allocation is not shifted or unreserved in order to ensure 4590 the register count is higher to workaround the bug. 4591 4592 .. note:: 4593 4594 This approach of using a tentative scratch V# and shifting the register 4595 numbers if used avoids having to perform register allocation a second 4596 time if the tentative V# is eliminated. This is more efficient and 4597 avoids the problem that the second register allocation may perform 4598 spilling which will fail as there is no longer a scratch V#. 4599 4600When the kernel prolog code is being emitted it is known whether the scratch V# 4601described above is actually used. If it is, the prolog code must set it up by 4602copying the Private Segment Buffer to the scratch V# registers and then adding 4603the Private Segment Wavefront Offset to the queue base address in the V#. The 4604result is a V# with a base address pointing to the beginning of the wavefront 4605scratch backing memory. 4606 4607The Private Segment Buffer is always requested, but the Private Segment 4608Wavefront Offset is only requested if it is used (see 4609:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4610 4611.. _amdgpu-amdhsa-memory-model: 4612 4613Memory Model 4614~~~~~~~~~~~~ 4615 4616This section describes the mapping of the LLVM memory model onto AMDGPU machine 4617code (see :ref:`memmodel`). 4618 4619The AMDGPU backend supports the memory synchronization scopes specified in 4620:ref:`amdgpu-memory-scopes`. 4621 4622The code sequences used to implement the memory model specify the order of 4623instructions that a single thread must execute. The ``s_waitcnt`` and cache 4624management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4625to other memory instructions executed by the same thread. This allows them to be 4626moved earlier or later which can allow them to be combined with other instances 4627of the same instruction, or hoisted/sunk out of loops to improve performance. 4628Only the instructions related to the memory model are given; additional 4629``s_waitcnt`` instructions are required to ensure registers are defined before 4630being used. These may be able to be combined with the memory model ``s_waitcnt`` 4631instructions as described above. 4632 4633The AMDGPU backend supports the following memory models: 4634 4635 HSA Memory Model [HSA]_ 4636 The HSA memory model uses a single happens-before relation for all address 4637 spaces (see :ref:`amdgpu-address-spaces`). 4638 OpenCL Memory Model [OpenCL]_ 4639 The OpenCL memory model which has separate happens-before relations for the 4640 global and local address spaces. Only a fence specifying both global and 4641 local address space, and seq_cst instructions join the relationships. Since 4642 the LLVM ``memfence`` instruction does not allow an address space to be 4643 specified the OpenCL fence has to conservatively assume both local and 4644 global address space was specified. However, optimizations can often be 4645 done to eliminate the additional ``s_waitcnt`` instructions when there are 4646 no intervening memory instructions which access the corresponding address 4647 space. The code sequences in the table indicate what can be omitted for the 4648 OpenCL memory. The target triple environment is used to determine if the 4649 source language is OpenCL (see :ref:`amdgpu-opencl`). 4650 4651``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4652operations. 4653 4654``buffer/global/flat_load/store/atomic`` instructions to global memory are 4655termed vector memory operations. 4656 4657Private address space uses ``buffer_load/store`` using the scratch V# 4658(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4659is accessing the memory, atomic memory orderings are not meaningful, and all 4660accesses are treated as non-atomic. 4661 4662Constant address space uses ``buffer/global_load`` instructions (or equivalent 4663scalar memory instructions). Since the constant address space contents do not 4664change during the execution of a kernel dispatch it is not legal to perform 4665stores, and atomic memory orderings are not meaningful, and all accesses are 4666treated as non-atomic. 4667 4668A memory synchronization scope wider than work-group is not meaningful for the 4669group (LDS) address space and is treated as work-group. 4670 4671The memory model does not support the region address space which is treated as 4672non-atomic. 4673 4674Acquire memory ordering is not meaningful on store atomic instructions and is 4675treated as non-atomic. 4676 4677Release memory ordering is not meaningful on load atomic instructions and is 4678treated a non-atomic. 4679 4680Acquire-release memory ordering is not meaningful on load or store atomic 4681instructions and is treated as acquire and release respectively. 4682 4683The memory order also adds the single thread optimization constraints defined in 4684table 4685:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4686 4687 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4688 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4689 4690 ============ ============================================================== 4691 LLVM Memory Optimization Constraints 4692 Ordering 4693 ============ ============================================================== 4694 unordered *none* 4695 monotonic *none* 4696 acquire - If a load atomic/atomicrmw then no following load/load 4697 atomic/store/store atomic/atomicrmw/fence instruction can be 4698 moved before the acquire. 4699 - If a fence then same as load atomic, plus no preceding 4700 associated fence-paired-atomic can be moved after the fence. 4701 release - If a store atomic/atomicrmw then no preceding load/load 4702 atomic/store/store atomic/atomicrmw/fence instruction can be 4703 moved after the release. 4704 - If a fence then same as store atomic, plus no following 4705 associated fence-paired-atomic can be moved before the 4706 fence. 4707 acq_rel Same constraints as both acquire and release. 4708 seq_cst - If a load atomic then same constraints as acquire, plus no 4709 preceding sequentially consistent load atomic/store 4710 atomic/atomicrmw/fence instruction can be moved after the 4711 seq_cst. 4712 - If a store atomic then the same constraints as release, plus 4713 no following sequentially consistent load atomic/store 4714 atomic/atomicrmw/fence instruction can be moved before the 4715 seq_cst. 4716 - If an atomicrmw/fence then same constraints as acq_rel. 4717 ============ ============================================================== 4718 4719The code sequences used to implement the memory model are defined in the 4720following sections: 4721 4722* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4723* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4724* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4725 4726.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4727 4728Memory Model GFX6-GFX9 4729++++++++++++++++++++++ 4730 4731For GFX6-GFX9: 4732 4733* Each agent has multiple shader arrays (SA). 4734* Each SA has multiple compute units (CU). 4735* Each CU has multiple SIMDs that execute wavefronts. 4736* The wavefronts for a single work-group are executed in the same CU but may be 4737 executed by different SIMDs. 4738* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4739 executing on it. 4740* All LDS operations of a CU are performed as wavefront wide operations in a 4741 global order and involve no caching. Completion is reported to a wavefront in 4742 execution order. 4743* The LDS memory has multiple request queues shared by the SIMDs of a 4744 CU. Therefore, the LDS operations performed by different wavefronts of a 4745 work-group can be reordered relative to each other, which can result in 4746 reordering the visibility of vector memory operations with respect to LDS 4747 operations of other wavefronts in the same work-group. A ``s_waitcnt 4748 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4749 vector memory operations between wavefronts of a work-group, but not between 4750 operations performed by the same wavefront. 4751* The vector memory operations are performed as wavefront wide operations and 4752 completion is reported to a wavefront in execution order. The exception is 4753 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4754 vector memory order if they access LDS memory, and out of LDS operation order 4755 if they access global memory. 4756* The vector memory operations access a single vector L1 cache shared by all 4757 SIMDs a CU. Therefore, no special action is required for coherence between the 4758 lanes of a single wavefront, or for coherence between wavefronts in the same 4759 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4760 wavefronts executing in different work-groups as they may be executing on 4761 different CUs. 4762* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4763 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4764 scalar operations are used in a restricted way so do not impact the memory 4765 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4766* The vector and scalar memory operations use an L2 cache shared by all CUs on 4767 the same agent. 4768* The L2 cache has independent channels to service disjoint ranges of virtual 4769 addresses. 4770* Each CU has a separate request queue per channel. Therefore, the vector and 4771 scalar memory operations performed by wavefronts executing in different 4772 work-groups (which may be executing on different CUs) of an agent can be 4773 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4774 ensure synchronization between vector memory operations of different CUs. It 4775 ensures a previous vector memory operation has completed before executing a 4776 subsequent vector memory or LDS operation and so can be used to meet the 4777 requirements of acquire and release. 4778* The L2 cache can be kept coherent with other agents on some targets, or ranges 4779 of virtual addresses can be set up to bypass it to ensure system coherence. 4780 4781Scalar memory operations are only used to access memory that is proven to not 4782change during the execution of the kernel dispatch. This includes constant 4783address space and global address space for program scope ``const`` variables. 4784Therefore, the kernel machine code does not have to maintain the scalar cache to 4785ensure it is coherent with the vector caches. The scalar and vector caches are 4786invalidated between kernel dispatches by CP since constant address space data 4787may change between kernel dispatch executions. See 4788:ref:`amdgpu-amdhsa-memory-spaces`. 4789 4790The one exception is if scalar writes are used to spill SGPR registers. In this 4791case the AMDGPU backend ensures the memory location used to spill is never 4792accessed by vector memory operations at the same time. If scalar writes are used 4793then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4794return since the locations may be used for vector memory instructions by a 4795future wavefront that uses the same scratch area, or a function call that 4796creates a frame at the same address, respectively. There is no need for a 4797``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4798 4799For kernarg backing memory: 4800 4801* CP invalidates the L1 cache at the start of each kernel dispatch. 4802* On dGPU the kernarg backing memory is allocated in host memory accessed as 4803 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4804 causes it to be treated as non-volatile and so is not invalidated by 4805 ``*_vol``. 4806* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4807 and so the L2 cache will be coherent with the CPU and other agents. 4808 4809Scratch backing memory (which is used for the private address space) is accessed 4810with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4811only accessed by a single thread, and is always write-before-read, there is 4812never a need to invalidate these entries from the L1 cache. Hence all cache 4813invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4814 4815The code sequences used to implement the memory model for GFX6-GFX9 are defined 4816in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4817 4818 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4819 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4820 4821 ============ ============ ============== ========== ================================ 4822 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4823 Ordering Sync Scope Address GFX6-GFX9 4824 Space 4825 ============ ============ ============== ========== ================================ 4826 **Non-Atomic** 4827 ------------------------------------------------------------------------------------ 4828 load *none* *none* - global - !volatile & !nontemporal 4829 - generic 4830 - private 1. buffer/global/flat_load 4831 - constant 4832 - !volatile & nontemporal 4833 4834 1. buffer/global/flat_load 4835 glc=1 slc=1 4836 4837 - volatile 4838 4839 1. buffer/global/flat_load 4840 glc=1 4841 2. s_waitcnt vmcnt(0) 4842 4843 - Must happen before 4844 any following volatile 4845 global/generic 4846 load/store. 4847 - Ensures that 4848 volatile 4849 operations to 4850 different 4851 addresses will not 4852 be reordered by 4853 hardware. 4854 4855 load *none* *none* - local 1. ds_load 4856 store *none* *none* - global - !volatile & !nontemporal 4857 - generic 4858 - private 1. buffer/global/flat_store 4859 - constant 4860 - !volatile & nontemporal 4861 4862 1. buffer/global/flat_store 4863 glc=1 slc=1 4864 4865 - volatile 4866 4867 1. buffer/global/flat_store 4868 2. s_waitcnt vmcnt(0) 4869 4870 - Must happen before 4871 any following volatile 4872 global/generic 4873 load/store. 4874 - Ensures that 4875 volatile 4876 operations to 4877 different 4878 addresses will not 4879 be reordered by 4880 hardware. 4881 4882 store *none* *none* - local 1. ds_store 4883 **Unordered Atomic** 4884 ------------------------------------------------------------------------------------ 4885 load atomic unordered *any* *any* *Same as non-atomic*. 4886 store atomic unordered *any* *any* *Same as non-atomic*. 4887 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4888 **Monotonic Atomic** 4889 ------------------------------------------------------------------------------------ 4890 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4891 - wavefront - local 4892 - workgroup - generic 4893 load atomic monotonic - agent - global 1. buffer/global/flat_load 4894 - system - generic glc=1 4895 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4896 - wavefront - generic 4897 - workgroup 4898 - agent 4899 - system 4900 store atomic monotonic - singlethread - local 1. ds_store 4901 - wavefront 4902 - workgroup 4903 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4904 - wavefront - generic 4905 - workgroup 4906 - agent 4907 - system 4908 atomicrmw monotonic - singlethread - local 1. ds_atomic 4909 - wavefront 4910 - workgroup 4911 **Acquire Atomic** 4912 ------------------------------------------------------------------------------------ 4913 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4914 - wavefront - local 4915 - generic 4916 load atomic acquire - workgroup - global 1. buffer/global_load 4917 load atomic acquire - workgroup - local 1. ds/flat_load 4918 - generic 2. s_waitcnt lgkmcnt(0) 4919 4920 - If OpenCL, omit. 4921 - Must happen before 4922 any following 4923 global/generic 4924 load/load 4925 atomic/store/store 4926 atomic/atomicrmw. 4927 - Ensures any 4928 following global 4929 data read is no 4930 older than a local load 4931 atomic value being 4932 acquired. 4933 4934 load atomic acquire - agent - global 1. buffer/global_load 4935 - system glc=1 4936 2. s_waitcnt vmcnt(0) 4937 4938 - Must happen before 4939 following 4940 buffer_wbinvl1_vol. 4941 - Ensures the load 4942 has completed 4943 before invalidating 4944 the cache. 4945 4946 3. buffer_wbinvl1_vol 4947 4948 - Must happen before 4949 any following 4950 global/generic 4951 load/load 4952 atomic/atomicrmw. 4953 - Ensures that 4954 following 4955 loads will not see 4956 stale global data. 4957 4958 load atomic acquire - agent - generic 1. flat_load glc=1 4959 - system 2. s_waitcnt vmcnt(0) & 4960 lgkmcnt(0) 4961 4962 - If OpenCL omit 4963 lgkmcnt(0). 4964 - Must happen before 4965 following 4966 buffer_wbinvl1_vol. 4967 - Ensures the flat_load 4968 has completed 4969 before invalidating 4970 the cache. 4971 4972 3. buffer_wbinvl1_vol 4973 4974 - Must happen before 4975 any following 4976 global/generic 4977 load/load 4978 atomic/atomicrmw. 4979 - Ensures that 4980 following loads 4981 will not see stale 4982 global data. 4983 4984 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 4985 - wavefront - local 4986 - generic 4987 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 4988 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 4989 - generic 2. s_waitcnt lgkmcnt(0) 4990 4991 - If OpenCL, omit. 4992 - Must happen before 4993 any following 4994 global/generic 4995 load/load 4996 atomic/store/store 4997 atomic/atomicrmw. 4998 - Ensures any 4999 following global 5000 data read is no 5001 older than a local 5002 atomicrmw value 5003 being acquired. 5004 5005 atomicrmw acquire - agent - global 1. buffer/global_atomic 5006 - system 2. s_waitcnt vmcnt(0) 5007 5008 - Must happen before 5009 following 5010 buffer_wbinvl1_vol. 5011 - Ensures the 5012 atomicrmw has 5013 completed before 5014 invalidating the 5015 cache. 5016 5017 3. buffer_wbinvl1_vol 5018 5019 - Must happen before 5020 any following 5021 global/generic 5022 load/load 5023 atomic/atomicrmw. 5024 - Ensures that 5025 following loads 5026 will not see stale 5027 global data. 5028 5029 atomicrmw acquire - agent - generic 1. flat_atomic 5030 - system 2. s_waitcnt vmcnt(0) & 5031 lgkmcnt(0) 5032 5033 - If OpenCL, omit 5034 lgkmcnt(0). 5035 - Must happen before 5036 following 5037 buffer_wbinvl1_vol. 5038 - Ensures the 5039 atomicrmw has 5040 completed before 5041 invalidating the 5042 cache. 5043 5044 3. buffer_wbinvl1_vol 5045 5046 - Must happen before 5047 any following 5048 global/generic 5049 load/load 5050 atomic/atomicrmw. 5051 - Ensures that 5052 following loads 5053 will not see stale 5054 global data. 5055 5056 fence acquire - singlethread *none* *none* 5057 - wavefront 5058 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5059 5060 - If OpenCL and 5061 address space is 5062 not generic, omit. 5063 - However, since LLVM 5064 currently has no 5065 address space on 5066 the fence need to 5067 conservatively 5068 always generate. If 5069 fence had an 5070 address space then 5071 set to address 5072 space of OpenCL 5073 fence flag, or to 5074 generic if both 5075 local and global 5076 flags are 5077 specified. 5078 - Must happen after 5079 any preceding 5080 local/generic load 5081 atomic/atomicrmw 5082 with an equal or 5083 wider sync scope 5084 and memory ordering 5085 stronger than 5086 unordered (this is 5087 termed the 5088 fence-paired-atomic). 5089 - Must happen before 5090 any following 5091 global/generic 5092 load/load 5093 atomic/store/store 5094 atomic/atomicrmw. 5095 - Ensures any 5096 following global 5097 data read is no 5098 older than the 5099 value read by the 5100 fence-paired-atomic. 5101 5102 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5103 - system vmcnt(0) 5104 5105 - If OpenCL and 5106 address space is 5107 not generic, omit 5108 lgkmcnt(0). 5109 - However, since LLVM 5110 currently has no 5111 address space on 5112 the fence need to 5113 conservatively 5114 always generate 5115 (see comment for 5116 previous fence). 5117 - Could be split into 5118 separate s_waitcnt 5119 vmcnt(0) and 5120 s_waitcnt 5121 lgkmcnt(0) to allow 5122 them to be 5123 independently moved 5124 according to the 5125 following rules. 5126 - s_waitcnt vmcnt(0) 5127 must happen after 5128 any preceding 5129 global/generic load 5130 atomic/atomicrmw 5131 with an equal or 5132 wider sync scope 5133 and memory ordering 5134 stronger than 5135 unordered (this is 5136 termed the 5137 fence-paired-atomic). 5138 - s_waitcnt lgkmcnt(0) 5139 must happen after 5140 any preceding 5141 local/generic load 5142 atomic/atomicrmw 5143 with an equal or 5144 wider sync scope 5145 and memory ordering 5146 stronger than 5147 unordered (this is 5148 termed the 5149 fence-paired-atomic). 5150 - Must happen before 5151 the following 5152 buffer_wbinvl1_vol. 5153 - Ensures that the 5154 fence-paired atomic 5155 has completed 5156 before invalidating 5157 the 5158 cache. Therefore 5159 any following 5160 locations read must 5161 be no older than 5162 the value read by 5163 the 5164 fence-paired-atomic. 5165 5166 2. buffer_wbinvl1_vol 5167 5168 - Must happen before any 5169 following global/generic 5170 load/load 5171 atomic/store/store 5172 atomic/atomicrmw. 5173 - Ensures that 5174 following loads 5175 will not see stale 5176 global data. 5177 5178 **Release Atomic** 5179 ------------------------------------------------------------------------------------ 5180 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5181 - wavefront - local 5182 - generic 5183 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5184 - generic 5185 - If OpenCL, omit. 5186 - Must happen after 5187 any preceding 5188 local/generic 5189 load/store/load 5190 atomic/store 5191 atomic/atomicrmw. 5192 - Must happen before 5193 the following 5194 store. 5195 - Ensures that all 5196 memory operations 5197 to local have 5198 completed before 5199 performing the 5200 store that is being 5201 released. 5202 5203 2. buffer/global/flat_store 5204 store atomic release - workgroup - local 1. ds_store 5205 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5206 - system - generic vmcnt(0) 5207 5208 - If OpenCL and 5209 address space is 5210 not generic, omit 5211 lgkmcnt(0). 5212 - Could be split into 5213 separate s_waitcnt 5214 vmcnt(0) and 5215 s_waitcnt 5216 lgkmcnt(0) to allow 5217 them to be 5218 independently moved 5219 according to the 5220 following rules. 5221 - s_waitcnt vmcnt(0) 5222 must happen after 5223 any preceding 5224 global/generic 5225 load/store/load 5226 atomic/store 5227 atomic/atomicrmw. 5228 - s_waitcnt lgkmcnt(0) 5229 must happen after 5230 any preceding 5231 local/generic 5232 load/store/load 5233 atomic/store 5234 atomic/atomicrmw. 5235 - Must happen before 5236 the following 5237 store. 5238 - Ensures that all 5239 memory operations 5240 to memory have 5241 completed before 5242 performing the 5243 store that is being 5244 released. 5245 5246 2. buffer/global/flat_store 5247 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5248 - wavefront - local 5249 - generic 5250 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5251 - generic 5252 - If OpenCL, omit. 5253 - Must happen after 5254 any preceding 5255 local/generic 5256 load/store/load 5257 atomic/store 5258 atomic/atomicrmw. 5259 - Must happen before 5260 the following 5261 atomicrmw. 5262 - Ensures that all 5263 memory operations 5264 to local have 5265 completed before 5266 performing the 5267 atomicrmw that is 5268 being released. 5269 5270 2. buffer/global/flat_atomic 5271 atomicrmw release - workgroup - local 1. ds_atomic 5272 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5273 - system - generic vmcnt(0) 5274 5275 - If OpenCL, omit 5276 lgkmcnt(0). 5277 - Could be split into 5278 separate s_waitcnt 5279 vmcnt(0) and 5280 s_waitcnt 5281 lgkmcnt(0) to allow 5282 them to be 5283 independently moved 5284 according to the 5285 following rules. 5286 - s_waitcnt vmcnt(0) 5287 must happen after 5288 any preceding 5289 global/generic 5290 load/store/load 5291 atomic/store 5292 atomic/atomicrmw. 5293 - s_waitcnt lgkmcnt(0) 5294 must happen after 5295 any preceding 5296 local/generic 5297 load/store/load 5298 atomic/store 5299 atomic/atomicrmw. 5300 - Must happen before 5301 the following 5302 atomicrmw. 5303 - Ensures that all 5304 memory operations 5305 to global and local 5306 have completed 5307 before performing 5308 the atomicrmw that 5309 is being released. 5310 5311 2. buffer/global/flat_atomic 5312 fence release - singlethread *none* *none* 5313 - wavefront 5314 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5315 5316 - If OpenCL and 5317 address space is 5318 not generic, omit. 5319 - However, since LLVM 5320 currently has no 5321 address space on 5322 the fence need to 5323 conservatively 5324 always generate. If 5325 fence had an 5326 address space then 5327 set to address 5328 space of OpenCL 5329 fence flag, or to 5330 generic if both 5331 local and global 5332 flags are 5333 specified. 5334 - Must happen after 5335 any preceding 5336 local/generic 5337 load/load 5338 atomic/store/store 5339 atomic/atomicrmw. 5340 - Must happen before 5341 any following store 5342 atomic/atomicrmw 5343 with an equal or 5344 wider sync scope 5345 and memory ordering 5346 stronger than 5347 unordered (this is 5348 termed the 5349 fence-paired-atomic). 5350 - Ensures that all 5351 memory operations 5352 to local have 5353 completed before 5354 performing the 5355 following 5356 fence-paired-atomic. 5357 5358 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5359 - system vmcnt(0) 5360 5361 - If OpenCL and 5362 address space is 5363 not generic, omit 5364 lgkmcnt(0). 5365 - If OpenCL and 5366 address space is 5367 local, omit 5368 vmcnt(0). 5369 - However, since LLVM 5370 currently has no 5371 address space on 5372 the fence need to 5373 conservatively 5374 always generate. If 5375 fence had an 5376 address space then 5377 set to address 5378 space of OpenCL 5379 fence flag, or to 5380 generic if both 5381 local and global 5382 flags are 5383 specified. 5384 - Could be split into 5385 separate s_waitcnt 5386 vmcnt(0) and 5387 s_waitcnt 5388 lgkmcnt(0) to allow 5389 them to be 5390 independently moved 5391 according to the 5392 following rules. 5393 - s_waitcnt vmcnt(0) 5394 must happen after 5395 any preceding 5396 global/generic 5397 load/store/load 5398 atomic/store 5399 atomic/atomicrmw. 5400 - s_waitcnt lgkmcnt(0) 5401 must happen after 5402 any preceding 5403 local/generic 5404 load/store/load 5405 atomic/store 5406 atomic/atomicrmw. 5407 - Must happen before 5408 any following store 5409 atomic/atomicrmw 5410 with an equal or 5411 wider sync scope 5412 and memory ordering 5413 stronger than 5414 unordered (this is 5415 termed the 5416 fence-paired-atomic). 5417 - Ensures that all 5418 memory operations 5419 have 5420 completed before 5421 performing the 5422 following 5423 fence-paired-atomic. 5424 5425 **Acquire-Release Atomic** 5426 ------------------------------------------------------------------------------------ 5427 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5428 - wavefront - local 5429 - generic 5430 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5431 5432 - If OpenCL, omit. 5433 - Must happen after 5434 any preceding 5435 local/generic 5436 load/store/load 5437 atomic/store 5438 atomic/atomicrmw. 5439 - Must happen before 5440 the following 5441 atomicrmw. 5442 - Ensures that all 5443 memory operations 5444 to local have 5445 completed before 5446 performing the 5447 atomicrmw that is 5448 being released. 5449 5450 2. buffer/global_atomic 5451 5452 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5453 2. s_waitcnt lgkmcnt(0) 5454 5455 - If OpenCL, omit. 5456 - Must happen before 5457 any following 5458 global/generic 5459 load/load 5460 atomic/store/store 5461 atomic/atomicrmw. 5462 - Ensures any 5463 following global 5464 data read is no 5465 older than the local load 5466 atomic value being 5467 acquired. 5468 5469 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5470 5471 - If OpenCL, omit. 5472 - Must happen after 5473 any preceding 5474 local/generic 5475 load/store/load 5476 atomic/store 5477 atomic/atomicrmw. 5478 - Must happen before 5479 the following 5480 atomicrmw. 5481 - Ensures that all 5482 memory operations 5483 to local have 5484 completed before 5485 performing the 5486 atomicrmw that is 5487 being released. 5488 5489 2. flat_atomic 5490 3. s_waitcnt lgkmcnt(0) 5491 5492 - If OpenCL, omit. 5493 - Must happen before 5494 any following 5495 global/generic 5496 load/load 5497 atomic/store/store 5498 atomic/atomicrmw. 5499 - Ensures any 5500 following global 5501 data read is no 5502 older than a local load 5503 atomic value being 5504 acquired. 5505 5506 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5507 - system vmcnt(0) 5508 5509 - If OpenCL, omit 5510 lgkmcnt(0). 5511 - Could be split into 5512 separate s_waitcnt 5513 vmcnt(0) and 5514 s_waitcnt 5515 lgkmcnt(0) to allow 5516 them to be 5517 independently moved 5518 according to the 5519 following rules. 5520 - s_waitcnt vmcnt(0) 5521 must happen after 5522 any preceding 5523 global/generic 5524 load/store/load 5525 atomic/store 5526 atomic/atomicrmw. 5527 - s_waitcnt lgkmcnt(0) 5528 must happen after 5529 any preceding 5530 local/generic 5531 load/store/load 5532 atomic/store 5533 atomic/atomicrmw. 5534 - Must happen before 5535 the following 5536 atomicrmw. 5537 - Ensures that all 5538 memory operations 5539 to global have 5540 completed before 5541 performing the 5542 atomicrmw that is 5543 being released. 5544 5545 2. buffer/global_atomic 5546 3. s_waitcnt vmcnt(0) 5547 5548 - Must happen before 5549 following 5550 buffer_wbinvl1_vol. 5551 - Ensures the 5552 atomicrmw has 5553 completed before 5554 invalidating the 5555 cache. 5556 5557 4. buffer_wbinvl1_vol 5558 5559 - Must happen before 5560 any following 5561 global/generic 5562 load/load 5563 atomic/atomicrmw. 5564 - Ensures that 5565 following loads 5566 will not see stale 5567 global data. 5568 5569 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5570 - system vmcnt(0) 5571 5572 - If OpenCL, omit 5573 lgkmcnt(0). 5574 - Could be split into 5575 separate s_waitcnt 5576 vmcnt(0) and 5577 s_waitcnt 5578 lgkmcnt(0) to allow 5579 them to be 5580 independently moved 5581 according to the 5582 following rules. 5583 - s_waitcnt vmcnt(0) 5584 must happen after 5585 any preceding 5586 global/generic 5587 load/store/load 5588 atomic/store 5589 atomic/atomicrmw. 5590 - s_waitcnt lgkmcnt(0) 5591 must happen after 5592 any preceding 5593 local/generic 5594 load/store/load 5595 atomic/store 5596 atomic/atomicrmw. 5597 - Must happen before 5598 the following 5599 atomicrmw. 5600 - Ensures that all 5601 memory operations 5602 to global have 5603 completed before 5604 performing the 5605 atomicrmw that is 5606 being released. 5607 5608 2. flat_atomic 5609 3. s_waitcnt vmcnt(0) & 5610 lgkmcnt(0) 5611 5612 - If OpenCL, omit 5613 lgkmcnt(0). 5614 - Must happen before 5615 following 5616 buffer_wbinvl1_vol. 5617 - Ensures the 5618 atomicrmw has 5619 completed before 5620 invalidating the 5621 cache. 5622 5623 4. buffer_wbinvl1_vol 5624 5625 - Must happen before 5626 any following 5627 global/generic 5628 load/load 5629 atomic/atomicrmw. 5630 - Ensures that 5631 following loads 5632 will not see stale 5633 global data. 5634 5635 fence acq_rel - singlethread *none* *none* 5636 - wavefront 5637 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5638 5639 - If OpenCL and 5640 address space is 5641 not generic, omit. 5642 - However, 5643 since LLVM 5644 currently has no 5645 address space on 5646 the fence need to 5647 conservatively 5648 always generate 5649 (see comment for 5650 previous fence). 5651 - Must happen after 5652 any preceding 5653 local/generic 5654 load/load 5655 atomic/store/store 5656 atomic/atomicrmw. 5657 - Must happen before 5658 any following 5659 global/generic 5660 load/load 5661 atomic/store/store 5662 atomic/atomicrmw. 5663 - Ensures that all 5664 memory operations 5665 to local have 5666 completed before 5667 performing any 5668 following global 5669 memory operations. 5670 - Ensures that the 5671 preceding 5672 local/generic load 5673 atomic/atomicrmw 5674 with an equal or 5675 wider sync scope 5676 and memory ordering 5677 stronger than 5678 unordered (this is 5679 termed the 5680 acquire-fence-paired-atomic) 5681 has completed 5682 before following 5683 global memory 5684 operations. This 5685 satisfies the 5686 requirements of 5687 acquire. 5688 - Ensures that all 5689 previous memory 5690 operations have 5691 completed before a 5692 following 5693 local/generic store 5694 atomic/atomicrmw 5695 with an equal or 5696 wider sync scope 5697 and memory ordering 5698 stronger than 5699 unordered (this is 5700 termed the 5701 release-fence-paired-atomic). 5702 This satisfies the 5703 requirements of 5704 release. 5705 5706 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5707 - system vmcnt(0) 5708 5709 - If OpenCL and 5710 address space is 5711 not generic, omit 5712 lgkmcnt(0). 5713 - However, since LLVM 5714 currently has no 5715 address space on 5716 the fence need to 5717 conservatively 5718 always generate 5719 (see comment for 5720 previous fence). 5721 - Could be split into 5722 separate s_waitcnt 5723 vmcnt(0) and 5724 s_waitcnt 5725 lgkmcnt(0) to allow 5726 them to be 5727 independently moved 5728 according to the 5729 following rules. 5730 - s_waitcnt vmcnt(0) 5731 must happen after 5732 any preceding 5733 global/generic 5734 load/store/load 5735 atomic/store 5736 atomic/atomicrmw. 5737 - s_waitcnt lgkmcnt(0) 5738 must happen after 5739 any preceding 5740 local/generic 5741 load/store/load 5742 atomic/store 5743 atomic/atomicrmw. 5744 - Must happen before 5745 the following 5746 buffer_wbinvl1_vol. 5747 - Ensures that the 5748 preceding 5749 global/local/generic 5750 load 5751 atomic/atomicrmw 5752 with an equal or 5753 wider sync scope 5754 and memory ordering 5755 stronger than 5756 unordered (this is 5757 termed the 5758 acquire-fence-paired-atomic) 5759 has completed 5760 before invalidating 5761 the cache. This 5762 satisfies the 5763 requirements of 5764 acquire. 5765 - Ensures that all 5766 previous memory 5767 operations have 5768 completed before a 5769 following 5770 global/local/generic 5771 store 5772 atomic/atomicrmw 5773 with an equal or 5774 wider sync scope 5775 and memory ordering 5776 stronger than 5777 unordered (this is 5778 termed the 5779 release-fence-paired-atomic). 5780 This satisfies the 5781 requirements of 5782 release. 5783 5784 2. buffer_wbinvl1_vol 5785 5786 - Must happen before 5787 any following 5788 global/generic 5789 load/load 5790 atomic/store/store 5791 atomic/atomicrmw. 5792 - Ensures that 5793 following loads 5794 will not see stale 5795 global data. This 5796 satisfies the 5797 requirements of 5798 acquire. 5799 5800 **Sequential Consistent Atomic** 5801 ------------------------------------------------------------------------------------ 5802 load atomic seq_cst - singlethread - global *Same as corresponding 5803 - wavefront - local load atomic acquire, 5804 - generic except must generated 5805 all instructions even 5806 for OpenCL.* 5807 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5808 - generic 5809 5810 - Must 5811 happen after 5812 preceding 5813 local/generic load 5814 atomic/store 5815 atomic/atomicrmw 5816 with memory 5817 ordering of seq_cst 5818 and with equal or 5819 wider sync scope. 5820 (Note that seq_cst 5821 fences have their 5822 own s_waitcnt 5823 lgkmcnt(0) and so do 5824 not need to be 5825 considered.) 5826 - Ensures any 5827 preceding 5828 sequential 5829 consistent local 5830 memory instructions 5831 have completed 5832 before executing 5833 this sequentially 5834 consistent 5835 instruction. This 5836 prevents reordering 5837 a seq_cst store 5838 followed by a 5839 seq_cst load. (Note 5840 that seq_cst is 5841 stronger than 5842 acquire/release as 5843 the reordering of 5844 load acquire 5845 followed by a store 5846 release is 5847 prevented by the 5848 s_waitcnt of 5849 the release, but 5850 there is nothing 5851 preventing a store 5852 release followed by 5853 load acquire from 5854 completing out of 5855 order. The s_waitcnt 5856 could be placed after 5857 seq_store or before 5858 the seq_load. We 5859 choose the load to 5860 make the s_waitcnt be 5861 as late as possible 5862 so that the store 5863 may have already 5864 completed.) 5865 5866 2. *Following 5867 instructions same as 5868 corresponding load 5869 atomic acquire, 5870 except must generated 5871 all instructions even 5872 for OpenCL.* 5873 load atomic seq_cst - workgroup - local *Same as corresponding 5874 load atomic acquire, 5875 except must generated 5876 all instructions even 5877 for OpenCL.* 5878 5879 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5880 - system - generic vmcnt(0) 5881 5882 - Could be split into 5883 separate s_waitcnt 5884 vmcnt(0) 5885 and s_waitcnt 5886 lgkmcnt(0) to allow 5887 them to be 5888 independently moved 5889 according to the 5890 following rules. 5891 - s_waitcnt lgkmcnt(0) 5892 must happen after 5893 preceding 5894 global/generic load 5895 atomic/store 5896 atomic/atomicrmw 5897 with memory 5898 ordering of seq_cst 5899 and with equal or 5900 wider sync scope. 5901 (Note that seq_cst 5902 fences have their 5903 own s_waitcnt 5904 lgkmcnt(0) and so do 5905 not need to be 5906 considered.) 5907 - s_waitcnt vmcnt(0) 5908 must happen after 5909 preceding 5910 global/generic load 5911 atomic/store 5912 atomic/atomicrmw 5913 with memory 5914 ordering of seq_cst 5915 and with equal or 5916 wider sync scope. 5917 (Note that seq_cst 5918 fences have their 5919 own s_waitcnt 5920 vmcnt(0) and so do 5921 not need to be 5922 considered.) 5923 - Ensures any 5924 preceding 5925 sequential 5926 consistent global 5927 memory instructions 5928 have completed 5929 before executing 5930 this sequentially 5931 consistent 5932 instruction. This 5933 prevents reordering 5934 a seq_cst store 5935 followed by a 5936 seq_cst load. (Note 5937 that seq_cst is 5938 stronger than 5939 acquire/release as 5940 the reordering of 5941 load acquire 5942 followed by a store 5943 release is 5944 prevented by the 5945 s_waitcnt of 5946 the release, but 5947 there is nothing 5948 preventing a store 5949 release followed by 5950 load acquire from 5951 completing out of 5952 order. The s_waitcnt 5953 could be placed after 5954 seq_store or before 5955 the seq_load. We 5956 choose the load to 5957 make the s_waitcnt be 5958 as late as possible 5959 so that the store 5960 may have already 5961 completed.) 5962 5963 2. *Following 5964 instructions same as 5965 corresponding load 5966 atomic acquire, 5967 except must generated 5968 all instructions even 5969 for OpenCL.* 5970 store atomic seq_cst - singlethread - global *Same as corresponding 5971 - wavefront - local store atomic release, 5972 - workgroup - generic except must generated 5973 - agent all instructions even 5974 - system for OpenCL.* 5975 atomicrmw seq_cst - singlethread - global *Same as corresponding 5976 - wavefront - local atomicrmw acq_rel, 5977 - workgroup - generic except must generated 5978 - agent all instructions even 5979 - system for OpenCL.* 5980 fence seq_cst - singlethread *none* *Same as corresponding 5981 - wavefront fence acq_rel, 5982 - workgroup except must generated 5983 - agent all instructions even 5984 - system for OpenCL.* 5985 ============ ============ ============== ========== ================================ 5986 5987.. _amdgpu-amdhsa-memory-model-gfx90a: 5988 5989Memory Model GFX90A 5990+++++++++++++++++++ 5991 5992For GFX90A: 5993 5994* Each agent has multiple shader arrays (SA). 5995* Each SA has multiple compute units (CU). 5996* Each CU has multiple SIMDs that execute wavefronts. 5997* The wavefronts for a single work-group are executed in the same CU but may be 5998 executed by different SIMDs. The exception is when in tgsplit execution mode 5999 when the wavefronts may be executed by different SIMDs in different CUs. 6000* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6001 executing on it. The exception is when in tgsplit execution mode when no LDS 6002 is allocated as wavefronts of the same work-group can be in different CUs. 6003* All LDS operations of a CU are performed as wavefront wide operations in a 6004 global order and involve no caching. Completion is reported to a wavefront in 6005 execution order. 6006* The LDS memory has multiple request queues shared by the SIMDs of a 6007 CU. Therefore, the LDS operations performed by different wavefronts of a 6008 work-group can be reordered relative to each other, which can result in 6009 reordering the visibility of vector memory operations with respect to LDS 6010 operations of other wavefronts in the same work-group. A ``s_waitcnt 6011 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6012 vector memory operations between wavefronts of a work-group, but not between 6013 operations performed by the same wavefront. 6014* The vector memory operations are performed as wavefront wide operations and 6015 completion is reported to a wavefront in execution order. The exception is 6016 that ``flat_load/store/atomic`` instructions can report out of vector memory 6017 order if they access LDS memory, and out of LDS operation order if they access 6018 global memory. 6019* The vector memory operations access a single vector L1 cache shared by all 6020 SIMDs a CU. Therefore: 6021 6022 * No special action is required for coherence between the lanes of a single 6023 wavefront. 6024 6025 * No special action is required for coherence between wavefronts in the same 6026 work-group since they execute on the same CU. The exception is when in 6027 tgsplit execution mode as wavefronts of the same work-group can be in 6028 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6029 the following item. 6030 6031 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6032 executing in different work-groups as they may be executing on different 6033 CUs. 6034 6035* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6036 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6037 scalar operations are used in a restricted way so do not impact the memory 6038 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6039* The vector and scalar memory operations use an L2 cache shared by all CUs on 6040 the same agent. 6041 6042 * The L2 cache has independent channels to service disjoint ranges of virtual 6043 addresses. 6044 * Each CU has a separate request queue per channel. Therefore, the vector and 6045 scalar memory operations performed by wavefronts executing in different 6046 work-groups (which may be executing on different CUs), or the same 6047 work-group if executing in tgsplit mode, of an agent can be reordered 6048 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6049 synchronization between vector memory operations of different CUs. It 6050 ensures a previous vector memory operation has completed before executing a 6051 subsequent vector memory or LDS operation and so can be used to meet the 6052 requirements of acquire and release. 6053 * The L2 cache of one agent can be kept coherent with other agents by using 6054 the MTYPE RW (read-write) for memory local to the L2, and MTYPE NC 6055 (non-coherent) with the PTE C-bit set for memory not local to the L2. 6056 6057 * Any local memory cache lines will be automatically invalidated by writes 6058 from CUs associated with other L2 caches, or writes from the CPU, due to 6059 the cache probe caused by the PTE C-bit. 6060 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6061 Subsequent access from the GPU will automatically invalidate or writeback 6062 the CPU cache due to the L2 probe filter. 6063 * Since all work-groups on the same agent share the same L2, no L2 6064 invalidation or writeback is required for coherence. 6065 * To ensure coherence of local memory writes of work-groups in different 6066 agents a ``buffer_wbl2`` is required. It will writeback dirty L2 cache 6067 lines. 6068 * To ensure coherence of local memory reads of work-groups in different 6069 agents a ``buffer_invl2`` is required. It will invalidate non-local L2 6070 cache lines. 6071 6072 * PCIe access from the GPU to the CPU memory can be kept coherent by using the 6073 MTYPE UC (uncached) which bypasses the L2. 6074 6075Scalar memory operations are only used to access memory that is proven to not 6076change during the execution of the kernel dispatch. This includes constant 6077address space and global address space for program scope ``const`` variables. 6078Therefore, the kernel machine code does not have to maintain the scalar cache to 6079ensure it is coherent with the vector caches. The scalar and vector caches are 6080invalidated between kernel dispatches by CP since constant address space data 6081may change between kernel dispatch executions. See 6082:ref:`amdgpu-amdhsa-memory-spaces`. 6083 6084The one exception is if scalar writes are used to spill SGPR registers. In this 6085case the AMDGPU backend ensures the memory location used to spill is never 6086accessed by vector memory operations at the same time. If scalar writes are used 6087then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6088return since the locations may be used for vector memory instructions by a 6089future wavefront that uses the same scratch area, or a function call that 6090creates a frame at the same address, respectively. There is no need for a 6091``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6092 6093For kernarg backing memory: 6094 6095* CP invalidates the L1 cache at the start of each kernel dispatch. 6096* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6097 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6098 cache. This also causes it to be treated as non-volatile and so is not 6099 invalidated by ``*_vol``. 6100* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6101 so the L2 cache will be coherent with the CPU and other agents. 6102 6103Scratch backing memory (which is used for the private address space) is accessed 6104with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6105only accessed by a single thread, and is always write-before-read, there is 6106never a need to invalidate these entries from the L1 cache. Hence all cache 6107invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6108 6109The code sequences used to implement the memory model for GFX90A are defined 6110in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6111 6112 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6113 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6114 6115 ============ ============ ============== ========== ================================ 6116 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6117 Ordering Sync Scope Address GFX90A 6118 Space 6119 ============ ============ ============== ========== ================================ 6120 **Non-Atomic** 6121 ------------------------------------------------------------------------------------ 6122 load *none* *none* - global - !volatile & !nontemporal 6123 - generic 6124 - private 1. buffer/global/flat_load 6125 - constant 6126 - !volatile & nontemporal 6127 6128 1. buffer/global/flat_load 6129 glc=1 slc=1 6130 6131 - volatile 6132 6133 1. buffer/global/flat_load 6134 glc=1 scc=1 6135 2. s_waitcnt vmcnt(0) 6136 6137 - Must happen before 6138 any following volatile 6139 global/generic 6140 load/store. 6141 - Ensures that 6142 volatile 6143 operations to 6144 different 6145 addresses will not 6146 be reordered by 6147 hardware. 6148 6149 load *none* *none* - local 1. ds_load 6150 store *none* *none* - global - !volatile & !nontemporal 6151 - generic 6152 - private 1. buffer/global/flat_store 6153 - constant 6154 - !volatile & nontemporal 6155 6156 1. buffer/global/flat_store 6157 glc=1 slc=1 6158 6159 - volatile 6160 6161 1. buffer/global/flat_store 6162 scc=1 6163 2. s_waitcnt vmcnt(0) 6164 6165 - Must happen before 6166 any following volatile 6167 global/generic 6168 load/store. 6169 - Ensures that 6170 volatile 6171 operations to 6172 different 6173 addresses will not 6174 be reordered by 6175 hardware. 6176 6177 store *none* *none* - local 1. ds_store 6178 **Unordered Atomic** 6179 ------------------------------------------------------------------------------------ 6180 load atomic unordered *any* *any* *Same as non-atomic*. 6181 store atomic unordered *any* *any* *Same as non-atomic*. 6182 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6183 **Monotonic Atomic** 6184 ------------------------------------------------------------------------------------ 6185 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6186 - wavefront - generic 6187 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6188 - generic glc=1 6189 6190 - If not TgSplit execution 6191 mode, omit glc=1. 6192 6193 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6194 - wavefront local address space cannot 6195 - workgroup be used.* 6196 6197 1. ds_load 6198 load atomic monotonic - agent - global 1. buffer/global/flat_load 6199 - generic glc=1 6200 load atomic monotonic - system - global 1. buffer/global/flat_load 6201 - generic glc=1 scc=1 6202 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6203 - wavefront - generic 6204 - workgroup 6205 - agent 6206 store atomic monotonic - system - global 1. buffer/global/flat_store 6207 - generic scc=1 6208 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6209 - wavefront local address space cannot 6210 - workgroup be used.* 6211 6212 1. ds_store 6213 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6214 - wavefront - generic 6215 - workgroup 6216 - agent 6217 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6218 - generic scc=1 6219 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6220 - wavefront local address space cannot 6221 - workgroup be used.* 6222 6223 1. ds_atomic 6224 **Acquire Atomic** 6225 ------------------------------------------------------------------------------------ 6226 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6227 - wavefront - local 6228 - generic 6229 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6230 6231 - If not TgSplit execution 6232 mode, omit glc=1. 6233 6234 2. s_waitcnt vmcnt(0) 6235 6236 - If not TgSplit execution 6237 mode, omit. 6238 - Must happen before the 6239 following buffer_wbinvl1_vol. 6240 6241 3. buffer_wbinvl1_vol 6242 6243 - If not TgSplit execution 6244 mode, omit. 6245 - Must happen before 6246 any following 6247 global/generic 6248 load/load 6249 atomic/store/store 6250 atomic/atomicrmw. 6251 - Ensures that 6252 following 6253 loads will not see 6254 stale data. 6255 6256 load atomic acquire - workgroup - local *If TgSplit execution mode, 6257 local address space cannot 6258 be used.* 6259 6260 1. ds_load 6261 2. s_waitcnt lgkmcnt(0) 6262 6263 - If OpenCL, omit. 6264 - Must happen before 6265 any following 6266 global/generic 6267 load/load 6268 atomic/store/store 6269 atomic/atomicrmw. 6270 - Ensures any 6271 following global 6272 data read is no 6273 older than the local load 6274 atomic value being 6275 acquired. 6276 6277 load atomic acquire - workgroup - generic 1. flat_load glc=1 6278 6279 - If not TgSplit execution 6280 mode, omit glc=1. 6281 6282 2. s_waitcnt lgkm/vmcnt(0) 6283 6284 - Use lgkmcnt(0) if not 6285 TgSplit execution mode 6286 and vmcnt(0) if TgSplit 6287 execution mode. 6288 - If OpenCL, omit lgkmcnt(0). 6289 - Must happen before 6290 the following 6291 buffer_wbinvl1_vol and any 6292 following global/generic 6293 load/load 6294 atomic/store/store 6295 atomic/atomicrmw. 6296 - Ensures any 6297 following global 6298 data read is no 6299 older than a local load 6300 atomic value being 6301 acquired. 6302 6303 3. buffer_wbinvl1_vol 6304 6305 - If not TgSplit execution 6306 mode, omit. 6307 - Ensures that 6308 following 6309 loads will not see 6310 stale data. 6311 6312 load atomic acquire - agent - global 1. buffer/global_load 6313 glc=1 6314 2. s_waitcnt vmcnt(0) 6315 6316 - Must happen before 6317 following 6318 buffer_wbinvl1_vol. 6319 - Ensures the load 6320 has completed 6321 before invalidating 6322 the cache. 6323 6324 3. buffer_wbinvl1_vol 6325 6326 - Must happen before 6327 any following 6328 global/generic 6329 load/load 6330 atomic/atomicrmw. 6331 - Ensures that 6332 following 6333 loads will not see 6334 stale global data. 6335 6336 load atomic acquire - system - global 1. buffer/global/flat_load 6337 glc=1 scc=1 6338 2. s_waitcnt vmcnt(0) 6339 6340 - Must happen before 6341 following buffer_invl2 and 6342 buffer_wbinvl1_vol. 6343 - Ensures the load 6344 has completed 6345 before invalidating 6346 the cache. 6347 6348 3. buffer_invl2; 6349 buffer_wbinvl1_vol 6350 6351 - Must happen before 6352 any following 6353 global/generic 6354 load/load 6355 atomic/atomicrmw. 6356 - Ensures that 6357 following 6358 loads will not see 6359 stale MTYPE NC global data. 6360 MTYPE RW and CC memory will 6361 never be stale due to the 6362 memory probes. 6363 6364 load atomic acquire - agent - generic 1. flat_load glc=1 6365 2. s_waitcnt vmcnt(0) & 6366 lgkmcnt(0) 6367 6368 - If TgSplit execution mode, 6369 omit lgkmcnt(0). 6370 - If OpenCL omit 6371 lgkmcnt(0). 6372 - Must happen before 6373 following 6374 buffer_wbinvl1_vol. 6375 - Ensures the flat_load 6376 has completed 6377 before invalidating 6378 the cache. 6379 6380 3. buffer_wbinvl1_vol 6381 6382 - Must happen before 6383 any following 6384 global/generic 6385 load/load 6386 atomic/atomicrmw. 6387 - Ensures that 6388 following loads 6389 will not see stale 6390 global data. 6391 6392 load atomic acquire - system - generic 1. flat_load glc=1 scc=1 6393 2. s_waitcnt vmcnt(0) & 6394 lgkmcnt(0) 6395 6396 - If TgSplit execution mode, 6397 omit lgkmcnt(0). 6398 - If OpenCL omit 6399 lgkmcnt(0). 6400 - Must happen before 6401 following 6402 buffer_invl2 and 6403 buffer_wbinvl1_vol. 6404 - Ensures the flat_load 6405 has completed 6406 before invalidating 6407 the caches. 6408 6409 3. buffer_invl2; 6410 buffer_wbinvl1_vol 6411 6412 - Must happen before 6413 any following 6414 global/generic 6415 load/load 6416 atomic/atomicrmw. 6417 - Ensures that 6418 following 6419 loads will not see 6420 stale MTYPE NC global data. 6421 MTYPE RW and CC memory will 6422 never be stale due to the 6423 memory probes. 6424 6425 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6426 - wavefront - generic 6427 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6428 - wavefront local address space cannot 6429 be used.* 6430 6431 1. ds_atomic 6432 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6433 2. s_waitcnt vmcnt(0) 6434 6435 - If not TgSplit execution 6436 mode, omit. 6437 - Must happen before the 6438 following buffer_wbinvl1_vol. 6439 - Ensures the atomicrmw 6440 has completed 6441 before invalidating 6442 the cache. 6443 6444 3. buffer_wbinvl1_vol 6445 6446 - If not TgSplit execution 6447 mode, omit. 6448 - Must happen before 6449 any following 6450 global/generic 6451 load/load 6452 atomic/atomicrmw. 6453 - Ensures that 6454 following loads 6455 will not see stale 6456 global data. 6457 6458 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6459 local address space cannot 6460 be used.* 6461 6462 1. ds_atomic 6463 2. s_waitcnt lgkmcnt(0) 6464 6465 - If OpenCL, omit. 6466 - Must happen before 6467 any following 6468 global/generic 6469 load/load 6470 atomic/store/store 6471 atomic/atomicrmw. 6472 - Ensures any 6473 following global 6474 data read is no 6475 older than the local 6476 atomicrmw value 6477 being acquired. 6478 6479 atomicrmw acquire - workgroup - generic 1. flat_atomic 6480 2. s_waitcnt lgkm/vmcnt(0) 6481 6482 - Use lgkmcnt(0) if not 6483 TgSplit execution mode 6484 and vmcnt(0) if TgSplit 6485 execution mode. 6486 - If OpenCL, omit lgkmcnt(0). 6487 - Must happen before 6488 the following 6489 buffer_wbinvl1_vol and 6490 any following 6491 global/generic 6492 load/load 6493 atomic/store/store 6494 atomic/atomicrmw. 6495 - Ensures any 6496 following global 6497 data read is no 6498 older than a local 6499 atomicrmw value 6500 being acquired. 6501 6502 3. buffer_wbinvl1_vol 6503 6504 - If not TgSplit execution 6505 mode, omit. 6506 - Ensures that 6507 following 6508 loads will not see 6509 stale data. 6510 6511 atomicrmw acquire - agent - global 1. buffer/global_atomic 6512 2. s_waitcnt vmcnt(0) 6513 6514 - Must happen before 6515 following 6516 buffer_wbinvl1_vol. 6517 - Ensures the 6518 atomicrmw has 6519 completed before 6520 invalidating the 6521 cache. 6522 6523 3. buffer_wbinvl1_vol 6524 6525 - Must happen before 6526 any following 6527 global/generic 6528 load/load 6529 atomic/atomicrmw. 6530 - Ensures that 6531 following loads 6532 will not see stale 6533 global data. 6534 6535 atomicrmw acquire - system - global 1. buffer/global_atomic 6536 scc=1 6537 2. s_waitcnt vmcnt(0) 6538 6539 - Must happen before 6540 following buffer_invl2 and 6541 buffer_wbinvl1_vol. 6542 - Ensures the 6543 atomicrmw has 6544 completed before 6545 invalidating the 6546 caches. 6547 6548 3. buffer_invl2; 6549 buffer_wbinvl1_vol 6550 6551 - Must happen before 6552 any following 6553 global/generic 6554 load/load 6555 atomic/atomicrmw. 6556 - Ensures that 6557 following 6558 loads will not see 6559 stale MTYPE NC global data. 6560 MTYPE RW and CC memory will 6561 never be stale due to the 6562 memory probes. 6563 6564 atomicrmw acquire - agent - generic 1. flat_atomic 6565 2. s_waitcnt vmcnt(0) & 6566 lgkmcnt(0) 6567 6568 - If TgSplit execution mode, 6569 omit lgkmcnt(0). 6570 - If OpenCL, omit 6571 lgkmcnt(0). 6572 - Must happen before 6573 following 6574 buffer_wbinvl1_vol. 6575 - Ensures the 6576 atomicrmw has 6577 completed before 6578 invalidating the 6579 cache. 6580 6581 3. buffer_wbinvl1_vol 6582 6583 - Must happen before 6584 any following 6585 global/generic 6586 load/load 6587 atomic/atomicrmw. 6588 - Ensures that 6589 following loads 6590 will not see stale 6591 global data. 6592 6593 atomicrmw acquire - system - generic 1. flat_atomic scc=1 6594 2. s_waitcnt vmcnt(0) & 6595 lgkmcnt(0) 6596 6597 - If TgSplit execution mode, 6598 omit lgkmcnt(0). 6599 - If OpenCL, omit 6600 lgkmcnt(0). 6601 - Must happen before 6602 following 6603 buffer_invl2 and 6604 buffer_wbinvl1_vol. 6605 - Ensures the 6606 atomicrmw has 6607 completed before 6608 invalidating the 6609 caches. 6610 6611 3. buffer_invl2; 6612 buffer_wbinvl1_vol 6613 6614 - Must happen before 6615 any following 6616 global/generic 6617 load/load 6618 atomic/atomicrmw. 6619 - Ensures that 6620 following 6621 loads will not see 6622 stale MTYPE NC global data. 6623 MTYPE RW and CC memory will 6624 never be stale due to the 6625 memory probes. 6626 6627 fence acquire - singlethread *none* *none* 6628 - wavefront 6629 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6630 6631 - Use lgkmcnt(0) if not 6632 TgSplit execution mode 6633 and vmcnt(0) if TgSplit 6634 execution mode. 6635 - If OpenCL and 6636 address space is 6637 not generic, omit 6638 lgkmcnt(0). 6639 - If OpenCL and 6640 address space is 6641 local, omit 6642 vmcnt(0). 6643 - However, since LLVM 6644 currently has no 6645 address space on 6646 the fence need to 6647 conservatively 6648 always generate. If 6649 fence had an 6650 address space then 6651 set to address 6652 space of OpenCL 6653 fence flag, or to 6654 generic if both 6655 local and global 6656 flags are 6657 specified. 6658 - s_waitcnt vmcnt(0) 6659 must happen after 6660 any preceding 6661 global/generic load 6662 atomic/ 6663 atomicrmw 6664 with an equal or 6665 wider sync scope 6666 and memory ordering 6667 stronger than 6668 unordered (this is 6669 termed the 6670 fence-paired-atomic). 6671 - s_waitcnt lgkmcnt(0) 6672 must happen after 6673 any preceding 6674 local/generic load 6675 atomic/atomicrmw 6676 with an equal or 6677 wider sync scope 6678 and memory ordering 6679 stronger than 6680 unordered (this is 6681 termed the 6682 fence-paired-atomic). 6683 - Must happen before 6684 the following 6685 buffer_wbinvl1_vol and 6686 any following 6687 global/generic 6688 load/load 6689 atomic/store/store 6690 atomic/atomicrmw. 6691 - Ensures any 6692 following global 6693 data read is no 6694 older than the 6695 value read by the 6696 fence-paired-atomic. 6697 6698 3. buffer_wbinvl1_vol 6699 6700 - If not TgSplit execution 6701 mode, omit. 6702 - Ensures that 6703 following 6704 loads will not see 6705 stale data. 6706 6707 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6708 vmcnt(0) 6709 6710 - If TgSplit execution mode, 6711 omit lgkmcnt(0). 6712 - If OpenCL and 6713 address space is 6714 not generic, omit 6715 lgkmcnt(0). 6716 - However, since LLVM 6717 currently has no 6718 address space on 6719 the fence need to 6720 conservatively 6721 always generate 6722 (see comment for 6723 previous fence). 6724 - Could be split into 6725 separate s_waitcnt 6726 vmcnt(0) and 6727 s_waitcnt 6728 lgkmcnt(0) to allow 6729 them to be 6730 independently moved 6731 according to the 6732 following rules. 6733 - s_waitcnt vmcnt(0) 6734 must happen after 6735 any preceding 6736 global/generic load 6737 atomic/atomicrmw 6738 with an equal or 6739 wider sync scope 6740 and memory ordering 6741 stronger than 6742 unordered (this is 6743 termed the 6744 fence-paired-atomic). 6745 - s_waitcnt lgkmcnt(0) 6746 must happen after 6747 any preceding 6748 local/generic load 6749 atomic/atomicrmw 6750 with an equal or 6751 wider sync scope 6752 and memory ordering 6753 stronger than 6754 unordered (this is 6755 termed the 6756 fence-paired-atomic). 6757 - Must happen before 6758 the following 6759 buffer_wbinvl1_vol. 6760 - Ensures that the 6761 fence-paired atomic 6762 has completed 6763 before invalidating 6764 the 6765 cache. Therefore 6766 any following 6767 locations read must 6768 be no older than 6769 the value read by 6770 the 6771 fence-paired-atomic. 6772 6773 2. buffer_wbinvl1_vol 6774 6775 - Must happen before any 6776 following global/generic 6777 load/load 6778 atomic/store/store 6779 atomic/atomicrmw. 6780 - Ensures that 6781 following loads 6782 will not see stale 6783 global data. 6784 6785 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6786 vmcnt(0) 6787 6788 - If TgSplit execution mode, 6789 omit lgkmcnt(0). 6790 - If OpenCL and 6791 address space is 6792 not generic, omit 6793 lgkmcnt(0). 6794 - However, since LLVM 6795 currently has no 6796 address space on 6797 the fence need to 6798 conservatively 6799 always generate 6800 (see comment for 6801 previous fence). 6802 - Could be split into 6803 separate s_waitcnt 6804 vmcnt(0) and 6805 s_waitcnt 6806 lgkmcnt(0) to allow 6807 them to be 6808 independently moved 6809 according to the 6810 following rules. 6811 - s_waitcnt vmcnt(0) 6812 must happen after 6813 any preceding 6814 global/generic load 6815 atomic/atomicrmw 6816 with an equal or 6817 wider sync scope 6818 and memory ordering 6819 stronger than 6820 unordered (this is 6821 termed the 6822 fence-paired-atomic). 6823 - s_waitcnt lgkmcnt(0) 6824 must happen after 6825 any preceding 6826 local/generic load 6827 atomic/atomicrmw 6828 with an equal or 6829 wider sync scope 6830 and memory ordering 6831 stronger than 6832 unordered (this is 6833 termed the 6834 fence-paired-atomic). 6835 - Must happen before 6836 the following buffer_invl2 and 6837 buffer_wbinvl1_vol. 6838 - Ensures that the 6839 fence-paired atomic 6840 has completed 6841 before invalidating 6842 the 6843 cache. Therefore 6844 any following 6845 locations read must 6846 be no older than 6847 the value read by 6848 the 6849 fence-paired-atomic. 6850 6851 2. buffer_invl2; 6852 buffer_wbinvl1_vol 6853 6854 - Must happen before any 6855 following global/generic 6856 load/load 6857 atomic/store/store 6858 atomic/atomicrmw. 6859 - Ensures that 6860 following loads 6861 will not see stale 6862 global data. 6863 6864 **Release Atomic** 6865 ------------------------------------------------------------------------------------ 6866 store atomic release - singlethread - global 1. buffer/global/flat_store 6867 - wavefront - generic 6868 store atomic release - singlethread - local *If TgSplit execution mode, 6869 - wavefront local address space cannot 6870 be used.* 6871 6872 1. ds_store 6873 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6874 - generic 6875 - Use lgkmcnt(0) if not 6876 TgSplit execution mode 6877 and vmcnt(0) if TgSplit 6878 execution mode. 6879 - If OpenCL, omit lgkmcnt(0). 6880 - s_waitcnt vmcnt(0) 6881 must happen after 6882 any preceding 6883 global/generic load/store/ 6884 load atomic/store atomic/ 6885 atomicrmw. 6886 - s_waitcnt lgkmcnt(0) 6887 must happen after 6888 any preceding 6889 local/generic 6890 load/store/load 6891 atomic/store 6892 atomic/atomicrmw. 6893 - Must happen before 6894 the following 6895 store. 6896 - Ensures that all 6897 memory operations 6898 have 6899 completed before 6900 performing the 6901 store that is being 6902 released. 6903 6904 2. buffer/global/flat_store 6905 store atomic release - workgroup - local *If TgSplit execution mode, 6906 local address space cannot 6907 be used.* 6908 6909 1. ds_store 6910 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6911 - generic vmcnt(0) 6912 6913 - If TgSplit execution mode, 6914 omit lgkmcnt(0). 6915 - If OpenCL and 6916 address space is 6917 not generic, omit 6918 lgkmcnt(0). 6919 - Could be split into 6920 separate s_waitcnt 6921 vmcnt(0) and 6922 s_waitcnt 6923 lgkmcnt(0) to allow 6924 them to be 6925 independently moved 6926 according to the 6927 following rules. 6928 - s_waitcnt vmcnt(0) 6929 must happen after 6930 any preceding 6931 global/generic 6932 load/store/load 6933 atomic/store 6934 atomic/atomicrmw. 6935 - s_waitcnt lgkmcnt(0) 6936 must happen after 6937 any preceding 6938 local/generic 6939 load/store/load 6940 atomic/store 6941 atomic/atomicrmw. 6942 - Must happen before 6943 the following 6944 store. 6945 - Ensures that all 6946 memory operations 6947 to memory have 6948 completed before 6949 performing the 6950 store that is being 6951 released. 6952 6953 2. buffer/global/flat_store 6954 store atomic release - system - global 1. buffer_wbl2 6955 - generic 6956 - Must happen before 6957 following s_waitcnt. 6958 - Performs L2 writeback to 6959 ensure previous 6960 global/generic 6961 store/atomicrmw are 6962 visible at system scope. 6963 6964 2. s_waitcnt lgkmcnt(0) & 6965 vmcnt(0) 6966 6967 - If TgSplit execution mode, 6968 omit lgkmcnt(0). 6969 - If OpenCL and 6970 address space is 6971 not generic, omit 6972 lgkmcnt(0). 6973 - Could be split into 6974 separate s_waitcnt 6975 vmcnt(0) and 6976 s_waitcnt 6977 lgkmcnt(0) to allow 6978 them to be 6979 independently moved 6980 according to the 6981 following rules. 6982 - s_waitcnt vmcnt(0) 6983 must happen after any 6984 preceding 6985 global/generic 6986 load/store/load 6987 atomic/store 6988 atomic/atomicrmw. 6989 - s_waitcnt lgkmcnt(0) 6990 must happen after any 6991 preceding 6992 local/generic 6993 load/store/load 6994 atomic/store 6995 atomic/atomicrmw. 6996 - Must happen before 6997 the following 6998 store. 6999 - Ensures that all 7000 memory operations 7001 to memory and the L2 7002 writeback have 7003 completed before 7004 performing the 7005 store that is being 7006 released. 7007 7008 2. buffer/global/flat_store 7009 scc=1 7010 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7011 - wavefront - generic 7012 atomicrmw release - singlethread - local *If TgSplit execution mode, 7013 - wavefront local address space cannot 7014 be used.* 7015 7016 1. ds_atomic 7017 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7018 - generic 7019 - Use lgkmcnt(0) if not 7020 TgSplit execution mode 7021 and vmcnt(0) if TgSplit 7022 execution mode. 7023 - If OpenCL, omit 7024 lgkmcnt(0). 7025 - s_waitcnt vmcnt(0) 7026 must happen after 7027 any preceding 7028 global/generic load/store/ 7029 load atomic/store atomic/ 7030 atomicrmw. 7031 - s_waitcnt lgkmcnt(0) 7032 must happen after 7033 any preceding 7034 local/generic 7035 load/store/load 7036 atomic/store 7037 atomic/atomicrmw. 7038 - Must happen before 7039 the following 7040 atomicrmw. 7041 - Ensures that all 7042 memory operations 7043 have 7044 completed before 7045 performing the 7046 atomicrmw that is 7047 being released. 7048 7049 2. buffer/global/flat_atomic 7050 atomicrmw release - workgroup - local *If TgSplit execution mode, 7051 local address space cannot 7052 be used.* 7053 7054 1. ds_atomic 7055 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7056 - generic vmcnt(0) 7057 7058 - If TgSplit execution mode, 7059 omit lgkmcnt(0). 7060 - If OpenCL, omit 7061 lgkmcnt(0). 7062 - Could be split into 7063 separate s_waitcnt 7064 vmcnt(0) and 7065 s_waitcnt 7066 lgkmcnt(0) to allow 7067 them to be 7068 independently moved 7069 according to the 7070 following rules. 7071 - s_waitcnt vmcnt(0) 7072 must happen after 7073 any preceding 7074 global/generic 7075 load/store/load 7076 atomic/store 7077 atomic/atomicrmw. 7078 - s_waitcnt lgkmcnt(0) 7079 must happen after 7080 any preceding 7081 local/generic 7082 load/store/load 7083 atomic/store 7084 atomic/atomicrmw. 7085 - Must happen before 7086 the following 7087 atomicrmw. 7088 - Ensures that all 7089 memory operations 7090 to global and local 7091 have completed 7092 before performing 7093 the atomicrmw that 7094 is being released. 7095 7096 2. buffer/global/flat_atomic 7097 atomicrmw release - system - global 1. buffer_wbl2 7098 - generic 7099 - Must happen before 7100 following s_waitcnt. 7101 - Performs L2 writeback to 7102 ensure previous 7103 global/generic 7104 store/atomicrmw are 7105 visible at system scope. 7106 7107 2. s_waitcnt lgkmcnt(0) & 7108 vmcnt(0) 7109 7110 - If TgSplit execution mode, 7111 omit lgkmcnt(0). 7112 - If OpenCL, omit 7113 lgkmcnt(0). 7114 - Could be split into 7115 separate s_waitcnt 7116 vmcnt(0) and 7117 s_waitcnt 7118 lgkmcnt(0) to allow 7119 them to be 7120 independently moved 7121 according to the 7122 following rules. 7123 - s_waitcnt vmcnt(0) 7124 must happen after 7125 any preceding 7126 global/generic 7127 load/store/load 7128 atomic/store 7129 atomic/atomicrmw. 7130 - s_waitcnt lgkmcnt(0) 7131 must happen after 7132 any preceding 7133 local/generic 7134 load/store/load 7135 atomic/store 7136 atomic/atomicrmw. 7137 - Must happen before 7138 the following 7139 atomicrmw. 7140 - Ensures that all 7141 memory operations 7142 to memory and the L2 7143 writeback have 7144 completed before 7145 performing the 7146 store that is being 7147 released. 7148 7149 3. buffer/global/flat_atomic 7150 scc=1 7151 fence release - singlethread *none* *none* 7152 - wavefront 7153 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7154 7155 - Use lgkmcnt(0) if not 7156 TgSplit execution mode 7157 and vmcnt(0) if TgSplit 7158 execution mode. 7159 - If OpenCL and 7160 address space is 7161 not generic, omit 7162 lgkmcnt(0). 7163 - If OpenCL and 7164 address space is 7165 local, omit 7166 vmcnt(0). 7167 - However, since LLVM 7168 currently has no 7169 address space on 7170 the fence need to 7171 conservatively 7172 always generate. If 7173 fence had an 7174 address space then 7175 set to address 7176 space of OpenCL 7177 fence flag, or to 7178 generic if both 7179 local and global 7180 flags are 7181 specified. 7182 - s_waitcnt vmcnt(0) 7183 must happen after 7184 any preceding 7185 global/generic 7186 load/store/ 7187 load atomic/store atomic/ 7188 atomicrmw. 7189 - s_waitcnt lgkmcnt(0) 7190 must happen after 7191 any preceding 7192 local/generic 7193 load/load 7194 atomic/store/store 7195 atomic/atomicrmw. 7196 - Must happen before 7197 any following store 7198 atomic/atomicrmw 7199 with an equal or 7200 wider sync scope 7201 and memory ordering 7202 stronger than 7203 unordered (this is 7204 termed the 7205 fence-paired-atomic). 7206 - Ensures that all 7207 memory operations 7208 have 7209 completed before 7210 performing the 7211 following 7212 fence-paired-atomic. 7213 7214 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7215 vmcnt(0) 7216 7217 - If TgSplit execution mode, 7218 omit lgkmcnt(0). 7219 - If OpenCL and 7220 address space is 7221 not generic, omit 7222 lgkmcnt(0). 7223 - If OpenCL and 7224 address space is 7225 local, omit 7226 vmcnt(0). 7227 - However, since LLVM 7228 currently has no 7229 address space on 7230 the fence need to 7231 conservatively 7232 always generate. If 7233 fence had an 7234 address space then 7235 set to address 7236 space of OpenCL 7237 fence flag, or to 7238 generic if both 7239 local and global 7240 flags are 7241 specified. 7242 - Could be split into 7243 separate s_waitcnt 7244 vmcnt(0) and 7245 s_waitcnt 7246 lgkmcnt(0) to allow 7247 them to be 7248 independently moved 7249 according to the 7250 following rules. 7251 - s_waitcnt vmcnt(0) 7252 must happen after 7253 any preceding 7254 global/generic 7255 load/store/load 7256 atomic/store 7257 atomic/atomicrmw. 7258 - s_waitcnt lgkmcnt(0) 7259 must happen after 7260 any preceding 7261 local/generic 7262 load/store/load 7263 atomic/store 7264 atomic/atomicrmw. 7265 - Must happen before 7266 any following store 7267 atomic/atomicrmw 7268 with an equal or 7269 wider sync scope 7270 and memory ordering 7271 stronger than 7272 unordered (this is 7273 termed the 7274 fence-paired-atomic). 7275 - Ensures that all 7276 memory operations 7277 have 7278 completed before 7279 performing the 7280 following 7281 fence-paired-atomic. 7282 7283 fence release - system *none* 1. buffer_wbl2 7284 7285 - If OpenCL and 7286 address space is 7287 local, omit. 7288 - Must happen before 7289 following s_waitcnt. 7290 - Performs L2 writeback to 7291 ensure previous 7292 global/generic 7293 store/atomicrmw are 7294 visible at system scope. 7295 7296 2. s_waitcnt lgkmcnt(0) & 7297 vmcnt(0) 7298 7299 - If TgSplit execution mode, 7300 omit lgkmcnt(0). 7301 - If OpenCL and 7302 address space is 7303 not generic, omit 7304 lgkmcnt(0). 7305 - If OpenCL and 7306 address space is 7307 local, omit 7308 vmcnt(0). 7309 - However, since LLVM 7310 currently has no 7311 address space on 7312 the fence need to 7313 conservatively 7314 always generate. If 7315 fence had an 7316 address space then 7317 set to address 7318 space of OpenCL 7319 fence flag, or to 7320 generic if both 7321 local and global 7322 flags are 7323 specified. 7324 - Could be split into 7325 separate s_waitcnt 7326 vmcnt(0) and 7327 s_waitcnt 7328 lgkmcnt(0) to allow 7329 them to be 7330 independently moved 7331 according to the 7332 following rules. 7333 - s_waitcnt vmcnt(0) 7334 must happen after 7335 any preceding 7336 global/generic 7337 load/store/load 7338 atomic/store 7339 atomic/atomicrmw. 7340 - s_waitcnt lgkmcnt(0) 7341 must happen after 7342 any preceding 7343 local/generic 7344 load/store/load 7345 atomic/store 7346 atomic/atomicrmw. 7347 - Must happen before 7348 any following store 7349 atomic/atomicrmw 7350 with an equal or 7351 wider sync scope 7352 and memory ordering 7353 stronger than 7354 unordered (this is 7355 termed the 7356 fence-paired-atomic). 7357 - Ensures that all 7358 memory operations 7359 have 7360 completed before 7361 performing the 7362 following 7363 fence-paired-atomic. 7364 7365 **Acquire-Release Atomic** 7366 ------------------------------------------------------------------------------------ 7367 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7368 - wavefront - generic 7369 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7370 - wavefront local address space cannot 7371 be used.* 7372 7373 1. ds_atomic 7374 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7375 7376 - Use lgkmcnt(0) if not 7377 TgSplit execution mode 7378 and vmcnt(0) if TgSplit 7379 execution mode. 7380 - If OpenCL, omit 7381 lgkmcnt(0). 7382 - Must happen after 7383 any preceding 7384 local/generic 7385 load/store/load 7386 atomic/store 7387 atomic/atomicrmw. 7388 - s_waitcnt vmcnt(0) 7389 must happen after 7390 any preceding 7391 global/generic load/store/ 7392 load atomic/store atomic/ 7393 atomicrmw. 7394 - s_waitcnt lgkmcnt(0) 7395 must happen after 7396 any preceding 7397 local/generic 7398 load/store/load 7399 atomic/store 7400 atomic/atomicrmw. 7401 - Must happen before 7402 the following 7403 atomicrmw. 7404 - Ensures that all 7405 memory operations 7406 have 7407 completed before 7408 performing the 7409 atomicrmw that is 7410 being released. 7411 7412 2. buffer/global_atomic 7413 3. s_waitcnt vmcnt(0) 7414 7415 - If not TgSplit execution 7416 mode, omit. 7417 - Must happen before 7418 the following 7419 buffer_wbinvl1_vol. 7420 - Ensures any 7421 following global 7422 data read is no 7423 older than the 7424 atomicrmw value 7425 being acquired. 7426 7427 4. buffer_wbinvl1_vol 7428 7429 - If not TgSplit execution 7430 mode, omit. 7431 - Ensures that 7432 following 7433 loads will not see 7434 stale data. 7435 7436 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7437 local address space cannot 7438 be used.* 7439 7440 1. ds_atomic 7441 2. s_waitcnt lgkmcnt(0) 7442 7443 - If OpenCL, omit. 7444 - Must happen before 7445 any following 7446 global/generic 7447 load/load 7448 atomic/store/store 7449 atomic/atomicrmw. 7450 - Ensures any 7451 following global 7452 data read is no 7453 older than the local load 7454 atomic value being 7455 acquired. 7456 7457 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7458 7459 - Use lgkmcnt(0) if not 7460 TgSplit execution mode 7461 and vmcnt(0) if TgSplit 7462 execution mode. 7463 - If OpenCL, omit 7464 lgkmcnt(0). 7465 - s_waitcnt vmcnt(0) 7466 must happen after 7467 any preceding 7468 global/generic load/store/ 7469 load atomic/store atomic/ 7470 atomicrmw. 7471 - s_waitcnt lgkmcnt(0) 7472 must happen after 7473 any preceding 7474 local/generic 7475 load/store/load 7476 atomic/store 7477 atomic/atomicrmw. 7478 - Must happen before 7479 the following 7480 atomicrmw. 7481 - Ensures that all 7482 memory operations 7483 have 7484 completed before 7485 performing the 7486 atomicrmw that is 7487 being released. 7488 7489 2. flat_atomic 7490 3. s_waitcnt lgkmcnt(0) & 7491 vmcnt(0) 7492 7493 - If not TgSplit execution 7494 mode, omit vmcnt(0). 7495 - If OpenCL, omit 7496 lgkmcnt(0). 7497 - Must happen before 7498 the following 7499 buffer_wbinvl1_vol and 7500 any following 7501 global/generic 7502 load/load 7503 atomic/store/store 7504 atomic/atomicrmw. 7505 - Ensures any 7506 following global 7507 data read is no 7508 older than a local load 7509 atomic value being 7510 acquired. 7511 7512 3. buffer_wbinvl1_vol 7513 7514 - If not TgSplit execution 7515 mode, omit. 7516 - Ensures that 7517 following 7518 loads will not see 7519 stale data. 7520 7521 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7522 vmcnt(0) 7523 7524 - If TgSplit execution mode, 7525 omit lgkmcnt(0). 7526 - If OpenCL, omit 7527 lgkmcnt(0). 7528 - Could be split into 7529 separate s_waitcnt 7530 vmcnt(0) and 7531 s_waitcnt 7532 lgkmcnt(0) to allow 7533 them to be 7534 independently moved 7535 according to the 7536 following rules. 7537 - s_waitcnt vmcnt(0) 7538 must happen after 7539 any preceding 7540 global/generic 7541 load/store/load 7542 atomic/store 7543 atomic/atomicrmw. 7544 - s_waitcnt lgkmcnt(0) 7545 must happen after 7546 any preceding 7547 local/generic 7548 load/store/load 7549 atomic/store 7550 atomic/atomicrmw. 7551 - Must happen before 7552 the following 7553 atomicrmw. 7554 - Ensures that all 7555 memory operations 7556 to global have 7557 completed before 7558 performing the 7559 atomicrmw that is 7560 being released. 7561 7562 2. buffer/global_atomic 7563 3. s_waitcnt vmcnt(0) 7564 7565 - Must happen before 7566 following 7567 buffer_wbinvl1_vol. 7568 - Ensures the 7569 atomicrmw has 7570 completed before 7571 invalidating the 7572 cache. 7573 7574 4. buffer_wbinvl1_vol 7575 7576 - Must happen before 7577 any following 7578 global/generic 7579 load/load 7580 atomic/atomicrmw. 7581 - Ensures that 7582 following loads 7583 will not see stale 7584 global data. 7585 7586 atomicrmw acq_rel - system - global 1. buffer_wbl2 7587 7588 - Must happen before 7589 following s_waitcnt. 7590 - Performs L2 writeback to 7591 ensure previous 7592 global/generic 7593 store/atomicrmw are 7594 visible at system scope. 7595 7596 2. s_waitcnt lgkmcnt(0) & 7597 vmcnt(0) 7598 7599 - If TgSplit execution mode, 7600 omit lgkmcnt(0). 7601 - If OpenCL, omit 7602 lgkmcnt(0). 7603 - Could be split into 7604 separate s_waitcnt 7605 vmcnt(0) and 7606 s_waitcnt 7607 lgkmcnt(0) to allow 7608 them to be 7609 independently moved 7610 according to the 7611 following rules. 7612 - s_waitcnt vmcnt(0) 7613 must happen after 7614 any preceding 7615 global/generic 7616 load/store/load 7617 atomic/store 7618 atomic/atomicrmw. 7619 - s_waitcnt lgkmcnt(0) 7620 must happen after 7621 any preceding 7622 local/generic 7623 load/store/load 7624 atomic/store 7625 atomic/atomicrmw. 7626 - Must happen before 7627 the following 7628 atomicrmw. 7629 - Ensures that all 7630 memory operations 7631 to global and L2 writeback 7632 have completed before 7633 performing the 7634 atomicrmw that is 7635 being released. 7636 7637 3. buffer/global_atomic 7638 scc=1 7639 4. s_waitcnt vmcnt(0) 7640 7641 - Must happen before 7642 following buffer_invl2 and 7643 buffer_wbinvl1_vol. 7644 - Ensures the 7645 atomicrmw has 7646 completed before 7647 invalidating the 7648 caches. 7649 7650 5. buffer_invl2; 7651 buffer_wbinvl1_vol 7652 7653 - Must happen before 7654 any following 7655 global/generic 7656 load/load 7657 atomic/atomicrmw. 7658 - Ensures that 7659 following loads 7660 will not see stale 7661 MTYPE NC global data. 7662 MTYPE RW and CC memory will 7663 never be stale due to the 7664 memory probes. 7665 7666 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7667 vmcnt(0) 7668 7669 - If TgSplit execution mode, 7670 omit lgkmcnt(0). 7671 - If OpenCL, omit 7672 lgkmcnt(0). 7673 - Could be split into 7674 separate s_waitcnt 7675 vmcnt(0) and 7676 s_waitcnt 7677 lgkmcnt(0) to allow 7678 them to be 7679 independently moved 7680 according to the 7681 following rules. 7682 - s_waitcnt vmcnt(0) 7683 must happen after 7684 any preceding 7685 global/generic 7686 load/store/load 7687 atomic/store 7688 atomic/atomicrmw. 7689 - s_waitcnt lgkmcnt(0) 7690 must happen after 7691 any preceding 7692 local/generic 7693 load/store/load 7694 atomic/store 7695 atomic/atomicrmw. 7696 - Must happen before 7697 the following 7698 atomicrmw. 7699 - Ensures that all 7700 memory operations 7701 to global have 7702 completed before 7703 performing the 7704 atomicrmw that is 7705 being released. 7706 7707 2. flat_atomic 7708 3. s_waitcnt vmcnt(0) & 7709 lgkmcnt(0) 7710 7711 - If TgSplit execution mode, 7712 omit lgkmcnt(0). 7713 - If OpenCL, omit 7714 lgkmcnt(0). 7715 - Must happen before 7716 following 7717 buffer_wbinvl1_vol. 7718 - Ensures the 7719 atomicrmw has 7720 completed before 7721 invalidating the 7722 cache. 7723 7724 4. buffer_wbinvl1_vol 7725 7726 - Must happen before 7727 any following 7728 global/generic 7729 load/load 7730 atomic/atomicrmw. 7731 - Ensures that 7732 following loads 7733 will not see stale 7734 global data. 7735 7736 atomicrmw acq_rel - system - generic 1. buffer_wbl2 7737 7738 - Must happen before 7739 following s_waitcnt. 7740 - Performs L2 writeback to 7741 ensure previous 7742 global/generic 7743 store/atomicrmw are 7744 visible at system scope. 7745 7746 2. s_waitcnt lgkmcnt(0) & 7747 vmcnt(0) 7748 7749 - If TgSplit execution mode, 7750 omit lgkmcnt(0). 7751 - If OpenCL, omit 7752 lgkmcnt(0). 7753 - Could be split into 7754 separate s_waitcnt 7755 vmcnt(0) and 7756 s_waitcnt 7757 lgkmcnt(0) to allow 7758 them to be 7759 independently moved 7760 according to the 7761 following rules. 7762 - s_waitcnt vmcnt(0) 7763 must happen after 7764 any preceding 7765 global/generic 7766 load/store/load 7767 atomic/store 7768 atomic/atomicrmw. 7769 - s_waitcnt lgkmcnt(0) 7770 must happen after 7771 any preceding 7772 local/generic 7773 load/store/load 7774 atomic/store 7775 atomic/atomicrmw. 7776 - Must happen before 7777 the following 7778 atomicrmw. 7779 - Ensures that all 7780 memory operations 7781 to global and L2 writeback 7782 have completed before 7783 performing the 7784 atomicrmw that is 7785 being released. 7786 7787 3. flat_atomic scc=1 7788 4. s_waitcnt vmcnt(0) & 7789 lgkmcnt(0) 7790 7791 - If TgSplit execution mode, 7792 omit lgkmcnt(0). 7793 - If OpenCL, omit 7794 lgkmcnt(0). 7795 - Must happen before 7796 following buffer_invl2 and 7797 buffer_wbinvl1_vol. 7798 - Ensures the 7799 atomicrmw has 7800 completed before 7801 invalidating the 7802 caches. 7803 7804 5. buffer_invl2; 7805 buffer_wbinvl1_vol 7806 7807 - Must happen before 7808 any following 7809 global/generic 7810 load/load 7811 atomic/atomicrmw. 7812 - Ensures that 7813 following loads 7814 will not see stale 7815 MTYPE NC global data. 7816 MTYPE RW and CC memory will 7817 never be stale due to the 7818 memory probes. 7819 7820 fence acq_rel - singlethread *none* *none* 7821 - wavefront 7822 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7823 7824 - Use lgkmcnt(0) if not 7825 TgSplit execution mode 7826 and vmcnt(0) if TgSplit 7827 execution mode. 7828 - If OpenCL and 7829 address space is 7830 not generic, omit 7831 lgkmcnt(0). 7832 - If OpenCL and 7833 address space is 7834 local, omit 7835 vmcnt(0). 7836 - However, 7837 since LLVM 7838 currently has no 7839 address space on 7840 the fence need to 7841 conservatively 7842 always generate 7843 (see comment for 7844 previous fence). 7845 - s_waitcnt vmcnt(0) 7846 must happen after 7847 any preceding 7848 global/generic 7849 load/store/ 7850 load atomic/store atomic/ 7851 atomicrmw. 7852 - s_waitcnt lgkmcnt(0) 7853 must happen after 7854 any preceding 7855 local/generic 7856 load/load 7857 atomic/store/store 7858 atomic/atomicrmw. 7859 - Must happen before 7860 any following 7861 global/generic 7862 load/load 7863 atomic/store/store 7864 atomic/atomicrmw. 7865 - Ensures that all 7866 memory operations 7867 have 7868 completed before 7869 performing any 7870 following global 7871 memory operations. 7872 - Ensures that the 7873 preceding 7874 local/generic load 7875 atomic/atomicrmw 7876 with an equal or 7877 wider sync scope 7878 and memory ordering 7879 stronger than 7880 unordered (this is 7881 termed the 7882 acquire-fence-paired-atomic) 7883 has completed 7884 before following 7885 global memory 7886 operations. This 7887 satisfies the 7888 requirements of 7889 acquire. 7890 - Ensures that all 7891 previous memory 7892 operations have 7893 completed before a 7894 following 7895 local/generic store 7896 atomic/atomicrmw 7897 with an equal or 7898 wider sync scope 7899 and memory ordering 7900 stronger than 7901 unordered (this is 7902 termed the 7903 release-fence-paired-atomic). 7904 This satisfies the 7905 requirements of 7906 release. 7907 - Must happen before 7908 the following 7909 buffer_wbinvl1_vol. 7910 - Ensures that the 7911 acquire-fence-paired 7912 atomic has completed 7913 before invalidating 7914 the 7915 cache. Therefore 7916 any following 7917 locations read must 7918 be no older than 7919 the value read by 7920 the 7921 acquire-fence-paired-atomic. 7922 7923 3. buffer_wbinvl1_vol 7924 7925 - If not TgSplit execution 7926 mode, omit. 7927 - Ensures that 7928 following 7929 loads will not see 7930 stale data. 7931 7932 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7933 vmcnt(0) 7934 7935 - If TgSplit execution mode, 7936 omit lgkmcnt(0). 7937 - If OpenCL and 7938 address space is 7939 not generic, omit 7940 lgkmcnt(0). 7941 - However, since LLVM 7942 currently has no 7943 address space on 7944 the fence need to 7945 conservatively 7946 always generate 7947 (see comment for 7948 previous fence). 7949 - Could be split into 7950 separate s_waitcnt 7951 vmcnt(0) and 7952 s_waitcnt 7953 lgkmcnt(0) to allow 7954 them to be 7955 independently moved 7956 according to the 7957 following rules. 7958 - s_waitcnt vmcnt(0) 7959 must happen after 7960 any preceding 7961 global/generic 7962 load/store/load 7963 atomic/store 7964 atomic/atomicrmw. 7965 - s_waitcnt lgkmcnt(0) 7966 must happen after 7967 any preceding 7968 local/generic 7969 load/store/load 7970 atomic/store 7971 atomic/atomicrmw. 7972 - Must happen before 7973 the following 7974 buffer_wbinvl1_vol. 7975 - Ensures that the 7976 preceding 7977 global/local/generic 7978 load 7979 atomic/atomicrmw 7980 with an equal or 7981 wider sync scope 7982 and memory ordering 7983 stronger than 7984 unordered (this is 7985 termed the 7986 acquire-fence-paired-atomic) 7987 has completed 7988 before invalidating 7989 the cache. This 7990 satisfies the 7991 requirements of 7992 acquire. 7993 - Ensures that all 7994 previous memory 7995 operations have 7996 completed before a 7997 following 7998 global/local/generic 7999 store 8000 atomic/atomicrmw 8001 with an equal or 8002 wider sync scope 8003 and memory ordering 8004 stronger than 8005 unordered (this is 8006 termed the 8007 release-fence-paired-atomic). 8008 This satisfies the 8009 requirements of 8010 release. 8011 8012 2. buffer_wbinvl1_vol 8013 8014 - Must happen before 8015 any following 8016 global/generic 8017 load/load 8018 atomic/store/store 8019 atomic/atomicrmw. 8020 - Ensures that 8021 following loads 8022 will not see stale 8023 global data. This 8024 satisfies the 8025 requirements of 8026 acquire. 8027 8028 fence acq_rel - system *none* 1. buffer_wbl2 8029 8030 - If OpenCL and 8031 address space is 8032 local, omit. 8033 - Must happen before 8034 following s_waitcnt. 8035 - Performs L2 writeback to 8036 ensure previous 8037 global/generic 8038 store/atomicrmw are 8039 visible at system scope. 8040 8041 2. s_waitcnt lgkmcnt(0) & 8042 vmcnt(0) 8043 8044 - If TgSplit execution mode, 8045 omit lgkmcnt(0). 8046 - If OpenCL and 8047 address space is 8048 not generic, omit 8049 lgkmcnt(0). 8050 - However, since LLVM 8051 currently has no 8052 address space on 8053 the fence need to 8054 conservatively 8055 always generate 8056 (see comment for 8057 previous fence). 8058 - Could be split into 8059 separate s_waitcnt 8060 vmcnt(0) and 8061 s_waitcnt 8062 lgkmcnt(0) to allow 8063 them to be 8064 independently moved 8065 according to the 8066 following rules. 8067 - s_waitcnt vmcnt(0) 8068 must happen after 8069 any preceding 8070 global/generic 8071 load/store/load 8072 atomic/store 8073 atomic/atomicrmw. 8074 - s_waitcnt lgkmcnt(0) 8075 must happen after 8076 any preceding 8077 local/generic 8078 load/store/load 8079 atomic/store 8080 atomic/atomicrmw. 8081 - Must happen before 8082 the following buffer_invl2 and 8083 buffer_wbinvl1_vol. 8084 - Ensures that the 8085 preceding 8086 global/local/generic 8087 load 8088 atomic/atomicrmw 8089 with an equal or 8090 wider sync scope 8091 and memory ordering 8092 stronger than 8093 unordered (this is 8094 termed the 8095 acquire-fence-paired-atomic) 8096 has completed 8097 before invalidating 8098 the cache. This 8099 satisfies the 8100 requirements of 8101 acquire. 8102 - Ensures that all 8103 previous memory 8104 operations have 8105 completed before a 8106 following 8107 global/local/generic 8108 store 8109 atomic/atomicrmw 8110 with an equal or 8111 wider sync scope 8112 and memory ordering 8113 stronger than 8114 unordered (this is 8115 termed the 8116 release-fence-paired-atomic). 8117 This satisfies the 8118 requirements of 8119 release. 8120 8121 3. buffer_invl2; 8122 buffer_wbinvl1_vol 8123 8124 - Must happen before 8125 any following 8126 global/generic 8127 load/load 8128 atomic/store/store 8129 atomic/atomicrmw. 8130 - Ensures that 8131 following loads 8132 will not see stale 8133 MTYPE NC global data. 8134 MTYPE RW and CC memory will 8135 never be stale due to the 8136 memory probes. 8137 8138 **Sequential Consistent Atomic** 8139 ------------------------------------------------------------------------------------ 8140 load atomic seq_cst - singlethread - global *Same as corresponding 8141 - wavefront - local load atomic acquire, 8142 - generic except must generated 8143 all instructions even 8144 for OpenCL.* 8145 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8146 - generic 8147 - Use lgkmcnt(0) if not 8148 TgSplit execution mode 8149 and vmcnt(0) if TgSplit 8150 execution mode. 8151 - s_waitcnt lgkmcnt(0) must 8152 happen after 8153 preceding 8154 local/generic load 8155 atomic/store 8156 atomic/atomicrmw 8157 with memory 8158 ordering of seq_cst 8159 and with equal or 8160 wider sync scope. 8161 (Note that seq_cst 8162 fences have their 8163 own s_waitcnt 8164 lgkmcnt(0) and so do 8165 not need to be 8166 considered.) 8167 - s_waitcnt vmcnt(0) 8168 must happen after 8169 preceding 8170 global/generic load 8171 atomic/store 8172 atomic/atomicrmw 8173 with memory 8174 ordering of seq_cst 8175 and with equal or 8176 wider sync scope. 8177 (Note that seq_cst 8178 fences have their 8179 own s_waitcnt 8180 vmcnt(0) and so do 8181 not need to be 8182 considered.) 8183 - Ensures any 8184 preceding 8185 sequential 8186 consistent global/local 8187 memory instructions 8188 have completed 8189 before executing 8190 this sequentially 8191 consistent 8192 instruction. This 8193 prevents reordering 8194 a seq_cst store 8195 followed by a 8196 seq_cst load. (Note 8197 that seq_cst is 8198 stronger than 8199 acquire/release as 8200 the reordering of 8201 load acquire 8202 followed by a store 8203 release is 8204 prevented by the 8205 s_waitcnt of 8206 the release, but 8207 there is nothing 8208 preventing a store 8209 release followed by 8210 load acquire from 8211 completing out of 8212 order. The s_waitcnt 8213 could be placed after 8214 seq_store or before 8215 the seq_load. We 8216 choose the load to 8217 make the s_waitcnt be 8218 as late as possible 8219 so that the store 8220 may have already 8221 completed.) 8222 8223 2. *Following 8224 instructions same as 8225 corresponding load 8226 atomic acquire, 8227 except must generated 8228 all instructions even 8229 for OpenCL.* 8230 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8231 local address space cannot 8232 be used.* 8233 8234 *Same as corresponding 8235 load atomic acquire, 8236 except must generated 8237 all instructions even 8238 for OpenCL.* 8239 8240 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8241 - system - generic vmcnt(0) 8242 8243 - If TgSplit execution mode, 8244 omit lgkmcnt(0). 8245 - Could be split into 8246 separate s_waitcnt 8247 vmcnt(0) 8248 and s_waitcnt 8249 lgkmcnt(0) to allow 8250 them to be 8251 independently moved 8252 according to the 8253 following rules. 8254 - s_waitcnt lgkmcnt(0) 8255 must happen after 8256 preceding 8257 global/generic load 8258 atomic/store 8259 atomic/atomicrmw 8260 with memory 8261 ordering of seq_cst 8262 and with equal or 8263 wider sync scope. 8264 (Note that seq_cst 8265 fences have their 8266 own s_waitcnt 8267 lgkmcnt(0) and so do 8268 not need to be 8269 considered.) 8270 - s_waitcnt vmcnt(0) 8271 must happen after 8272 preceding 8273 global/generic load 8274 atomic/store 8275 atomic/atomicrmw 8276 with memory 8277 ordering of seq_cst 8278 and with equal or 8279 wider sync scope. 8280 (Note that seq_cst 8281 fences have their 8282 own s_waitcnt 8283 vmcnt(0) and so do 8284 not need to be 8285 considered.) 8286 - Ensures any 8287 preceding 8288 sequential 8289 consistent global 8290 memory instructions 8291 have completed 8292 before executing 8293 this sequentially 8294 consistent 8295 instruction. This 8296 prevents reordering 8297 a seq_cst store 8298 followed by a 8299 seq_cst load. (Note 8300 that seq_cst is 8301 stronger than 8302 acquire/release as 8303 the reordering of 8304 load acquire 8305 followed by a store 8306 release is 8307 prevented by the 8308 s_waitcnt of 8309 the release, but 8310 there is nothing 8311 preventing a store 8312 release followed by 8313 load acquire from 8314 completing out of 8315 order. The s_waitcnt 8316 could be placed after 8317 seq_store or before 8318 the seq_load. We 8319 choose the load to 8320 make the s_waitcnt be 8321 as late as possible 8322 so that the store 8323 may have already 8324 completed.) 8325 8326 2. *Following 8327 instructions same as 8328 corresponding load 8329 atomic acquire, 8330 except must generated 8331 all instructions even 8332 for OpenCL.* 8333 store atomic seq_cst - singlethread - global *Same as corresponding 8334 - wavefront - local store atomic release, 8335 - workgroup - generic except must generated 8336 - agent all instructions even 8337 - system for OpenCL.* 8338 atomicrmw seq_cst - singlethread - global *Same as corresponding 8339 - wavefront - local atomicrmw acq_rel, 8340 - workgroup - generic except must generated 8341 - agent all instructions even 8342 - system for OpenCL.* 8343 fence seq_cst - singlethread *none* *Same as corresponding 8344 - wavefront fence acq_rel, 8345 - workgroup except must generated 8346 - agent all instructions even 8347 - system for OpenCL.* 8348 ============ ============ ============== ========== ================================ 8349 8350.. _amdgpu-amdhsa-memory-model-gfx10: 8351 8352Memory Model GFX10 8353++++++++++++++++++ 8354 8355For GFX10: 8356 8357* Each agent has multiple shader arrays (SA). 8358* Each SA has multiple work-group processors (WGP). 8359* Each WGP has multiple compute units (CU). 8360* Each CU has multiple SIMDs that execute wavefronts. 8361* The wavefronts for a single work-group are executed in the same 8362 WGP. In CU wavefront execution mode the wavefronts may be executed by 8363 different SIMDs in the same CU. In WGP wavefront execution mode the 8364 wavefronts may be executed by different SIMDs in different CUs in the same 8365 WGP. 8366* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8367 executing on it. 8368* All LDS operations of a WGP are performed as wavefront wide operations in a 8369 global order and involve no caching. Completion is reported to a wavefront in 8370 execution order. 8371* The LDS memory has multiple request queues shared by the SIMDs of a 8372 WGP. Therefore, the LDS operations performed by different wavefronts of a 8373 work-group can be reordered relative to each other, which can result in 8374 reordering the visibility of vector memory operations with respect to LDS 8375 operations of other wavefronts in the same work-group. A ``s_waitcnt 8376 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8377 vector memory operations between wavefronts of a work-group, but not between 8378 operations performed by the same wavefront. 8379* The vector memory operations are performed as wavefront wide operations. 8380 Completion of load/store/sample operations are reported to a wavefront in 8381 execution order of other load/store/sample operations performed by that 8382 wavefront. 8383* The vector memory operations access a vector L0 cache. There is a single L0 8384 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8385 special action is required for coherence between the lanes of a single 8386 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8387 wavefronts executing in the same work-group as they may be executing on SIMDs 8388 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8389 required for coherence between wavefronts executing in different work-groups 8390 as they may be executing on different WGPs. 8391* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8392 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8393 operations are used in a restricted way so do not impact the memory model. See 8394 :ref:`amdgpu-amdhsa-memory-spaces`. 8395* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8396 the same SA. Therefore, no special action is required for coherence between 8397 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8398 required for coherence between wavefronts executing in different work-groups 8399 as they may be executing on different SAs that access different L1s. 8400* The L1 caches have independent quadrants to service disjoint ranges of virtual 8401 addresses. 8402* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8403 vector and scalar memory operations performed by different wavefronts, whether 8404 executing in the same or different work-groups (which may be executing on 8405 different CUs accessing different L0s), can be reordered relative to each 8406 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8407 synchronization between vector memory operations of different wavefronts. It 8408 ensures a previous vector memory operation has completed before executing a 8409 subsequent vector memory or LDS operation and so can be used to meet the 8410 requirements of acquire, release and sequential consistency. 8411* The L1 caches use an L2 cache shared by all SAs on the same agent. 8412* The L2 cache has independent channels to service disjoint ranges of virtual 8413 addresses. 8414* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8415 quadrant has a separate request queue per L2 channel. Therefore, the vector 8416 and scalar memory operations performed by wavefronts executing in different 8417 work-groups (which may be executing on different SAs) of an agent can be 8418 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8419 required to ensure synchronization between vector memory operations of 8420 different SAs. It ensures a previous vector memory operation has completed 8421 before executing a subsequent vector memory and so can be used to meet the 8422 requirements of acquire, release and sequential consistency. 8423* The L2 cache can be kept coherent with other agents on some targets, or ranges 8424 of virtual addresses can be set up to bypass it to ensure system coherence. 8425 8426Scalar memory operations are only used to access memory that is proven to not 8427change during the execution of the kernel dispatch. This includes constant 8428address space and global address space for program scope ``const`` variables. 8429Therefore, the kernel machine code does not have to maintain the scalar cache to 8430ensure it is coherent with the vector caches. The scalar and vector caches are 8431invalidated between kernel dispatches by CP since constant address space data 8432may change between kernel dispatch executions. See 8433:ref:`amdgpu-amdhsa-memory-spaces`. 8434 8435The one exception is if scalar writes are used to spill SGPR registers. In this 8436case the AMDGPU backend ensures the memory location used to spill is never 8437accessed by vector memory operations at the same time. If scalar writes are used 8438then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8439return since the locations may be used for vector memory instructions by a 8440future wavefront that uses the same scratch area, or a function call that 8441creates a frame at the same address, respectively. There is no need for a 8442``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8443 8444For kernarg backing memory: 8445 8446* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8447* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8448 needing to invalidate the L2 cache. 8449* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8450 so the L2 cache will be coherent with the CPU and other agents. 8451 8452Scratch backing memory (which is used for the private address space) is accessed 8453with MTYPE NC (non-coherent). Since the private address space is only accessed 8454by a single thread, and is always write-before-read, there is never a need to 8455invalidate these entries from the L0 or L1 caches. 8456 8457Wavefronts are executed in native mode with in-order reporting of loads and 8458sample instructions. In this mode vmcnt reports completion of load, atomic with 8459return and sample instructions in order, and the vscnt reports the completion of 8460store and atomic without return in order. See ``MEM_ORDERED`` field in 8461:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8462 8463Wavefronts can be executed in WGP or CU wavefront execution mode: 8464 8465* In WGP wavefront execution mode the wavefronts of a work-group are executed 8466 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8467 CU L0 caches is required for work-group synchronization. Also accesses to L1 8468 at work-group scope need to be explicitly ordered as the accesses from 8469 different CUs are not ordered. 8470* In CU wavefront execution mode the wavefronts of a work-group are executed on 8471 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8472 the work-group access the same L0 which in turn ensures L1 accesses are 8473 ordered and so do not require explicit management of the caches for 8474 work-group synchronization. 8475 8476See ``WGP_MODE`` field in 8477:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8478:ref:`amdgpu-target-features`. 8479 8480The code sequences used to implement the memory model for GFX10 are defined in 8481table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8482 8483 .. table:: AMDHSA Memory Model Code Sequences GFX10 8484 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8485 8486 ============ ============ ============== ========== ================================ 8487 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8488 Ordering Sync Scope Address GFX10 8489 Space 8490 ============ ============ ============== ========== ================================ 8491 **Non-Atomic** 8492 ------------------------------------------------------------------------------------ 8493 load *none* *none* - global - !volatile & !nontemporal 8494 - generic 8495 - private 1. buffer/global/flat_load 8496 - constant 8497 - !volatile & nontemporal 8498 8499 1. buffer/global/flat_load 8500 slc=1 8501 8502 - volatile 8503 8504 1. buffer/global/flat_load 8505 glc=1 dlc=1 8506 2. s_waitcnt vmcnt(0) 8507 8508 - Must happen before 8509 any following volatile 8510 global/generic 8511 load/store. 8512 - Ensures that 8513 volatile 8514 operations to 8515 different 8516 addresses will not 8517 be reordered by 8518 hardware. 8519 8520 load *none* *none* - local 1. ds_load 8521 store *none* *none* - global - !volatile & !nontemporal 8522 - generic 8523 - private 1. buffer/global/flat_store 8524 - constant 8525 - !volatile & nontemporal 8526 8527 1. buffer/global/flat_store 8528 slc=1 8529 8530 - volatile 8531 8532 1. buffer/global/flat_store 8533 2. s_waitcnt vscnt(0) 8534 8535 - Must happen before 8536 any following volatile 8537 global/generic 8538 load/store. 8539 - Ensures that 8540 volatile 8541 operations to 8542 different 8543 addresses will not 8544 be reordered by 8545 hardware. 8546 8547 store *none* *none* - local 1. ds_store 8548 **Unordered Atomic** 8549 ------------------------------------------------------------------------------------ 8550 load atomic unordered *any* *any* *Same as non-atomic*. 8551 store atomic unordered *any* *any* *Same as non-atomic*. 8552 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8553 **Monotonic Atomic** 8554 ------------------------------------------------------------------------------------ 8555 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8556 - wavefront - generic 8557 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8558 - generic glc=1 8559 8560 - If CU wavefront execution 8561 mode, omit glc=1. 8562 8563 load atomic monotonic - singlethread - local 1. ds_load 8564 - wavefront 8565 - workgroup 8566 load atomic monotonic - agent - global 1. buffer/global/flat_load 8567 - system - generic glc=1 dlc=1 8568 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8569 - wavefront - generic 8570 - workgroup 8571 - agent 8572 - system 8573 store atomic monotonic - singlethread - local 1. ds_store 8574 - wavefront 8575 - workgroup 8576 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8577 - wavefront - generic 8578 - workgroup 8579 - agent 8580 - system 8581 atomicrmw monotonic - singlethread - local 1. ds_atomic 8582 - wavefront 8583 - workgroup 8584 **Acquire Atomic** 8585 ------------------------------------------------------------------------------------ 8586 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8587 - wavefront - local 8588 - generic 8589 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8590 8591 - If CU wavefront execution 8592 mode, omit glc=1. 8593 8594 2. s_waitcnt vmcnt(0) 8595 8596 - If CU wavefront execution 8597 mode, omit. 8598 - Must happen before 8599 the following buffer_gl0_inv 8600 and before any following 8601 global/generic 8602 load/load 8603 atomic/store/store 8604 atomic/atomicrmw. 8605 8606 3. buffer_gl0_inv 8607 8608 - If CU wavefront execution 8609 mode, omit. 8610 - Ensures that 8611 following 8612 loads will not see 8613 stale data. 8614 8615 load atomic acquire - workgroup - local 1. ds_load 8616 2. s_waitcnt lgkmcnt(0) 8617 8618 - If OpenCL, omit. 8619 - Must happen before 8620 the following buffer_gl0_inv 8621 and before any following 8622 global/generic load/load 8623 atomic/store/store 8624 atomic/atomicrmw. 8625 - Ensures any 8626 following global 8627 data read is no 8628 older than the local load 8629 atomic value being 8630 acquired. 8631 8632 3. buffer_gl0_inv 8633 8634 - If CU wavefront execution 8635 mode, omit. 8636 - If OpenCL, omit. 8637 - Ensures that 8638 following 8639 loads will not see 8640 stale data. 8641 8642 load atomic acquire - workgroup - generic 1. flat_load glc=1 8643 8644 - If CU wavefront execution 8645 mode, omit glc=1. 8646 8647 2. s_waitcnt lgkmcnt(0) & 8648 vmcnt(0) 8649 8650 - If CU wavefront execution 8651 mode, omit vmcnt(0). 8652 - If OpenCL, omit 8653 lgkmcnt(0). 8654 - Must happen before 8655 the following 8656 buffer_gl0_inv and any 8657 following global/generic 8658 load/load 8659 atomic/store/store 8660 atomic/atomicrmw. 8661 - Ensures any 8662 following global 8663 data read is no 8664 older than a local load 8665 atomic value being 8666 acquired. 8667 8668 3. buffer_gl0_inv 8669 8670 - If CU wavefront execution 8671 mode, omit. 8672 - Ensures that 8673 following 8674 loads will not see 8675 stale data. 8676 8677 load atomic acquire - agent - global 1. buffer/global_load 8678 - system glc=1 dlc=1 8679 2. s_waitcnt vmcnt(0) 8680 8681 - Must happen before 8682 following 8683 buffer_gl*_inv. 8684 - Ensures the load 8685 has completed 8686 before invalidating 8687 the caches. 8688 8689 3. buffer_gl0_inv; 8690 buffer_gl1_inv 8691 8692 - Must happen before 8693 any following 8694 global/generic 8695 load/load 8696 atomic/atomicrmw. 8697 - Ensures that 8698 following 8699 loads will not see 8700 stale global data. 8701 8702 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8703 - system 2. s_waitcnt vmcnt(0) & 8704 lgkmcnt(0) 8705 8706 - If OpenCL omit 8707 lgkmcnt(0). 8708 - Must happen before 8709 following 8710 buffer_gl*_invl. 8711 - Ensures the flat_load 8712 has completed 8713 before invalidating 8714 the caches. 8715 8716 3. buffer_gl0_inv; 8717 buffer_gl1_inv 8718 8719 - Must happen before 8720 any following 8721 global/generic 8722 load/load 8723 atomic/atomicrmw. 8724 - Ensures that 8725 following loads 8726 will not see stale 8727 global data. 8728 8729 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8730 - wavefront - local 8731 - generic 8732 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8733 2. s_waitcnt vm/vscnt(0) 8734 8735 - If CU wavefront execution 8736 mode, omit. 8737 - Use vmcnt(0) if atomic with 8738 return and vscnt(0) if 8739 atomic with no-return. 8740 - Must happen before 8741 the following buffer_gl0_inv 8742 and before any following 8743 global/generic 8744 load/load 8745 atomic/store/store 8746 atomic/atomicrmw. 8747 8748 3. buffer_gl0_inv 8749 8750 - If CU wavefront execution 8751 mode, omit. 8752 - Ensures that 8753 following 8754 loads will not see 8755 stale data. 8756 8757 atomicrmw acquire - workgroup - local 1. ds_atomic 8758 2. s_waitcnt lgkmcnt(0) 8759 8760 - If OpenCL, omit. 8761 - Must happen before 8762 the following 8763 buffer_gl0_inv. 8764 - Ensures any 8765 following global 8766 data read is no 8767 older than the local 8768 atomicrmw value 8769 being acquired. 8770 8771 3. buffer_gl0_inv 8772 8773 - If OpenCL omit. 8774 - Ensures that 8775 following 8776 loads will not see 8777 stale data. 8778 8779 atomicrmw acquire - workgroup - generic 1. flat_atomic 8780 2. s_waitcnt lgkmcnt(0) & 8781 vm/vscnt(0) 8782 8783 - If CU wavefront execution 8784 mode, omit vm/vscnt(0). 8785 - If OpenCL, omit lgkmcnt(0). 8786 - Use vmcnt(0) if atomic with 8787 return and vscnt(0) if 8788 atomic with no-return. 8789 - Must happen before 8790 the following 8791 buffer_gl0_inv. 8792 - Ensures any 8793 following global 8794 data read is no 8795 older than a local 8796 atomicrmw value 8797 being acquired. 8798 8799 3. buffer_gl0_inv 8800 8801 - If CU wavefront execution 8802 mode, omit. 8803 - Ensures that 8804 following 8805 loads will not see 8806 stale data. 8807 8808 atomicrmw acquire - agent - global 1. buffer/global_atomic 8809 - system 2. s_waitcnt vm/vscnt(0) 8810 8811 - Use vmcnt(0) if atomic with 8812 return and vscnt(0) if 8813 atomic with no-return. 8814 - Must happen before 8815 following 8816 buffer_gl*_inv. 8817 - Ensures the 8818 atomicrmw has 8819 completed before 8820 invalidating the 8821 caches. 8822 8823 3. buffer_gl0_inv; 8824 buffer_gl1_inv 8825 8826 - Must happen before 8827 any following 8828 global/generic 8829 load/load 8830 atomic/atomicrmw. 8831 - Ensures that 8832 following loads 8833 will not see stale 8834 global data. 8835 8836 atomicrmw acquire - agent - generic 1. flat_atomic 8837 - system 2. s_waitcnt vm/vscnt(0) & 8838 lgkmcnt(0) 8839 8840 - If OpenCL, omit 8841 lgkmcnt(0). 8842 - Use vmcnt(0) if atomic with 8843 return and vscnt(0) if 8844 atomic with no-return. 8845 - Must happen before 8846 following 8847 buffer_gl*_inv. 8848 - Ensures the 8849 atomicrmw has 8850 completed before 8851 invalidating the 8852 caches. 8853 8854 3. buffer_gl0_inv; 8855 buffer_gl1_inv 8856 8857 - Must happen before 8858 any following 8859 global/generic 8860 load/load 8861 atomic/atomicrmw. 8862 - Ensures that 8863 following loads 8864 will not see stale 8865 global data. 8866 8867 fence acquire - singlethread *none* *none* 8868 - wavefront 8869 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 8870 vmcnt(0) & vscnt(0) 8871 8872 - If CU wavefront execution 8873 mode, omit vmcnt(0) and 8874 vscnt(0). 8875 - If OpenCL and 8876 address space is 8877 not generic, omit 8878 lgkmcnt(0). 8879 - If OpenCL and 8880 address space is 8881 local, omit 8882 vmcnt(0) and vscnt(0). 8883 - However, since LLVM 8884 currently has no 8885 address space on 8886 the fence need to 8887 conservatively 8888 always generate. If 8889 fence had an 8890 address space then 8891 set to address 8892 space of OpenCL 8893 fence flag, or to 8894 generic if both 8895 local and global 8896 flags are 8897 specified. 8898 - Could be split into 8899 separate s_waitcnt 8900 vmcnt(0), s_waitcnt 8901 vscnt(0) and s_waitcnt 8902 lgkmcnt(0) to allow 8903 them to be 8904 independently moved 8905 according to the 8906 following rules. 8907 - s_waitcnt vmcnt(0) 8908 must happen after 8909 any preceding 8910 global/generic load 8911 atomic/ 8912 atomicrmw-with-return-value 8913 with an equal or 8914 wider sync scope 8915 and memory ordering 8916 stronger than 8917 unordered (this is 8918 termed the 8919 fence-paired-atomic). 8920 - s_waitcnt vscnt(0) 8921 must happen after 8922 any preceding 8923 global/generic 8924 atomicrmw-no-return-value 8925 with an equal or 8926 wider sync scope 8927 and memory ordering 8928 stronger than 8929 unordered (this is 8930 termed the 8931 fence-paired-atomic). 8932 - s_waitcnt lgkmcnt(0) 8933 must happen after 8934 any preceding 8935 local/generic load 8936 atomic/atomicrmw 8937 with an equal or 8938 wider sync scope 8939 and memory ordering 8940 stronger than 8941 unordered (this is 8942 termed the 8943 fence-paired-atomic). 8944 - Must happen before 8945 the following 8946 buffer_gl0_inv. 8947 - Ensures that the 8948 fence-paired atomic 8949 has completed 8950 before invalidating 8951 the 8952 cache. Therefore 8953 any following 8954 locations read must 8955 be no older than 8956 the value read by 8957 the 8958 fence-paired-atomic. 8959 8960 3. buffer_gl0_inv 8961 8962 - If CU wavefront execution 8963 mode, omit. 8964 - Ensures that 8965 following 8966 loads will not see 8967 stale data. 8968 8969 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 8970 - system vmcnt(0) & vscnt(0) 8971 8972 - If OpenCL and 8973 address space is 8974 not generic, omit 8975 lgkmcnt(0). 8976 - If OpenCL and 8977 address space is 8978 local, omit 8979 vmcnt(0) and vscnt(0). 8980 - However, since LLVM 8981 currently has no 8982 address space on 8983 the fence need to 8984 conservatively 8985 always generate 8986 (see comment for 8987 previous fence). 8988 - Could be split into 8989 separate s_waitcnt 8990 vmcnt(0), s_waitcnt 8991 vscnt(0) and s_waitcnt 8992 lgkmcnt(0) to allow 8993 them to be 8994 independently moved 8995 according to the 8996 following rules. 8997 - s_waitcnt vmcnt(0) 8998 must happen after 8999 any preceding 9000 global/generic load 9001 atomic/ 9002 atomicrmw-with-return-value 9003 with an equal or 9004 wider sync scope 9005 and memory ordering 9006 stronger than 9007 unordered (this is 9008 termed the 9009 fence-paired-atomic). 9010 - s_waitcnt vscnt(0) 9011 must happen after 9012 any preceding 9013 global/generic 9014 atomicrmw-no-return-value 9015 with an equal or 9016 wider sync scope 9017 and memory ordering 9018 stronger than 9019 unordered (this is 9020 termed the 9021 fence-paired-atomic). 9022 - s_waitcnt lgkmcnt(0) 9023 must happen after 9024 any preceding 9025 local/generic load 9026 atomic/atomicrmw 9027 with an equal or 9028 wider sync scope 9029 and memory ordering 9030 stronger than 9031 unordered (this is 9032 termed the 9033 fence-paired-atomic). 9034 - Must happen before 9035 the following 9036 buffer_gl*_inv. 9037 - Ensures that the 9038 fence-paired atomic 9039 has completed 9040 before invalidating 9041 the 9042 caches. Therefore 9043 any following 9044 locations read must 9045 be no older than 9046 the value read by 9047 the 9048 fence-paired-atomic. 9049 9050 2. buffer_gl0_inv; 9051 buffer_gl1_inv 9052 9053 - Must happen before any 9054 following global/generic 9055 load/load 9056 atomic/store/store 9057 atomic/atomicrmw. 9058 - Ensures that 9059 following loads 9060 will not see stale 9061 global data. 9062 9063 **Release Atomic** 9064 ------------------------------------------------------------------------------------ 9065 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9066 - wavefront - local 9067 - generic 9068 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9069 - generic vmcnt(0) & vscnt(0) 9070 9071 - If CU wavefront execution 9072 mode, omit vmcnt(0) and 9073 vscnt(0). 9074 - If OpenCL, omit 9075 lgkmcnt(0). 9076 - Could be split into 9077 separate s_waitcnt 9078 vmcnt(0), s_waitcnt 9079 vscnt(0) and s_waitcnt 9080 lgkmcnt(0) to allow 9081 them to be 9082 independently moved 9083 according to the 9084 following rules. 9085 - s_waitcnt vmcnt(0) 9086 must happen after 9087 any preceding 9088 global/generic load/load 9089 atomic/ 9090 atomicrmw-with-return-value. 9091 - s_waitcnt vscnt(0) 9092 must happen after 9093 any preceding 9094 global/generic 9095 store/store 9096 atomic/ 9097 atomicrmw-no-return-value. 9098 - s_waitcnt lgkmcnt(0) 9099 must happen after 9100 any preceding 9101 local/generic 9102 load/store/load 9103 atomic/store 9104 atomic/atomicrmw. 9105 - Must happen before 9106 the following 9107 store. 9108 - Ensures that all 9109 memory operations 9110 have 9111 completed before 9112 performing the 9113 store that is being 9114 released. 9115 9116 2. buffer/global/flat_store 9117 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9118 9119 - If CU wavefront execution 9120 mode, omit. 9121 - If OpenCL, omit. 9122 - Could be split into 9123 separate s_waitcnt 9124 vmcnt(0) and s_waitcnt 9125 vscnt(0) to allow 9126 them to be 9127 independently moved 9128 according to the 9129 following rules. 9130 - s_waitcnt vmcnt(0) 9131 must happen after 9132 any preceding 9133 global/generic load/load 9134 atomic/ 9135 atomicrmw-with-return-value. 9136 - s_waitcnt vscnt(0) 9137 must happen after 9138 any preceding 9139 global/generic 9140 store/store atomic/ 9141 atomicrmw-no-return-value. 9142 - Must happen before 9143 the following 9144 store. 9145 - Ensures that all 9146 global memory 9147 operations have 9148 completed before 9149 performing the 9150 store that is being 9151 released. 9152 9153 2. ds_store 9154 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9155 - system - generic vmcnt(0) & vscnt(0) 9156 9157 - If OpenCL and 9158 address space is 9159 not generic, omit 9160 lgkmcnt(0). 9161 - Could be split into 9162 separate s_waitcnt 9163 vmcnt(0), s_waitcnt vscnt(0) 9164 and s_waitcnt 9165 lgkmcnt(0) to allow 9166 them to be 9167 independently moved 9168 according to the 9169 following rules. 9170 - s_waitcnt vmcnt(0) 9171 must happen after 9172 any preceding 9173 global/generic 9174 load/load 9175 atomic/ 9176 atomicrmw-with-return-value. 9177 - s_waitcnt vscnt(0) 9178 must happen after 9179 any preceding 9180 global/generic 9181 store/store atomic/ 9182 atomicrmw-no-return-value. 9183 - s_waitcnt lgkmcnt(0) 9184 must happen after 9185 any preceding 9186 local/generic 9187 load/store/load 9188 atomic/store 9189 atomic/atomicrmw. 9190 - Must happen before 9191 the following 9192 store. 9193 - Ensures that all 9194 memory operations 9195 have 9196 completed before 9197 performing the 9198 store that is being 9199 released. 9200 9201 2. buffer/global/flat_store 9202 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9203 - wavefront - local 9204 - generic 9205 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9206 - generic vmcnt(0) & vscnt(0) 9207 9208 - If CU wavefront execution 9209 mode, omit vmcnt(0) and 9210 vscnt(0). 9211 - If OpenCL, omit lgkmcnt(0). 9212 - Could be split into 9213 separate s_waitcnt 9214 vmcnt(0), s_waitcnt 9215 vscnt(0) and s_waitcnt 9216 lgkmcnt(0) to allow 9217 them to be 9218 independently moved 9219 according to the 9220 following rules. 9221 - s_waitcnt vmcnt(0) 9222 must happen after 9223 any preceding 9224 global/generic load/load 9225 atomic/ 9226 atomicrmw-with-return-value. 9227 - s_waitcnt vscnt(0) 9228 must happen after 9229 any preceding 9230 global/generic 9231 store/store 9232 atomic/ 9233 atomicrmw-no-return-value. 9234 - s_waitcnt lgkmcnt(0) 9235 must happen after 9236 any preceding 9237 local/generic 9238 load/store/load 9239 atomic/store 9240 atomic/atomicrmw. 9241 - Must happen before 9242 the following 9243 atomicrmw. 9244 - Ensures that all 9245 memory operations 9246 have 9247 completed before 9248 performing the 9249 atomicrmw that is 9250 being released. 9251 9252 2. buffer/global/flat_atomic 9253 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9254 9255 - If CU wavefront execution 9256 mode, omit. 9257 - If OpenCL, omit. 9258 - Could be split into 9259 separate s_waitcnt 9260 vmcnt(0) and s_waitcnt 9261 vscnt(0) to allow 9262 them to be 9263 independently moved 9264 according to the 9265 following rules. 9266 - s_waitcnt vmcnt(0) 9267 must happen after 9268 any preceding 9269 global/generic load/load 9270 atomic/ 9271 atomicrmw-with-return-value. 9272 - s_waitcnt vscnt(0) 9273 must happen after 9274 any preceding 9275 global/generic 9276 store/store atomic/ 9277 atomicrmw-no-return-value. 9278 - Must happen before 9279 the following 9280 store. 9281 - Ensures that all 9282 global memory 9283 operations have 9284 completed before 9285 performing the 9286 store that is being 9287 released. 9288 9289 2. ds_atomic 9290 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9291 - system - generic vmcnt(0) & vscnt(0) 9292 9293 - If OpenCL, omit 9294 lgkmcnt(0). 9295 - Could be split into 9296 separate s_waitcnt 9297 vmcnt(0), s_waitcnt 9298 vscnt(0) and s_waitcnt 9299 lgkmcnt(0) to allow 9300 them to be 9301 independently moved 9302 according to the 9303 following rules. 9304 - s_waitcnt vmcnt(0) 9305 must happen after 9306 any preceding 9307 global/generic 9308 load/load atomic/ 9309 atomicrmw-with-return-value. 9310 - s_waitcnt vscnt(0) 9311 must happen after 9312 any preceding 9313 global/generic 9314 store/store atomic/ 9315 atomicrmw-no-return-value. 9316 - s_waitcnt lgkmcnt(0) 9317 must happen after 9318 any preceding 9319 local/generic 9320 load/store/load 9321 atomic/store 9322 atomic/atomicrmw. 9323 - Must happen before 9324 the following 9325 atomicrmw. 9326 - Ensures that all 9327 memory operations 9328 to global and local 9329 have completed 9330 before performing 9331 the atomicrmw that 9332 is being released. 9333 9334 2. buffer/global/flat_atomic 9335 fence release - singlethread *none* *none* 9336 - wavefront 9337 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9338 vmcnt(0) & vscnt(0) 9339 9340 - If CU wavefront execution 9341 mode, omit vmcnt(0) and 9342 vscnt(0). 9343 - If OpenCL and 9344 address space is 9345 not generic, omit 9346 lgkmcnt(0). 9347 - If OpenCL and 9348 address space is 9349 local, omit 9350 vmcnt(0) and vscnt(0). 9351 - However, since LLVM 9352 currently has no 9353 address space on 9354 the fence need to 9355 conservatively 9356 always generate. If 9357 fence had an 9358 address space then 9359 set to address 9360 space of OpenCL 9361 fence flag, or to 9362 generic if both 9363 local and global 9364 flags are 9365 specified. 9366 - Could be split into 9367 separate s_waitcnt 9368 vmcnt(0), s_waitcnt 9369 vscnt(0) and s_waitcnt 9370 lgkmcnt(0) to allow 9371 them to be 9372 independently moved 9373 according to the 9374 following rules. 9375 - s_waitcnt vmcnt(0) 9376 must happen after 9377 any preceding 9378 global/generic 9379 load/load 9380 atomic/ 9381 atomicrmw-with-return-value. 9382 - s_waitcnt vscnt(0) 9383 must happen after 9384 any preceding 9385 global/generic 9386 store/store atomic/ 9387 atomicrmw-no-return-value. 9388 - s_waitcnt lgkmcnt(0) 9389 must happen after 9390 any preceding 9391 local/generic 9392 load/store/load 9393 atomic/store atomic/ 9394 atomicrmw. 9395 - Must happen before 9396 any following store 9397 atomic/atomicrmw 9398 with an equal or 9399 wider sync scope 9400 and memory ordering 9401 stronger than 9402 unordered (this is 9403 termed the 9404 fence-paired-atomic). 9405 - Ensures that all 9406 memory operations 9407 have 9408 completed before 9409 performing the 9410 following 9411 fence-paired-atomic. 9412 9413 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9414 - system vmcnt(0) & vscnt(0) 9415 9416 - If OpenCL and 9417 address space is 9418 not generic, omit 9419 lgkmcnt(0). 9420 - If OpenCL and 9421 address space is 9422 local, omit 9423 vmcnt(0) and vscnt(0). 9424 - However, since LLVM 9425 currently has no 9426 address space on 9427 the fence need to 9428 conservatively 9429 always generate. If 9430 fence had an 9431 address space then 9432 set to address 9433 space of OpenCL 9434 fence flag, or to 9435 generic if both 9436 local and global 9437 flags are 9438 specified. 9439 - Could be split into 9440 separate s_waitcnt 9441 vmcnt(0), s_waitcnt 9442 vscnt(0) and s_waitcnt 9443 lgkmcnt(0) to allow 9444 them to be 9445 independently moved 9446 according to the 9447 following rules. 9448 - s_waitcnt vmcnt(0) 9449 must happen after 9450 any preceding 9451 global/generic 9452 load/load atomic/ 9453 atomicrmw-with-return-value. 9454 - s_waitcnt vscnt(0) 9455 must happen after 9456 any preceding 9457 global/generic 9458 store/store atomic/ 9459 atomicrmw-no-return-value. 9460 - s_waitcnt lgkmcnt(0) 9461 must happen after 9462 any preceding 9463 local/generic 9464 load/store/load 9465 atomic/store 9466 atomic/atomicrmw. 9467 - Must happen before 9468 any following store 9469 atomic/atomicrmw 9470 with an equal or 9471 wider sync scope 9472 and memory ordering 9473 stronger than 9474 unordered (this is 9475 termed the 9476 fence-paired-atomic). 9477 - Ensures that all 9478 memory operations 9479 have 9480 completed before 9481 performing the 9482 following 9483 fence-paired-atomic. 9484 9485 **Acquire-Release Atomic** 9486 ------------------------------------------------------------------------------------ 9487 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9488 - wavefront - local 9489 - generic 9490 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9491 vmcnt(0) & vscnt(0) 9492 9493 - If CU wavefront execution 9494 mode, omit vmcnt(0) and 9495 vscnt(0). 9496 - If OpenCL, omit 9497 lgkmcnt(0). 9498 - Must happen after 9499 any preceding 9500 local/generic 9501 load/store/load 9502 atomic/store 9503 atomic/atomicrmw. 9504 - Could be split into 9505 separate s_waitcnt 9506 vmcnt(0), s_waitcnt 9507 vscnt(0), and s_waitcnt 9508 lgkmcnt(0) to allow 9509 them to be 9510 independently moved 9511 according to the 9512 following rules. 9513 - s_waitcnt vmcnt(0) 9514 must happen after 9515 any preceding 9516 global/generic load/load 9517 atomic/ 9518 atomicrmw-with-return-value. 9519 - s_waitcnt vscnt(0) 9520 must happen after 9521 any preceding 9522 global/generic 9523 store/store 9524 atomic/ 9525 atomicrmw-no-return-value. 9526 - s_waitcnt lgkmcnt(0) 9527 must happen after 9528 any preceding 9529 local/generic 9530 load/store/load 9531 atomic/store 9532 atomic/atomicrmw. 9533 - Must happen before 9534 the following 9535 atomicrmw. 9536 - Ensures that all 9537 memory operations 9538 have 9539 completed before 9540 performing the 9541 atomicrmw that is 9542 being released. 9543 9544 2. buffer/global_atomic 9545 3. s_waitcnt vm/vscnt(0) 9546 9547 - If CU wavefront execution 9548 mode, omit. 9549 - Use vmcnt(0) if atomic with 9550 return and vscnt(0) if 9551 atomic with no-return. 9552 - Must happen before 9553 the following 9554 buffer_gl0_inv. 9555 - Ensures any 9556 following global 9557 data read is no 9558 older than the 9559 atomicrmw value 9560 being acquired. 9561 9562 4. buffer_gl0_inv 9563 9564 - If CU wavefront execution 9565 mode, omit. 9566 - Ensures that 9567 following 9568 loads will not see 9569 stale data. 9570 9571 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9572 9573 - If CU wavefront execution 9574 mode, omit. 9575 - If OpenCL, omit. 9576 - Could be split into 9577 separate s_waitcnt 9578 vmcnt(0) and s_waitcnt 9579 vscnt(0) to allow 9580 them to be 9581 independently moved 9582 according to the 9583 following rules. 9584 - s_waitcnt vmcnt(0) 9585 must happen after 9586 any preceding 9587 global/generic load/load 9588 atomic/ 9589 atomicrmw-with-return-value. 9590 - s_waitcnt vscnt(0) 9591 must happen after 9592 any preceding 9593 global/generic 9594 store/store atomic/ 9595 atomicrmw-no-return-value. 9596 - Must happen before 9597 the following 9598 store. 9599 - Ensures that all 9600 global memory 9601 operations have 9602 completed before 9603 performing the 9604 store that is being 9605 released. 9606 9607 2. ds_atomic 9608 3. s_waitcnt lgkmcnt(0) 9609 9610 - If OpenCL, omit. 9611 - Must happen before 9612 the following 9613 buffer_gl0_inv. 9614 - Ensures any 9615 following global 9616 data read is no 9617 older than the local load 9618 atomic value being 9619 acquired. 9620 9621 4. buffer_gl0_inv 9622 9623 - If CU wavefront execution 9624 mode, omit. 9625 - If OpenCL omit. 9626 - Ensures that 9627 following 9628 loads will not see 9629 stale data. 9630 9631 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9632 vmcnt(0) & vscnt(0) 9633 9634 - If CU wavefront execution 9635 mode, omit vmcnt(0) and 9636 vscnt(0). 9637 - If OpenCL, omit lgkmcnt(0). 9638 - Could be split into 9639 separate s_waitcnt 9640 vmcnt(0), s_waitcnt 9641 vscnt(0) and s_waitcnt 9642 lgkmcnt(0) to allow 9643 them to be 9644 independently moved 9645 according to the 9646 following rules. 9647 - s_waitcnt vmcnt(0) 9648 must happen after 9649 any preceding 9650 global/generic load/load 9651 atomic/ 9652 atomicrmw-with-return-value. 9653 - s_waitcnt vscnt(0) 9654 must happen after 9655 any preceding 9656 global/generic 9657 store/store 9658 atomic/ 9659 atomicrmw-no-return-value. 9660 - s_waitcnt lgkmcnt(0) 9661 must happen after 9662 any preceding 9663 local/generic 9664 load/store/load 9665 atomic/store 9666 atomic/atomicrmw. 9667 - Must happen before 9668 the following 9669 atomicrmw. 9670 - Ensures that all 9671 memory operations 9672 have 9673 completed before 9674 performing the 9675 atomicrmw that is 9676 being released. 9677 9678 2. flat_atomic 9679 3. s_waitcnt lgkmcnt(0) & 9680 vmcnt(0) & vscnt(0) 9681 9682 - If CU wavefront execution 9683 mode, omit vmcnt(0) and 9684 vscnt(0). 9685 - If OpenCL, omit lgkmcnt(0). 9686 - Must happen before 9687 the following 9688 buffer_gl0_inv. 9689 - Ensures any 9690 following global 9691 data read is no 9692 older than the load 9693 atomic value being 9694 acquired. 9695 9696 3. buffer_gl0_inv 9697 9698 - If CU wavefront execution 9699 mode, omit. 9700 - Ensures that 9701 following 9702 loads will not see 9703 stale data. 9704 9705 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9706 - system vmcnt(0) & vscnt(0) 9707 9708 - If OpenCL, omit 9709 lgkmcnt(0). 9710 - Could be split into 9711 separate s_waitcnt 9712 vmcnt(0), s_waitcnt 9713 vscnt(0) and s_waitcnt 9714 lgkmcnt(0) to allow 9715 them to be 9716 independently moved 9717 according to the 9718 following rules. 9719 - s_waitcnt vmcnt(0) 9720 must happen after 9721 any preceding 9722 global/generic 9723 load/load atomic/ 9724 atomicrmw-with-return-value. 9725 - s_waitcnt vscnt(0) 9726 must happen after 9727 any preceding 9728 global/generic 9729 store/store atomic/ 9730 atomicrmw-no-return-value. 9731 - s_waitcnt lgkmcnt(0) 9732 must happen after 9733 any preceding 9734 local/generic 9735 load/store/load 9736 atomic/store 9737 atomic/atomicrmw. 9738 - Must happen before 9739 the following 9740 atomicrmw. 9741 - Ensures that all 9742 memory operations 9743 to global have 9744 completed before 9745 performing the 9746 atomicrmw that is 9747 being released. 9748 9749 2. buffer/global_atomic 9750 3. s_waitcnt vm/vscnt(0) 9751 9752 - Use vmcnt(0) if atomic with 9753 return and vscnt(0) if 9754 atomic with no-return. 9755 - Must happen before 9756 following 9757 buffer_gl*_inv. 9758 - Ensures the 9759 atomicrmw has 9760 completed before 9761 invalidating the 9762 caches. 9763 9764 4. buffer_gl0_inv; 9765 buffer_gl1_inv 9766 9767 - Must happen before 9768 any following 9769 global/generic 9770 load/load 9771 atomic/atomicrmw. 9772 - Ensures that 9773 following loads 9774 will not see stale 9775 global data. 9776 9777 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9778 - system vmcnt(0) & vscnt(0) 9779 9780 - If OpenCL, omit 9781 lgkmcnt(0). 9782 - Could be split into 9783 separate s_waitcnt 9784 vmcnt(0), s_waitcnt 9785 vscnt(0), and s_waitcnt 9786 lgkmcnt(0) to allow 9787 them to be 9788 independently moved 9789 according to the 9790 following rules. 9791 - s_waitcnt vmcnt(0) 9792 must happen after 9793 any preceding 9794 global/generic 9795 load/load atomic 9796 atomicrmw-with-return-value. 9797 - s_waitcnt vscnt(0) 9798 must happen after 9799 any preceding 9800 global/generic 9801 store/store atomic/ 9802 atomicrmw-no-return-value. 9803 - s_waitcnt lgkmcnt(0) 9804 must happen after 9805 any preceding 9806 local/generic 9807 load/store/load 9808 atomic/store 9809 atomic/atomicrmw. 9810 - Must happen before 9811 the following 9812 atomicrmw. 9813 - Ensures that all 9814 memory operations 9815 have 9816 completed before 9817 performing the 9818 atomicrmw that is 9819 being released. 9820 9821 2. flat_atomic 9822 3. s_waitcnt vm/vscnt(0) & 9823 lgkmcnt(0) 9824 9825 - If OpenCL, omit 9826 lgkmcnt(0). 9827 - Use vmcnt(0) if atomic with 9828 return and vscnt(0) if 9829 atomic with no-return. 9830 - Must happen before 9831 following 9832 buffer_gl*_inv. 9833 - Ensures the 9834 atomicrmw has 9835 completed before 9836 invalidating the 9837 caches. 9838 9839 4. buffer_gl0_inv; 9840 buffer_gl1_inv 9841 9842 - Must happen before 9843 any following 9844 global/generic 9845 load/load 9846 atomic/atomicrmw. 9847 - Ensures that 9848 following loads 9849 will not see stale 9850 global data. 9851 9852 fence acq_rel - singlethread *none* *none* 9853 - wavefront 9854 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9855 vmcnt(0) & vscnt(0) 9856 9857 - If CU wavefront execution 9858 mode, omit vmcnt(0) and 9859 vscnt(0). 9860 - If OpenCL and 9861 address space is 9862 not generic, omit 9863 lgkmcnt(0). 9864 - If OpenCL and 9865 address space is 9866 local, omit 9867 vmcnt(0) and vscnt(0). 9868 - However, 9869 since LLVM 9870 currently has no 9871 address space on 9872 the fence need to 9873 conservatively 9874 always generate 9875 (see comment for 9876 previous fence). 9877 - Could be split into 9878 separate s_waitcnt 9879 vmcnt(0), s_waitcnt 9880 vscnt(0) and s_waitcnt 9881 lgkmcnt(0) to allow 9882 them to be 9883 independently moved 9884 according to the 9885 following rules. 9886 - s_waitcnt vmcnt(0) 9887 must happen after 9888 any preceding 9889 global/generic 9890 load/load 9891 atomic/ 9892 atomicrmw-with-return-value. 9893 - s_waitcnt vscnt(0) 9894 must happen after 9895 any preceding 9896 global/generic 9897 store/store atomic/ 9898 atomicrmw-no-return-value. 9899 - s_waitcnt lgkmcnt(0) 9900 must happen after 9901 any preceding 9902 local/generic 9903 load/store/load 9904 atomic/store atomic/ 9905 atomicrmw. 9906 - Must happen before 9907 any following 9908 global/generic 9909 load/load 9910 atomic/store/store 9911 atomic/atomicrmw. 9912 - Ensures that all 9913 memory operations 9914 have 9915 completed before 9916 performing any 9917 following global 9918 memory operations. 9919 - Ensures that the 9920 preceding 9921 local/generic load 9922 atomic/atomicrmw 9923 with an equal or 9924 wider sync scope 9925 and memory ordering 9926 stronger than 9927 unordered (this is 9928 termed the 9929 acquire-fence-paired-atomic) 9930 has completed 9931 before following 9932 global memory 9933 operations. This 9934 satisfies the 9935 requirements of 9936 acquire. 9937 - Ensures that all 9938 previous memory 9939 operations have 9940 completed before a 9941 following 9942 local/generic store 9943 atomic/atomicrmw 9944 with an equal or 9945 wider sync scope 9946 and memory ordering 9947 stronger than 9948 unordered (this is 9949 termed the 9950 release-fence-paired-atomic). 9951 This satisfies the 9952 requirements of 9953 release. 9954 - Must happen before 9955 the following 9956 buffer_gl0_inv. 9957 - Ensures that the 9958 acquire-fence-paired 9959 atomic has completed 9960 before invalidating 9961 the 9962 cache. Therefore 9963 any following 9964 locations read must 9965 be no older than 9966 the value read by 9967 the 9968 acquire-fence-paired-atomic. 9969 9970 3. buffer_gl0_inv 9971 9972 - If CU wavefront execution 9973 mode, omit. 9974 - Ensures that 9975 following 9976 loads will not see 9977 stale data. 9978 9979 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 9980 - system vmcnt(0) & vscnt(0) 9981 9982 - If OpenCL and 9983 address space is 9984 not generic, omit 9985 lgkmcnt(0). 9986 - If OpenCL and 9987 address space is 9988 local, omit 9989 vmcnt(0) and vscnt(0). 9990 - However, since LLVM 9991 currently has no 9992 address space on 9993 the fence need to 9994 conservatively 9995 always generate 9996 (see comment for 9997 previous fence). 9998 - Could be split into 9999 separate s_waitcnt 10000 vmcnt(0), s_waitcnt 10001 vscnt(0) and s_waitcnt 10002 lgkmcnt(0) to allow 10003 them to be 10004 independently moved 10005 according to the 10006 following rules. 10007 - s_waitcnt vmcnt(0) 10008 must happen after 10009 any preceding 10010 global/generic 10011 load/load 10012 atomic/ 10013 atomicrmw-with-return-value. 10014 - s_waitcnt vscnt(0) 10015 must happen after 10016 any preceding 10017 global/generic 10018 store/store atomic/ 10019 atomicrmw-no-return-value. 10020 - s_waitcnt lgkmcnt(0) 10021 must happen after 10022 any preceding 10023 local/generic 10024 load/store/load 10025 atomic/store 10026 atomic/atomicrmw. 10027 - Must happen before 10028 the following 10029 buffer_gl*_inv. 10030 - Ensures that the 10031 preceding 10032 global/local/generic 10033 load 10034 atomic/atomicrmw 10035 with an equal or 10036 wider sync scope 10037 and memory ordering 10038 stronger than 10039 unordered (this is 10040 termed the 10041 acquire-fence-paired-atomic) 10042 has completed 10043 before invalidating 10044 the caches. This 10045 satisfies the 10046 requirements of 10047 acquire. 10048 - Ensures that all 10049 previous memory 10050 operations have 10051 completed before a 10052 following 10053 global/local/generic 10054 store 10055 atomic/atomicrmw 10056 with an equal or 10057 wider sync scope 10058 and memory ordering 10059 stronger than 10060 unordered (this is 10061 termed the 10062 release-fence-paired-atomic). 10063 This satisfies the 10064 requirements of 10065 release. 10066 10067 2. buffer_gl0_inv; 10068 buffer_gl1_inv 10069 10070 - Must happen before 10071 any following 10072 global/generic 10073 load/load 10074 atomic/store/store 10075 atomic/atomicrmw. 10076 - Ensures that 10077 following loads 10078 will not see stale 10079 global data. This 10080 satisfies the 10081 requirements of 10082 acquire. 10083 10084 **Sequential Consistent Atomic** 10085 ------------------------------------------------------------------------------------ 10086 load atomic seq_cst - singlethread - global *Same as corresponding 10087 - wavefront - local load atomic acquire, 10088 - generic except must generated 10089 all instructions even 10090 for OpenCL.* 10091 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10092 - generic vmcnt(0) & vscnt(0) 10093 10094 - If CU wavefront execution 10095 mode, omit vmcnt(0) and 10096 vscnt(0). 10097 - Could be split into 10098 separate s_waitcnt 10099 vmcnt(0), s_waitcnt 10100 vscnt(0), and s_waitcnt 10101 lgkmcnt(0) to allow 10102 them to be 10103 independently moved 10104 according to the 10105 following rules. 10106 - s_waitcnt lgkmcnt(0) must 10107 happen after 10108 preceding 10109 local/generic load 10110 atomic/store 10111 atomic/atomicrmw 10112 with memory 10113 ordering of seq_cst 10114 and with equal or 10115 wider sync scope. 10116 (Note that seq_cst 10117 fences have their 10118 own s_waitcnt 10119 lgkmcnt(0) and so do 10120 not need to be 10121 considered.) 10122 - s_waitcnt vmcnt(0) 10123 must happen after 10124 preceding 10125 global/generic load 10126 atomic/ 10127 atomicrmw-with-return-value 10128 with memory 10129 ordering of seq_cst 10130 and with equal or 10131 wider sync scope. 10132 (Note that seq_cst 10133 fences have their 10134 own s_waitcnt 10135 vmcnt(0) and so do 10136 not need to be 10137 considered.) 10138 - s_waitcnt vscnt(0) 10139 Must happen after 10140 preceding 10141 global/generic store 10142 atomic/ 10143 atomicrmw-no-return-value 10144 with memory 10145 ordering of seq_cst 10146 and with equal or 10147 wider sync scope. 10148 (Note that seq_cst 10149 fences have their 10150 own s_waitcnt 10151 vscnt(0) and so do 10152 not need to be 10153 considered.) 10154 - Ensures any 10155 preceding 10156 sequential 10157 consistent global/local 10158 memory instructions 10159 have completed 10160 before executing 10161 this sequentially 10162 consistent 10163 instruction. This 10164 prevents reordering 10165 a seq_cst store 10166 followed by a 10167 seq_cst load. (Note 10168 that seq_cst is 10169 stronger than 10170 acquire/release as 10171 the reordering of 10172 load acquire 10173 followed by a store 10174 release is 10175 prevented by the 10176 s_waitcnt of 10177 the release, but 10178 there is nothing 10179 preventing a store 10180 release followed by 10181 load acquire from 10182 completing out of 10183 order. The s_waitcnt 10184 could be placed after 10185 seq_store or before 10186 the seq_load. We 10187 choose the load to 10188 make the s_waitcnt be 10189 as late as possible 10190 so that the store 10191 may have already 10192 completed.) 10193 10194 2. *Following 10195 instructions same as 10196 corresponding load 10197 atomic acquire, 10198 except must generated 10199 all instructions even 10200 for OpenCL.* 10201 load atomic seq_cst - workgroup - local 10202 10203 1. s_waitcnt vmcnt(0) & vscnt(0) 10204 10205 - If CU wavefront execution 10206 mode, omit. 10207 - Could be split into 10208 separate s_waitcnt 10209 vmcnt(0) and s_waitcnt 10210 vscnt(0) to allow 10211 them to be 10212 independently moved 10213 according to the 10214 following rules. 10215 - s_waitcnt vmcnt(0) 10216 Must happen after 10217 preceding 10218 global/generic load 10219 atomic/ 10220 atomicrmw-with-return-value 10221 with memory 10222 ordering of seq_cst 10223 and with equal or 10224 wider sync scope. 10225 (Note that seq_cst 10226 fences have their 10227 own s_waitcnt 10228 vmcnt(0) and so do 10229 not need to be 10230 considered.) 10231 - s_waitcnt vscnt(0) 10232 Must happen after 10233 preceding 10234 global/generic store 10235 atomic/ 10236 atomicrmw-no-return-value 10237 with memory 10238 ordering of seq_cst 10239 and with equal or 10240 wider sync scope. 10241 (Note that seq_cst 10242 fences have their 10243 own s_waitcnt 10244 vscnt(0) and so do 10245 not need to be 10246 considered.) 10247 - Ensures any 10248 preceding 10249 sequential 10250 consistent global 10251 memory instructions 10252 have completed 10253 before executing 10254 this sequentially 10255 consistent 10256 instruction. This 10257 prevents reordering 10258 a seq_cst store 10259 followed by a 10260 seq_cst load. (Note 10261 that seq_cst is 10262 stronger than 10263 acquire/release as 10264 the reordering of 10265 load acquire 10266 followed by a store 10267 release is 10268 prevented by the 10269 s_waitcnt of 10270 the release, but 10271 there is nothing 10272 preventing a store 10273 release followed by 10274 load acquire from 10275 completing out of 10276 order. The s_waitcnt 10277 could be placed after 10278 seq_store or before 10279 the seq_load. We 10280 choose the load to 10281 make the s_waitcnt be 10282 as late as possible 10283 so that the store 10284 may have already 10285 completed.) 10286 10287 2. *Following 10288 instructions same as 10289 corresponding load 10290 atomic acquire, 10291 except must generated 10292 all instructions even 10293 for OpenCL.* 10294 10295 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10296 - system - generic vmcnt(0) & vscnt(0) 10297 10298 - Could be split into 10299 separate s_waitcnt 10300 vmcnt(0), s_waitcnt 10301 vscnt(0) and s_waitcnt 10302 lgkmcnt(0) to allow 10303 them to be 10304 independently moved 10305 according to the 10306 following rules. 10307 - s_waitcnt lgkmcnt(0) 10308 must happen after 10309 preceding 10310 local load 10311 atomic/store 10312 atomic/atomicrmw 10313 with memory 10314 ordering of seq_cst 10315 and with equal or 10316 wider sync scope. 10317 (Note that seq_cst 10318 fences have their 10319 own s_waitcnt 10320 lgkmcnt(0) and so do 10321 not need to be 10322 considered.) 10323 - s_waitcnt vmcnt(0) 10324 must happen after 10325 preceding 10326 global/generic load 10327 atomic/ 10328 atomicrmw-with-return-value 10329 with memory 10330 ordering of seq_cst 10331 and with equal or 10332 wider sync scope. 10333 (Note that seq_cst 10334 fences have their 10335 own s_waitcnt 10336 vmcnt(0) and so do 10337 not need to be 10338 considered.) 10339 - s_waitcnt vscnt(0) 10340 Must happen after 10341 preceding 10342 global/generic store 10343 atomic/ 10344 atomicrmw-no-return-value 10345 with memory 10346 ordering of seq_cst 10347 and with equal or 10348 wider sync scope. 10349 (Note that seq_cst 10350 fences have their 10351 own s_waitcnt 10352 vscnt(0) and so do 10353 not need to be 10354 considered.) 10355 - Ensures any 10356 preceding 10357 sequential 10358 consistent global 10359 memory instructions 10360 have completed 10361 before executing 10362 this sequentially 10363 consistent 10364 instruction. This 10365 prevents reordering 10366 a seq_cst store 10367 followed by a 10368 seq_cst load. (Note 10369 that seq_cst is 10370 stronger than 10371 acquire/release as 10372 the reordering of 10373 load acquire 10374 followed by a store 10375 release is 10376 prevented by the 10377 s_waitcnt of 10378 the release, but 10379 there is nothing 10380 preventing a store 10381 release followed by 10382 load acquire from 10383 completing out of 10384 order. The s_waitcnt 10385 could be placed after 10386 seq_store or before 10387 the seq_load. We 10388 choose the load to 10389 make the s_waitcnt be 10390 as late as possible 10391 so that the store 10392 may have already 10393 completed.) 10394 10395 2. *Following 10396 instructions same as 10397 corresponding load 10398 atomic acquire, 10399 except must generated 10400 all instructions even 10401 for OpenCL.* 10402 store atomic seq_cst - singlethread - global *Same as corresponding 10403 - wavefront - local store atomic release, 10404 - workgroup - generic except must generated 10405 - agent all instructions even 10406 - system for OpenCL.* 10407 atomicrmw seq_cst - singlethread - global *Same as corresponding 10408 - wavefront - local atomicrmw acq_rel, 10409 - workgroup - generic except must generated 10410 - agent all instructions even 10411 - system for OpenCL.* 10412 fence seq_cst - singlethread *none* *Same as corresponding 10413 - wavefront fence acq_rel, 10414 - workgroup except must generated 10415 - agent all instructions even 10416 - system for OpenCL.* 10417 ============ ============ ============== ========== ================================ 10418 10419Trap Handler ABI 10420~~~~~~~~~~~~~~~~ 10421 10422For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10423runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10424supports the ``s_trap`` instruction. For usage see: 10425 10426- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10427- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10428- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10429 10430 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10431 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10432 10433 =================== =============== =============== ======================================= 10434 Usage Code Sequence Trap Handler Description 10435 Inputs 10436 =================== =============== =============== ======================================= 10437 reserved ``s_trap 0x00`` Reserved by hardware. 10438 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10439 ``queue_ptr`` intrinsic (not implemented). 10440 ``VGPR0``: 10441 ``arg`` 10442 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10443 ``queue_ptr`` the trap instruction. The associated 10444 queue is signalled to put it into the 10445 error state. When the queue is put in 10446 the error state, the waves executing 10447 dispatches on the queue will be 10448 terminated. 10449 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10450 as a no-operation. The trap handler 10451 is entered and immediately returns to 10452 continue execution of the wavefront. 10453 - If the debugger is enabled, causes 10454 the debug trap to be reported by the 10455 debugger and the wavefront is put in 10456 the halt state with the PC at the 10457 instruction. The debugger must 10458 increment the PC and resume the wave. 10459 reserved ``s_trap 0x04`` Reserved. 10460 reserved ``s_trap 0x05`` Reserved. 10461 reserved ``s_trap 0x06`` Reserved. 10462 reserved ``s_trap 0x07`` Reserved. 10463 reserved ``s_trap 0x08`` Reserved. 10464 reserved ``s_trap 0xfe`` Reserved. 10465 reserved ``s_trap 0xff`` Reserved. 10466 =================== =============== =============== ======================================= 10467 10468.. 10469 10470 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10471 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10472 10473 =================== =============== =============== ======================================= 10474 Usage Code Sequence Trap Handler Description 10475 Inputs 10476 =================== =============== =============== ======================================= 10477 reserved ``s_trap 0x00`` Reserved by hardware. 10478 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10479 breakpoints. Causes wave to be halted 10480 with the PC at the trap instruction. 10481 The debugger is responsible to resume 10482 the wave, including the instruction 10483 that the breakpoint overwrote. 10484 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10485 ``queue_ptr`` the trap instruction. The associated 10486 queue is signalled to put it into the 10487 error state. When the queue is put in 10488 the error state, the waves executing 10489 dispatches on the queue will be 10490 terminated. 10491 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10492 as a no-operation. The trap handler 10493 is entered and immediately returns to 10494 continue execution of the wavefront. 10495 - If the debugger is enabled, causes 10496 the debug trap to be reported by the 10497 debugger and the wavefront is put in 10498 the halt state with the PC at the 10499 instruction. The debugger must 10500 increment the PC and resume the wave. 10501 reserved ``s_trap 0x04`` Reserved. 10502 reserved ``s_trap 0x05`` Reserved. 10503 reserved ``s_trap 0x06`` Reserved. 10504 reserved ``s_trap 0x07`` Reserved. 10505 reserved ``s_trap 0x08`` Reserved. 10506 reserved ``s_trap 0xfe`` Reserved. 10507 reserved ``s_trap 0xff`` Reserved. 10508 =================== =============== =============== ======================================= 10509 10510.. 10511 10512 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10513 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10514 10515 =================== =============== ================ ================= ======================================= 10516 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10517 =================== =============== ================ ================= ======================================= 10518 reserved ``s_trap 0x00`` Reserved by hardware. 10519 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10520 breakpoints. Causes wave to be halted 10521 with the PC at the trap instruction. 10522 The debugger is responsible to resume 10523 the wave, including the instruction 10524 that the breakpoint overwrote. 10525 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10526 ``queue_ptr`` the trap instruction. The associated 10527 queue is signalled to put it into the 10528 error state. When the queue is put in 10529 the error state, the waves executing 10530 dispatches on the queue will be 10531 terminated. 10532 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10533 as a no-operation. The trap handler 10534 is entered and immediately returns to 10535 continue execution of the wavefront. 10536 - If the debugger is enabled, causes 10537 the debug trap to be reported by the 10538 debugger and the wavefront is put in 10539 the halt state with the PC at the 10540 instruction. The debugger must 10541 increment the PC and resume the wave. 10542 reserved ``s_trap 0x04`` Reserved. 10543 reserved ``s_trap 0x05`` Reserved. 10544 reserved ``s_trap 0x06`` Reserved. 10545 reserved ``s_trap 0x07`` Reserved. 10546 reserved ``s_trap 0x08`` Reserved. 10547 reserved ``s_trap 0xfe`` Reserved. 10548 reserved ``s_trap 0xff`` Reserved. 10549 =================== =============== ================ ================= ======================================= 10550 10551.. _amdgpu-amdhsa-function-call-convention: 10552 10553Call Convention 10554~~~~~~~~~~~~~~~ 10555 10556.. note:: 10557 10558 This section is currently incomplete and has inaccuracies. It is WIP that will 10559 be updated as information is determined. 10560 10561See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10562addresses. Unswizzled addresses are normal linear addresses. 10563 10564.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10565 10566Kernel Functions 10567++++++++++++++++ 10568 10569This section describes the call convention ABI for the outer kernel function. 10570 10571See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10572convention. 10573 10574The following is not part of the AMDGPU kernel calling convention but describes 10575how the AMDGPU implements function calls: 10576 105771. Clang decides the kernarg layout to match the *HSA Programmer's Language 10578 Reference* [HSA]_. 10579 10580 - All structs are passed directly. 10581 - Lambda values are passed *TBA*. 10582 10583 .. TODO:: 10584 10585 - Does this really follow HSA rules? Or are structs >16 bytes passed 10586 by-value struct? 10587 - What is ABI for lambda values? 10588 105894. The kernel performs certain setup in its prolog, as described in 10590 :ref:`amdgpu-amdhsa-kernel-prolog`. 10591 10592.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10593 10594Non-Kernel Functions 10595++++++++++++++++++++ 10596 10597This section describes the call convention ABI for functions other than the 10598outer kernel function. 10599 10600If a kernel has function calls then scratch is always allocated and used for 10601the call stack which grows from low address to high address using the swizzled 10602scratch address space. 10603 10604On entry to a function: 10605 106061. SGPR0-3 contain a V# with the following properties (see 10607 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10608 10609 * Base address pointing to the beginning of the wavefront scratch backing 10610 memory. 10611 * Swizzled with dword element size and stride of wavefront size elements. 10612 106132. The FLAT_SCRATCH register pair is setup. See 10614 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 106153. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10616 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 106174. The EXEC register is set to the lanes active on entry to the function. 106185. MODE register: *TBD* 106196. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10620 below. 106217. SGPR30-31 return address (RA). The code address that the function must 10622 return to when it completes. The value is undefined if the function is *no 10623 return*. 106248. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10625 offset relative to the beginning of the wavefront scratch backing memory. 10626 10627 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10628 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10629 manner. 10630 10631 The unswizzled SP value can be converted into the swizzled SP value by: 10632 10633 | swizzled SP = unswizzled SP / wavefront size 10634 10635 This may be used to obtain the private address space address of stack 10636 objects and to convert this address to a flat address by adding the flat 10637 scratch aperture base address. 10638 10639 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10640 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10641 10642 .. note:: 10643 10644 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10645 OpenCL language which has the largest base type defined as 16 bytes. 10646 10647 On entry, the swizzled SP value is the address of the first function 10648 argument passed on the stack. Other stack passed arguments are positive 10649 offsets from the entry swizzled SP value. 10650 10651 The function may use positive offsets beyond the last stack passed argument 10652 for stack allocated local variables and register spill slots. If necessary, 10653 the function may align these to greater alignment than 16 bytes. After these 10654 the function may dynamically allocate space for such things as runtime sized 10655 ``alloca`` local allocations. 10656 10657 If the function calls another function, it will place any stack allocated 10658 arguments after the last local allocation and adjust SGPR32 to the address 10659 after the last local allocation. 10660 106619. All other registers are unspecified. 1066210. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10663 to the function. 10664 10665On exit from a function: 10666 106671. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10668 described below. Any registers used are considered clobbered registers. 106692. The following registers are preserved and have the same value as on entry: 10670 10671 * FLAT_SCRATCH 10672 * EXEC 10673 * GFX6-GFX8: M0 10674 * All SGPR registers except the clobbered registers of SGPR4-31. 10675 * VGPR40-47 10676 * VGPR56-63 10677 * VGPR72-79 10678 * VGPR88-95 10679 * VGPR104-111 10680 * VGPR120-127 10681 * VGPR136-143 10682 * VGPR152-159 10683 * VGPR168-175 10684 * VGPR184-191 10685 * VGPR200-207 10686 * VGPR216-223 10687 * VGPR232-239 10688 * VGPR248-255 10689 10690 .. note:: 10691 10692 Except the argument registers, the VGPRs clobbered and the preserved 10693 registers are intermixed at regular intervals in order to keep a 10694 similar ratio independent of the number of allocated VGPRs. 10695 10696 * Lanes of all VGPRs that are inactive at the call site. 10697 10698 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10699 optimization may mark some of clobbered SGPR and VGPR registers as 10700 preserved if it can be determined that the called function does not change 10701 their value. 10702 107032. The PC is set to the RA provided on entry. 107043. MODE register: *TBD*. 107054. All other registers are clobbered. 107065. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10707 function is available to the caller. 10708 10709.. TODO:: 10710 10711 - On gfx908 are all ACC registers clobbered? 10712 10713 - How are function results returned? The address of structured types is passed 10714 by reference, but what about other types? 10715 10716The function input arguments are made up of the formal arguments explicitly 10717declared by the source language function plus the implicit input arguments used 10718by the implementation. 10719 10720The source language input arguments are: 10721 107221. Any source language implicit ``this`` or ``self`` argument comes first as a 10723 pointer type. 107242. Followed by the function formal arguments in left to right source order. 10725 10726The source language result arguments are: 10727 107281. The function result argument. 10729 10730The source language input or result struct type arguments that are less than or 10731equal to 16 bytes, are decomposed recursively into their base type fields, and 10732each field is passed as if a separate argument. For input arguments, if the 10733called function requires the struct to be in memory, for example because its 10734address is taken, then the function body is responsible for allocating a stack 10735location and copying the field arguments into it. Clang terms this *direct 10736struct*. 10737 10738The source language input struct type arguments that are greater than 16 bytes, 10739are passed by reference. The caller is responsible for allocating a stack 10740location to make a copy of the struct value and pass the address as the input 10741argument. The called function is responsible to perform the dereference when 10742accessing the input argument. Clang terms this *by-value struct*. 10743 10744A source language result struct type argument that is greater than 16 bytes, is 10745returned by reference. The caller is responsible for allocating a stack location 10746to hold the result value and passes the address as the last input argument 10747(before the implicit input arguments). In this case there are no result 10748arguments. The called function is responsible to perform the dereference when 10749storing the result value. Clang terms this *structured return (sret)*. 10750 10751*TODO: correct the ``sret`` definition.* 10752 10753.. TODO:: 10754 10755 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10756 pass as non-decomposed struct as stack argument? Or something else? Is the 10757 memory location in the caller stack frame, or a stack memory argument and so 10758 no address is passed as the caller can directly write to the argument stack 10759 location? But then the stack location is still live after return. If an 10760 argument stack location is it the first stack argument or the last one? 10761 10762Lambda argument types are treated as struct types with an implementation defined 10763set of fields. 10764 10765.. TODO:: 10766 10767 Need to specify the ABI for lambda types for AMDGPU. 10768 10769For AMDGPU backend all source language arguments (including the decomposed 10770struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10771they are passed in SGPRs. 10772 10773The AMDGPU backend walks the function call graph from the leaves to determine 10774which implicit input arguments are used, propagating to each caller of the 10775function. The used implicit arguments are appended to the function arguments 10776after the source language arguments in the following order: 10777 10778.. TODO:: 10779 10780 Is recursion or external functions supported? 10781 107821. Work-Item ID (1 VGPR) 10783 10784 The X, Y and Z work-item ID are packed into a single VGRP with the following 10785 layout. Only fields actually used by the function are set. The other bits 10786 are undefined. 10787 10788 The values come from the initial kernel execution state. See 10789 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10790 10791 .. table:: Work-item implicit argument layout 10792 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10793 10794 ======= ======= ============== 10795 Bits Size Field Name 10796 ======= ======= ============== 10797 9:0 10 bits X Work-Item ID 10798 19:10 10 bits Y Work-Item ID 10799 29:20 10 bits Z Work-Item ID 10800 31:30 2 bits Unused 10801 ======= ======= ============== 10802 108032. Dispatch Ptr (2 SGPRs) 10804 10805 The value comes from the initial kernel execution state. See 10806 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10807 108083. Queue Ptr (2 SGPRs) 10809 10810 The value comes from the initial kernel execution state. See 10811 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10812 108134. Kernarg Segment Ptr (2 SGPRs) 10814 10815 The value comes from the initial kernel execution state. See 10816 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10817 108185. Dispatch id (2 SGPRs) 10819 10820 The value comes from the initial kernel execution state. See 10821 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10822 108236. Work-Group ID X (1 SGPR) 10824 10825 The value comes from the initial kernel execution state. See 10826 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10827 108287. Work-Group ID Y (1 SGPR) 10829 10830 The value comes from the initial kernel execution state. See 10831 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10832 108338. Work-Group ID Z (1 SGPR) 10834 10835 The value comes from the initial kernel execution state. See 10836 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10837 108389. Implicit Argument Ptr (2 SGPRs) 10839 10840 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10841 global address space pointer to the first kernarg implicit argument. 10842 10843The input and result arguments are assigned in order in the following manner: 10844 10845.. note:: 10846 10847 There are likely some errors and omissions in the following description that 10848 need correction. 10849 10850 .. TODO:: 10851 10852 Check the Clang source code to decipher how function arguments and return 10853 results are handled. Also see the AMDGPU specific values used. 10854 10855* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 10856 VGPR31. 10857 10858 If there are more arguments than will fit in these registers, the remaining 10859 arguments are allocated on the stack in order on naturally aligned 10860 addresses. 10861 10862 .. TODO:: 10863 10864 How are overly aligned structures allocated on the stack? 10865 10866* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 10867 SGPR29. 10868 10869 If there are more arguments than will fit in these registers, the remaining 10870 arguments are allocated on the stack in order on naturally aligned 10871 addresses. 10872 10873Note that decomposed struct type arguments may have some fields passed in 10874registers and some in memory. 10875 10876.. TODO:: 10877 10878 So, a struct which can pass some fields as decomposed register arguments, will 10879 pass the rest as decomposed stack elements? But an argument that will not start 10880 in registers will not be decomposed and will be passed as a non-decomposed 10881 stack value? 10882 10883The following is not part of the AMDGPU function calling convention but 10884describes how the AMDGPU implements function calls: 10885 108861. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 10887 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 10888 are used, or for the reasons defined in ``SIFrameLowering``. 108892. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 10890 to access the incoming stack arguments in the function. The BP is needed 10891 only when the function requires the runtime stack alignment. 10892 108933. Allocating SGPR arguments on the stack are not supported. 10894 108954. No CFI is currently generated. See 10896 :ref:`amdgpu-dwarf-call-frame-information`. 10897 10898 .. note:: 10899 10900 CFI will be generated that defines the CFA as the unswizzled address 10901 relative to the wave scratch base in the unswizzled private address space 10902 of the lowest address stack allocated local variable. 10903 10904 ``DW_AT_frame_base`` will be defined as the swizzled address in the 10905 swizzled private address space by dividing the CFA by the wavefront size 10906 (since CFA is always at least dword aligned which matches the scratch 10907 swizzle element size). 10908 10909 If no dynamic stack alignment was performed, the stack allocated arguments 10910 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 10911 local variables and register spill slots are accessed as positive offsets 10912 relative to ``DW_AT_frame_base``. 10913 109145. Function argument passing is implemented by copying the input physical 10915 registers to virtual registers on entry. The register allocator can spill if 10916 necessary. These are copied back to physical registers at call sites. The 10917 net effect is that each function call can have these values in entirely 10918 distinct locations. The IPRA can help avoid shuffling argument registers. 109196. Call sites are implemented by setting up the arguments at positive offsets 10920 from SP. Then SP is incremented to account for the known frame size before 10921 the call and decremented after the call. 10922 10923 .. note:: 10924 10925 The CFI will reflect the changed calculation needed to compute the CFA 10926 from SP. 10927 109287. 4 byte spill slots are used in the stack frame. One slot is allocated for an 10929 emergency spill slot. Buffer instructions are used for stack accesses and 10930 not the ``flat_scratch`` instruction. 10931 10932 .. TODO:: 10933 10934 Explain when the emergency spill slot is used. 10935 10936.. TODO:: 10937 10938 Possible broken issues: 10939 10940 - Stack arguments must be aligned to required alignment. 10941 - Stack is aligned to max(16, max formal argument alignment) 10942 - Direct argument < 64 bits should check register budget. 10943 - Register budget calculation should respect ``inreg`` for SGPR. 10944 - SGPR overflow is not handled. 10945 - struct with 1 member unpeeling is not checking size of member. 10946 - ``sret`` is after ``this`` pointer. 10947 - Caller is not implementing stack realignment: need an extra pointer. 10948 - Should say AMDGPU passes FP rather than SP. 10949 - Should CFI define CFA as address of locals or arguments. Difference is 10950 apparent when have implemented dynamic alignment. 10951 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 10952 highest address of stack frame and use negative offset for locals. Would 10953 allow SP to be the same as FP and could support signal-handler-like as now 10954 have a real SP for the top of the stack. 10955 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 10956 arguments? 10957 10958AMDPAL 10959------ 10960 10961This section provides code conventions used when the target triple OS is 10962``amdpal`` (see :ref:`amdgpu-target-triples`). 10963 10964.. _amdgpu-amdpal-code-object-metadata-section: 10965 10966Code Object Metadata 10967~~~~~~~~~~~~~~~~~~~~ 10968 10969.. note:: 10970 10971 The metadata is currently in development and is subject to major 10972 changes. Only the current version is supported. *When this document 10973 was generated the version was 2.6.* 10974 10975Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 10976record (see :ref:`amdgpu-note-records-v3-v4`). 10977 10978The metadata is represented as Message Pack formatted binary data (see 10979[MsgPack]_). The top level is a Message Pack map that includes the keys 10980defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 10981and referenced tables. 10982 10983Additional information can be added to the maps. To avoid conflicts, any 10984key names should be prefixed by "*vendor-name*." where ``vendor-name`` 10985can be the name of the vendor and specific vendor tool that generates the 10986information. The prefix is abbreviated to simply "." when it appears 10987within a map that has been added by the same *vendor-name*. 10988 10989 .. table:: AMDPAL Code Object Metadata Map 10990 :name: amdgpu-amdpal-code-object-metadata-map-table 10991 10992 =================== ============== ========= ====================================================================== 10993 String Key Value Type Required? Description 10994 =================== ============== ========= ====================================================================== 10995 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 10996 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 10997 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 10998 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 10999 definition of the keys included in that map. 11000 =================== ============== ========= ====================================================================== 11001 11002.. 11003 11004 .. table:: AMDPAL Code Object Pipeline Metadata Map 11005 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 11006 11007 ====================================== ============== ========= =================================================== 11008 String Key Value Type Required? Description 11009 ====================================== ============== ========= =================================================== 11010 ".name" string Source name of the pipeline. 11011 ".type" string Pipeline type, e.g. VsPs. Values include: 11012 11013 - "VsPs" 11014 - "Gs" 11015 - "Cs" 11016 - "Ngg" 11017 - "Tess" 11018 - "GsTess" 11019 - "NggTess" 11020 11021 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 11022 2 integers 64 bits is the "stable" portion of the hash, used 11023 for e.g. shader replacement lookup. Upper 64 bits 11024 is the "unique" portion of the hash, used for 11025 e.g. pipeline cache lookup. The value is 11026 implementation defined, and can not be relied on 11027 between different builds of the compiler. 11028 ".shaders" map Per-API shader metadata. See 11029 :ref:`amdgpu-amdpal-code-object-shader-map-table` 11030 for the definition of the keys included in that 11031 map. 11032 ".hardware_stages" map Per-hardware stage metadata. See 11033 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 11034 for the definition of the keys included in that 11035 map. 11036 ".shader_functions" map Per-shader function metadata. See 11037 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 11038 for the definition of the keys included in that 11039 map. 11040 ".registers" map Required Hardware register configuration. See 11041 :ref:`amdgpu-amdpal-code-object-register-map-table` 11042 for the definition of the keys included in that 11043 map. 11044 ".user_data_limit" integer Number of user data entries accessed by this 11045 pipeline. 11046 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 11047 NoUserDataSpilling. 11048 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 11049 viewport array index feature. Pipelines which use 11050 this feature can render into all 16 viewports, 11051 whereas pipelines which do not use it are 11052 restricted to viewport #0. 11053 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 11054 handling data-passing between the ES and GS 11055 shader stages. This can be zero if the data is 11056 passed using off-chip buffers. This value should 11057 be used to program all user-SGPRs which have been 11058 marked with "UserDataMapping::EsGsLdsSize" 11059 (typically only the GS and VS HW stages will ever 11060 have a user-SGPR so marked). 11061 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 11062 (maximum number of threads in a subgroup). 11063 ".num_interpolants" integer Graphics only. Number of PS interpolants. 11064 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 11065 ".api" string Name of the client graphics API. 11066 ".api_create_info" binary Graphics API shader create info binary blob. Can 11067 be defined by the driver using the compiler if 11068 they want to be able to correlate API-specific 11069 information used during creation at a later time. 11070 ====================================== ============== ========= =================================================== 11071 11072.. 11073 11074 .. table:: AMDPAL Code Object Shader Map 11075 :name: amdgpu-amdpal-code-object-shader-map-table 11076 11077 11078 +-------------+--------------+-------------------------------------------------------------------+ 11079 |String Key |Value Type |Description | 11080 +=============+==============+===================================================================+ 11081 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 11082 |- ".vertex" | |for the definition of the keys included in that map. | 11083 |- ".hull" | | | 11084 |- ".domain" | | | 11085 |- ".geometry"| | | 11086 |- ".pixel" | | | 11087 +-------------+--------------+-------------------------------------------------------------------+ 11088 11089.. 11090 11091 .. table:: AMDPAL Code Object API Shader Metadata Map 11092 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 11093 11094 ==================== ============== ========= ===================================================================== 11095 String Key Value Type Required? Description 11096 ==================== ============== ========= ===================================================================== 11097 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 11098 2 integers is implementation defined, and can not be relied on between 11099 different builds of the compiler. 11100 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 11101 string include: 11102 11103 - ".ls" 11104 - ".hs" 11105 - ".es" 11106 - ".gs" 11107 - ".vs" 11108 - ".ps" 11109 - ".cs" 11110 11111 ==================== ============== ========= ===================================================================== 11112 11113.. 11114 11115 .. table:: AMDPAL Code Object Hardware Stage Map 11116 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 11117 11118 +-------------+--------------+-----------------------------------------------------------------------+ 11119 |String Key |Value Type |Description | 11120 +=============+==============+=======================================================================+ 11121 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 11122 |- ".hs" | |for the definition of the keys included in that map. | 11123 |- ".es" | | | 11124 |- ".gs" | | | 11125 |- ".vs" | | | 11126 |- ".ps" | | | 11127 |- ".cs" | | | 11128 +-------------+--------------+-----------------------------------------------------------------------+ 11129 11130.. 11131 11132 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 11133 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 11134 11135 ========================== ============== ========= =============================================================== 11136 String Key Value Type Required? Description 11137 ========================== ============== ========= =============================================================== 11138 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 11139 ".scratch_memory_size" integer Scratch memory size in bytes. 11140 ".lds_size" integer Local Data Share size in bytes. 11141 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 11142 ".vgpr_count" integer Number of VGPRs used. 11143 ".sgpr_count" integer Number of SGPRs used. 11144 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 11145 directive to instruct the compiler to limit the VGPR usage to 11146 be less than or equal to the specified value (only set if 11147 different from HW default). 11148 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 11149 default). 11150 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 11151 3 integers 11152 ".wavefront_size" integer Wavefront size (only set if different from HW default). 11153 ".uses_uavs" boolean The shader reads or writes UAVs. 11154 ".uses_rovs" boolean The shader reads or writes ROVs. 11155 ".writes_uavs" boolean The shader writes to one or more UAVs. 11156 ".writes_depth" boolean The shader writes out a depth value. 11157 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 11158 memory or GDS. 11159 ".uses_prim_id" boolean The shader uses PrimID. 11160 ========================== ============== ========= =============================================================== 11161 11162.. 11163 11164 .. table:: AMDPAL Code Object Shader Function Map 11165 :name: amdgpu-amdpal-code-object-shader-function-map-table 11166 11167 =============== ============== ==================================================================== 11168 String Key Value Type Description 11169 =============== ============== ==================================================================== 11170 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 11171 entry address. The value is the function's metadata. See 11172 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 11173 =============== ============== ==================================================================== 11174 11175.. 11176 11177 .. table:: AMDPAL Code Object Shader Function Metadata Map 11178 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 11179 11180 ============================= ============== ================================================================= 11181 String Key Value Type Description 11182 ============================= ============== ================================================================= 11183 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 11184 2 integers is implementation defined, and can not be relied on between 11185 different builds of the compiler. 11186 ".scratch_memory_size" sequence of Size in bytes of scratch memory used by the shader. 11187 2 integers 11188 ".lds_size" sequence of Size in bytes of LDS memory. 11189 2 integers 11190 ".vgpr_count" integer Number of VGPRs used by the shader. 11191 ".sgpr_count" integer Number of SGPRs used by the shader. 11192 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 11193 ".shader_subtype" string Shader subtype/kind. Values include: 11194 11195 - "Unknown" 11196 11197 ============================= ============== ================================================================= 11198 11199.. 11200 11201 .. table:: AMDPAL Code Object Register Map 11202 :name: amdgpu-amdpal-code-object-register-map-table 11203 11204 ========================== ============== ==================================================================== 11205 32-bit Integer Key Value Type Description 11206 ========================== ============== ==================================================================== 11207 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 11208 a GRBM register (i.e., driver accessible GPU register number, not 11209 shader GPR register number). The driver is required to program each 11210 specified register to the corresponding specified value when 11211 executing this pipeline. Typically, the ``reg offsets`` are the 11212 ``uint16_t`` offsets to each register as defined by the hardware 11213 chip headers. The register is set to the provided value. However, a 11214 ``reg offset`` that specifies a user data register (e.g., 11215 COMPUTE_USER_DATA_0) needs special treatment. See 11216 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 11217 information. 11218 ========================== ============== ==================================================================== 11219 11220.. _amdgpu-amdpal-code-object-user-data-section: 11221 11222User Data 11223+++++++++ 11224 11225Each hardware stage has a set of 32-bit physical SPI *user data registers* 11226(either 16 or 32 based on graphics IP and the stage) which can be 11227written from a command buffer and then loaded into SGPRs when waves are 11228launched via a subsequent dispatch or draw operation. This is the way 11229most arguments are passed from the application/runtime to a hardware 11230shader. 11231 11232PAL abstracts this functionality by exposing a set of 128 *user data 11233entries* per pipeline a client can use to pass arguments from a command 11234buffer to one or more shaders in that pipeline. The ELF code object must 11235specify a mapping from virtualized *user data entries* to physical *user 11236data registers*, and PAL is responsible for implementing that mapping, 11237including spilling overflow *user data entries* to memory if needed. 11238 11239Since the *user data registers* are GRBM-accessible SPI registers, this 11240mapping is actually embedded in the ``.registers`` metadata entry. For 11241most registers, the value in that map is a literal 32-bit value that 11242should be written to the register by the driver. However, when the 11243register is a *user data register* (any USER_DATA register e.g., 11244SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 11245the driver to write either a *user data entry* value or one of several 11246driver-internal values to the register. This encoding is described in 11247the following table: 11248 11249.. note:: 11250 11251 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 11252 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 11253 always be programmed to the address of the GlobalTable, and *user data 11254 register* 1 must always be programmed to the address of the PerShaderTable. 11255 11256.. 11257 11258 .. table:: AMDPAL User Data Mapping 11259 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 11260 11261 ========== ================= =============================================================================== 11262 Value Name Description 11263 ========== ================= =============================================================================== 11264 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 11265 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 11266 always point to *user data register* 0). 11267 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 11268 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 11269 for more detail (should always point to *user data register* 1). 11270 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 11271 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 11272 more detail. 11273 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 11274 reference the draw index in the vertex shader. Only supported by the first 11275 stage in a graphics pipeline. 11276 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 11277 a graphics pipeline. 11278 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 11279 graphics pipeline. 11280 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 11281 a buffer containing the grid dimensions for a Compute dispatch operation. The 11282 high half of the address is stored in the next sequential user-SGPR. Only 11283 supported by compute pipelines. 11284 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 11285 space used for the ES/GS pseudo-ring-buffer for passing data between shader 11286 stages. 11287 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 11288 pipeline instancing. 11289 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 11290 can only appear for one shader stage per pipeline. 11291 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 11292 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 11293 only appear for one shader stage per pipeline. 11294 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 11295 only appear for one shader stage per pipeline (PS). These replace color targets 11296 and are completely separate from any UAVs used by the shader. This is optional, 11297 and only used by the PS when UAV exports are used to replace color-target 11298 exports to optimize specific shaders. 11299 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 11300 some NGG pipelines to perform culling. This value contains the address of the 11301 first of two consecutive registers which provide the full GPU address. 11302 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 11303 ========== ================= =============================================================================== 11304 11305.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 11306 11307Per-Shader Table 11308################ 11309 11310Low 32 bits of the GPU address for an optional buffer in the ``.data`` 11311section of the ELF. The high 32 bits of the address match the high 32 bits 11312of the shader's program counter. 11313 11314The buffer can be anything the shader compiler needs it for, and 11315allows each shader to have its own region of the ``.data`` section. 11316Typically, this could be a table of buffer SRD's and the data pointed to 11317by the buffer SRD's, but it could be a flat-address region of memory as 11318well. Its layout and usage are defined by the shader compiler. 11319 11320Each shader's table in the ``.data`` section is referenced by the symbol 11321``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 11322hardware shader stage the data is for. E.g., 11323``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 11324 11325.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 11326 11327Spill Table 11328########### 11329 11330It is possible for a hardware shader to need access to more *user data 11331entries* than there are slots available in user data registers for one 11332or more hardware shader stages. In that case, the PAL runtime expects 11333the necessary *user data entries* to be spilled to GPU memory and use 11334one user data register to point to the spilled user data memory. The 11335value of the *user data entry* must then represent the location where 11336a shader expects to read the low 32-bits of the table's GPU virtual 11337address. The *spill table* itself represents a set of 32-bit values 11338managed by the PAL runtime in GPU-accessible memory that can be made 11339indirectly accessible to a hardware shader. 11340 11341Unspecified OS 11342-------------- 11343 11344This section provides code conventions used when the target triple OS is 11345empty (see :ref:`amdgpu-target-triples`). 11346 11347Trap Handler ABI 11348~~~~~~~~~~~~~~~~ 11349 11350For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11351not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11352instructions are handled as follows: 11353 11354 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11355 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11356 11357 =============== =============== =========================================== 11358 Usage Code Sequence Description 11359 =============== =============== =========================================== 11360 llvm.trap s_endpgm Causes wavefront to be terminated. 11361 llvm.debugtrap *none* Compiler warning given that there is no 11362 trap handler installed. 11363 =============== =============== =========================================== 11364 11365Source Languages 11366================ 11367 11368.. _amdgpu-opencl: 11369 11370OpenCL 11371------ 11372 11373When the language is OpenCL the following differences occur: 11374 113751. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 113762. The AMDGPU backend appends additional arguments to the kernel's explicit 11377 arguments for the AMDHSA OS (see 11378 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 113793. Additional metadata is generated 11380 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11381 11382 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11383 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11384 11385 ======== ==== ========= =========================================== 11386 Position Byte Byte Description 11387 Size Alignment 11388 ======== ==== ========= =========================================== 11389 1 8 8 OpenCL Global Offset X 11390 2 8 8 OpenCL Global Offset Y 11391 3 8 8 OpenCL Global Offset Z 11392 4 8 8 OpenCL address of printf buffer 11393 5 8 8 OpenCL address of virtual queue used by 11394 enqueue_kernel. 11395 6 8 8 OpenCL address of AqlWrap struct used by 11396 enqueue_kernel. 11397 7 8 8 Pointer argument used for Multi-gird 11398 synchronization. 11399 ======== ==== ========= =========================================== 11400 11401.. _amdgpu-hcc: 11402 11403HCC 11404--- 11405 11406When the language is HCC the following differences occur: 11407 114081. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11409 11410.. _amdgpu-assembler: 11411 11412Assembler 11413--------- 11414 11415AMDGPU backend has LLVM-MC based assembler which is currently in development. 11416It supports AMDGCN GFX6-GFX10. 11417 11418This section describes general syntax for instructions and operands. 11419 11420Instructions 11421~~~~~~~~~~~~ 11422 11423An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11424 11425 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11426 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11427 11428:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11429:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11430 11431The order of operands and modifiers is fixed. 11432Most modifiers are optional and may be omitted. 11433 11434Links to detailed instruction syntax description may be found in the following 11435table. Note that features under development are not included 11436in this description. 11437 11438 =================================== ======================================= 11439 Core ISA ISA Extensions 11440 =================================== ======================================= 11441 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11442 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11443 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11444 11445 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11446 11447 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11448 11449 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11450 11451 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11452 11453 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11454 11455 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11456 11457 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11458 =================================== ======================================= 11459 11460For more information about instructions, their semantics and supported 11461combinations of operands, refer to one of instruction set architecture manuals 11462[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_, 11463[AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11464 11465Operands 11466~~~~~~~~ 11467 11468Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11469 11470Modifiers 11471~~~~~~~~~ 11472 11473Detailed description of modifiers may be found 11474:doc:`here<AMDGPUModifierSyntax>`. 11475 11476Instruction Examples 11477~~~~~~~~~~~~~~~~~~~~ 11478 11479DS 11480++ 11481 11482.. code-block:: nasm 11483 11484 ds_add_u32 v2, v4 offset:16 11485 ds_write_src2_b64 v2 offset0:4 offset1:8 11486 ds_cmpst_f32 v2, v4, v6 11487 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11488 11489For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11490Manual. 11491 11492FLAT 11493++++ 11494 11495.. code-block:: nasm 11496 11497 flat_load_dword v1, v[3:4] 11498 flat_store_dwordx3 v[3:4], v[5:7] 11499 flat_atomic_swap v1, v[3:4], v5 glc 11500 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11501 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11502 11503For full list of supported instructions, refer to "FLAT instructions" in ISA 11504Manual. 11505 11506MUBUF 11507+++++ 11508 11509.. code-block:: nasm 11510 11511 buffer_load_dword v1, off, s[4:7], s1 11512 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11513 buffer_store_format_xy v[1:2], off, s[4:7], s1 11514 buffer_wbinvl1 11515 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11516 11517For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11518Manual. 11519 11520SMRD/SMEM 11521+++++++++ 11522 11523.. code-block:: nasm 11524 11525 s_load_dword s1, s[2:3], 0xfc 11526 s_load_dwordx8 s[8:15], s[2:3], s4 11527 s_load_dwordx16 s[88:103], s[2:3], s4 11528 s_dcache_inv_vol 11529 s_memtime s[4:5] 11530 11531For full list of supported instructions, refer to "Scalar Memory Operations" in 11532ISA Manual. 11533 11534SOP1 11535++++ 11536 11537.. code-block:: nasm 11538 11539 s_mov_b32 s1, s2 11540 s_mov_b64 s[0:1], 0x80000000 11541 s_cmov_b32 s1, 200 11542 s_wqm_b64 s[2:3], s[4:5] 11543 s_bcnt0_i32_b64 s1, s[2:3] 11544 s_swappc_b64 s[2:3], s[4:5] 11545 s_cbranch_join s[4:5] 11546 11547For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11548Manual. 11549 11550SOP2 11551++++ 11552 11553.. code-block:: nasm 11554 11555 s_add_u32 s1, s2, s3 11556 s_and_b64 s[2:3], s[4:5], s[6:7] 11557 s_cselect_b32 s1, s2, s3 11558 s_andn2_b32 s2, s4, s6 11559 s_lshr_b64 s[2:3], s[4:5], s6 11560 s_ashr_i32 s2, s4, s6 11561 s_bfm_b64 s[2:3], s4, s6 11562 s_bfe_i64 s[2:3], s[4:5], s6 11563 s_cbranch_g_fork s[4:5], s[6:7] 11564 11565For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11566Manual. 11567 11568SOPC 11569++++ 11570 11571.. code-block:: nasm 11572 11573 s_cmp_eq_i32 s1, s2 11574 s_bitcmp1_b32 s1, s2 11575 s_bitcmp0_b64 s[2:3], s4 11576 s_setvskip s3, s5 11577 11578For full list of supported instructions, refer to "SOPC Instructions" in ISA 11579Manual. 11580 11581SOPP 11582++++ 11583 11584.. code-block:: nasm 11585 11586 s_barrier 11587 s_nop 2 11588 s_endpgm 11589 s_waitcnt 0 ; Wait for all counters to be 0 11590 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11591 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11592 s_sethalt 9 11593 s_sleep 10 11594 s_sendmsg 0x1 11595 s_sendmsg sendmsg(MSG_INTERRUPT) 11596 s_trap 1 11597 11598For full list of supported instructions, refer to "SOPP Instructions" in ISA 11599Manual. 11600 11601Unless otherwise mentioned, little verification is performed on the operands 11602of SOPP Instructions, so it is up to the programmer to be familiar with the 11603range or acceptable values. 11604 11605VALU 11606++++ 11607 11608For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11609the assembler will automatically use optimal encoding based on its operands. To 11610force specific encoding, one can add a suffix to the opcode of the instruction: 11611 11612* _e32 for 32-bit VOP1/VOP2/VOPC 11613* _e64 for 64-bit VOP3 11614* _dpp for VOP_DPP 11615* _sdwa for VOP_SDWA 11616 11617VOP1/VOP2/VOP3/VOPC examples: 11618 11619.. code-block:: nasm 11620 11621 v_mov_b32 v1, v2 11622 v_mov_b32_e32 v1, v2 11623 v_nop 11624 v_cvt_f64_i32_e32 v[1:2], v2 11625 v_floor_f32_e32 v1, v2 11626 v_bfrev_b32_e32 v1, v2 11627 v_add_f32_e32 v1, v2, v3 11628 v_mul_i32_i24_e64 v1, v2, 3 11629 v_mul_i32_i24_e32 v1, -3, v3 11630 v_mul_i32_i24_e32 v1, -100, v3 11631 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11632 v_max_f16_e32 v1, v2, v3 11633 11634VOP_DPP examples: 11635 11636.. code-block:: nasm 11637 11638 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11639 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11640 v_mov_b32 v0, v0 wave_shl:1 11641 v_mov_b32 v0, v0 row_mirror 11642 v_mov_b32 v0, v0 row_bcast:31 11643 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11644 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11645 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11646 11647VOP_SDWA examples: 11648 11649.. code-block:: nasm 11650 11651 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11652 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11653 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11654 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11655 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11656 11657For full list of supported instructions, refer to "Vector ALU instructions". 11658 11659.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11660 11661Code Object V2 Predefined Symbols 11662~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11663 11664.. warning:: 11665 Code object V2 is not the default code object version emitted by 11666 this version of LLVM. 11667 11668The AMDGPU assembler defines and updates some symbols automatically. These 11669symbols do not affect code generation. 11670 11671.option.machine_version_major 11672+++++++++++++++++++++++++++++ 11673 11674Set to the GFX major generation number of the target being assembled for. For 11675example, when assembling for a "GFX9" target this will be set to the integer 11676value "9". The possible GFX major generation numbers are presented in 11677:ref:`amdgpu-processors`. 11678 11679.option.machine_version_minor 11680+++++++++++++++++++++++++++++ 11681 11682Set to the GFX minor generation number of the target being assembled for. For 11683example, when assembling for a "GFX810" target this will be set to the integer 11684value "1". The possible GFX minor generation numbers are presented in 11685:ref:`amdgpu-processors`. 11686 11687.option.machine_version_stepping 11688++++++++++++++++++++++++++++++++ 11689 11690Set to the GFX stepping generation number of the target being assembled for. 11691For example, when assembling for a "GFX704" target this will be set to the 11692integer value "4". The possible GFX stepping generation numbers are presented 11693in :ref:`amdgpu-processors`. 11694 11695.kernel.vgpr_count 11696++++++++++++++++++ 11697 11698Set to zero each time a 11699:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11700encountered. At each instruction, if the current value of this symbol is less 11701than or equal to the maximum VGPR number explicitly referenced within that 11702instruction then the symbol value is updated to equal that VGPR number plus 11703one. 11704 11705.kernel.sgpr_count 11706++++++++++++++++++ 11707 11708Set to zero each time a 11709:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11710encountered. At each instruction, if the current value of this symbol is less 11711than or equal to the maximum VGPR number explicitly referenced within that 11712instruction then the symbol value is updated to equal that SGPR number plus 11713one. 11714 11715.. _amdgpu-amdhsa-assembler-directives-v2: 11716 11717Code Object V2 Directives 11718~~~~~~~~~~~~~~~~~~~~~~~~~ 11719 11720.. warning:: 11721 Code object V2 is not the default code object version emitted by 11722 this version of LLVM. 11723 11724AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11725one can specify them with assembler directives. 11726 11727.hsa_code_object_version major, minor 11728+++++++++++++++++++++++++++++++++++++ 11729 11730*major* and *minor* are integers that specify the version of the HSA code 11731object that will be generated by the assembler. 11732 11733.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11734+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11735 11736 11737*major*, *minor*, and *stepping* are all integers that describe the instruction 11738set architecture (ISA) version of the assembly program. 11739 11740*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11741"AMD" and *arch* should always be equal to "AMDGPU". 11742 11743By default, the assembler will derive the ISA version, *vendor*, and *arch* 11744from the value of the -mcpu option that is passed to the assembler. 11745 11746.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11747 11748.amdgpu_hsa_kernel (name) 11749+++++++++++++++++++++++++ 11750 11751This directives specifies that the symbol with given name is a kernel entry 11752point (label) and the object should contain corresponding symbol of type 11753STT_AMDGPU_HSA_KERNEL. 11754 11755.amd_kernel_code_t 11756++++++++++++++++++ 11757 11758This directive marks the beginning of a list of key / value pairs that are used 11759to specify the amd_kernel_code_t object that will be emitted by the assembler. 11760The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11761amd_kernel_code_t values that are unspecified a default value will be used. The 11762default value for all keys is 0, with the following exceptions: 11763 11764- *amd_code_version_major* defaults to 1. 11765- *amd_kernel_code_version_minor* defaults to 2. 11766- *amd_machine_kind* defaults to 1. 11767- *amd_machine_version_major*, *machine_version_minor*, and 11768 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11769 that is passed to the assembler. 11770- *kernel_code_entry_byte_offset* defaults to 256. 11771- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11772 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11773 Note that wavefront size is specified as a power of two, so a value of **n** 11774 means a size of 2^ **n**. 11775- *call_convention* defaults to -1. 11776- *kernarg_segment_alignment*, *group_segment_alignment*, and 11777 *private_segment_alignment* default to 4. Note that alignments are specified 11778 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11779- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11780 GFX90A onwards. 11781- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11782 GFX10 onwards. 11783- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11784 11785The *.amd_kernel_code_t* directive must be placed immediately after the 11786function label and before any instructions. 11787 11788For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11789comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11790 11791.. _amdgpu-amdhsa-assembler-example-v2: 11792 11793Code Object V2 Example Source Code 11794~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11795 11796.. warning:: 11797 Code Object V2 is not the default code object version emitted by 11798 this version of LLVM. 11799 11800Here is an example of a minimal assembly source file, defining one HSA kernel: 11801 11802.. code:: 11803 :number-lines: 11804 11805 .hsa_code_object_version 1,0 11806 .hsa_code_object_isa 11807 11808 .hsatext 11809 .globl hello_world 11810 .p2align 8 11811 .amdgpu_hsa_kernel hello_world 11812 11813 hello_world: 11814 11815 .amd_kernel_code_t 11816 enable_sgpr_kernarg_segment_ptr = 1 11817 is_ptr64 = 1 11818 compute_pgm_rsrc1_vgprs = 0 11819 compute_pgm_rsrc1_sgprs = 0 11820 compute_pgm_rsrc2_user_sgpr = 2 11821 compute_pgm_rsrc1_wgp_mode = 0 11822 compute_pgm_rsrc1_mem_ordered = 0 11823 compute_pgm_rsrc1_fwd_progress = 1 11824 .end_amd_kernel_code_t 11825 11826 s_load_dwordx2 s[0:1], s[0:1] 0x0 11827 v_mov_b32 v0, 3.14159 11828 s_waitcnt lgkmcnt(0) 11829 v_mov_b32 v1, s0 11830 v_mov_b32 v2, s1 11831 flat_store_dword v[1:2], v0 11832 s_endpgm 11833 .Lfunc_end0: 11834 .size hello_world, .Lfunc_end0-hello_world 11835 11836.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11837 11838Code Object V3 to V4 Predefined Symbols 11839~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11840 11841The AMDGPU assembler defines and updates some symbols automatically. These 11842symbols do not affect code generation. 11843 11844.amdgcn.gfx_generation_number 11845+++++++++++++++++++++++++++++ 11846 11847Set to the GFX major generation number of the target being assembled for. For 11848example, when assembling for a "GFX9" target this will be set to the integer 11849value "9". The possible GFX major generation numbers are presented in 11850:ref:`amdgpu-processors`. 11851 11852.amdgcn.gfx_generation_minor 11853++++++++++++++++++++++++++++ 11854 11855Set to the GFX minor generation number of the target being assembled for. For 11856example, when assembling for a "GFX810" target this will be set to the integer 11857value "1". The possible GFX minor generation numbers are presented in 11858:ref:`amdgpu-processors`. 11859 11860.amdgcn.gfx_generation_stepping 11861+++++++++++++++++++++++++++++++ 11862 11863Set to the GFX stepping generation number of the target being assembled for. 11864For example, when assembling for a "GFX704" target this will be set to the 11865integer value "4". The possible GFX stepping generation numbers are presented 11866in :ref:`amdgpu-processors`. 11867 11868.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 11869 11870.amdgcn.next_free_vgpr 11871++++++++++++++++++++++ 11872 11873Set to zero before assembly begins. At each instruction, if the current value 11874of this symbol is less than or equal to the maximum VGPR number explicitly 11875referenced within that instruction then the symbol value is updated to equal 11876that VGPR number plus one. 11877 11878May be used to set the `.amdhsa_next_free_vgpr` directive in 11879:ref:`amdhsa-kernel-directives-table`. 11880 11881May be set at any time, e.g. manually set to zero at the start of each kernel. 11882 11883.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 11884 11885.amdgcn.next_free_sgpr 11886++++++++++++++++++++++ 11887 11888Set to zero before assembly begins. At each instruction, if the current value 11889of this symbol is less than or equal the maximum SGPR number explicitly 11890referenced within that instruction then the symbol value is updated to equal 11891that SGPR number plus one. 11892 11893May be used to set the `.amdhsa_next_free_spgr` directive in 11894:ref:`amdhsa-kernel-directives-table`. 11895 11896May be set at any time, e.g. manually set to zero at the start of each kernel. 11897 11898.. _amdgpu-amdhsa-assembler-directives-v3-v4: 11899 11900Code Object V3 to V4 Directives 11901~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11902 11903Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 11904architecture processors, and are not OS-specific. Directives which begin with 11905``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 11906``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 11907:ref:`amdgpu-processors`. 11908 11909.. _amdgpu-assembler-directive-amdgcn-target: 11910 11911.amdgcn_target <target-triple> "-" <target-id> 11912++++++++++++++++++++++++++++++++++++++++++++++ 11913 11914Optional directive which declares the ``<target-triple>-<target-id>`` supported 11915by the containing assembler source file. Used by the assembler to validate 11916command-line options such as ``-triple``, ``-mcpu``, and 11917``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 11918:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 11919 11920.. note:: 11921 11922 The target ID syntax used for code object V2 to V3 for this directive differs 11923 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 11924 11925.amdhsa_kernel <name> 11926+++++++++++++++++++++ 11927 11928Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 11929``<name>.kd``, in the current location of the current section. Only valid when 11930the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 11931instruction to execute, and does not need to be previously defined. 11932 11933Marks the beginning of a list of directives used to generate the bytes of a 11934kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 11935Directives which may appear in this list are described in 11936:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 11937be valid for the target being assembled for, and cannot be repeated. Directives 11938support the range of values specified by the field they reference in 11939:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 11940assumed to have its default value, unless it is marked as "Required", in which 11941case it is an error to omit the directive. This list of directives is 11942terminated by an ``.end_amdhsa_kernel`` directive. 11943 11944 .. table:: AMDHSA Kernel Assembler Directives 11945 :name: amdhsa-kernel-directives-table 11946 11947 ======================================================== =================== ============ =================== 11948 Directive Default Supported On Description 11949 ======================================================== =================== ============ =================== 11950 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 11951 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11952 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 11953 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11954 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 11955 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11956 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 11957 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11958 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 11959 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11960 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 11961 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11962 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 11963 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11964 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 11965 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11966 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 11967 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11968 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 11969 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11970 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 11971 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11972 Specific 11973 (wavefrontsize64) 11974 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 11975 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11976 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 11977 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11978 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 11979 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11980 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 11981 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11982 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 11983 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11984 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 11985 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11986 Possible values are defined in 11987 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 11988 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 11989 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 11990 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11991 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 11992 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11993 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11994 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 11995 Used to calculate ACCUM_OFFSET in 11996 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11997 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 11998 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11999 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12000 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 12001 scratch memory. Used to calculate 12002 GRANULATED_WAVEFRONT_SGPR_COUNT in 12003 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12004 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 12005 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12006 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12007 (xnack) 12008 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 12009 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12010 Possible values are defined in 12011 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12012 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 12013 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12014 Possible values are defined in 12015 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12016 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 12017 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12018 Possible values are defined in 12019 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12020 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 12021 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12022 Possible values are defined in 12023 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12024 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 12025 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12026 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 12027 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12028 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 12029 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12030 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 12031 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 12032 Specific 12033 (tgsplit) 12034 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 12035 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12036 Specific 12037 (cumode) 12038 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 12039 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12040 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 12041 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12042 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 12043 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12044 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 12045 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12046 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 12047 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12048 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 12049 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12050 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 12051 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12052 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 12053 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12054 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 12055 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12056 ======================================================== =================== ============ =================== 12057 12058.amdgpu_metadata 12059++++++++++++++++ 12060 12061Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 12062note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 12063 12064The contents must be in the [YAML]_ markup format, with the same structure and 12065semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 12066:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 12067 12068This directive is terminated by an ``.end_amdgpu_metadata`` directive. 12069 12070.. _amdgpu-amdhsa-assembler-example-v3-v4: 12071 12072Code Object V3 to V4 Example Source Code 12073~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12074 12075Here is an example of a minimal assembly source file, defining one HSA kernel: 12076 12077.. code:: 12078 :number-lines: 12079 12080 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12081 12082 .text 12083 .globl hello_world 12084 .p2align 8 12085 .type hello_world,@function 12086 hello_world: 12087 s_load_dwordx2 s[0:1], s[0:1] 0x0 12088 v_mov_b32 v0, 3.14159 12089 s_waitcnt lgkmcnt(0) 12090 v_mov_b32 v1, s0 12091 v_mov_b32 v2, s1 12092 flat_store_dword v[1:2], v0 12093 s_endpgm 12094 .Lfunc_end0: 12095 .size hello_world, .Lfunc_end0-hello_world 12096 12097 .rodata 12098 .p2align 6 12099 .amdhsa_kernel hello_world 12100 .amdhsa_user_sgpr_kernarg_segment_ptr 1 12101 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12102 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12103 .end_amdhsa_kernel 12104 12105 .amdgpu_metadata 12106 --- 12107 amdhsa.version: 12108 - 1 12109 - 0 12110 amdhsa.kernels: 12111 - .name: hello_world 12112 .symbol: hello_world.kd 12113 .kernarg_segment_size: 48 12114 .group_segment_fixed_size: 0 12115 .private_segment_fixed_size: 0 12116 .kernarg_segment_align: 4 12117 .wavefront_size: 64 12118 .sgpr_count: 2 12119 .vgpr_count: 3 12120 .max_flat_workgroup_size: 256 12121 ... 12122 .end_amdgpu_metadata 12123 12124If an assembly source file contains multiple kernels and/or functions, the 12125:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 12126:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 12127the ``.set <symbol>, <expression>`` directive. For example, in the case of two 12128kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 12129to group the function with the kernel that calls it and reset the symbols 12130between the two connected components: 12131 12132.. code:: 12133 :number-lines: 12134 12135 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12136 12137 // gpr tracking symbols are implicitly set to zero 12138 12139 .text 12140 .globl kern0 12141 .p2align 8 12142 .type kern0,@function 12143 kern0: 12144 // ... 12145 s_endpgm 12146 .Lkern0_end: 12147 .size kern0, .Lkern0_end-kern0 12148 12149 .rodata 12150 .p2align 6 12151 .amdhsa_kernel kern0 12152 // ... 12153 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12154 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12155 .end_amdhsa_kernel 12156 12157 // reset symbols to begin tracking usage in func1 and kern1 12158 .set .amdgcn.next_free_vgpr, 0 12159 .set .amdgcn.next_free_sgpr, 0 12160 12161 .text 12162 .hidden func1 12163 .global func1 12164 .p2align 2 12165 .type func1,@function 12166 func1: 12167 // ... 12168 s_setpc_b64 s[30:31] 12169 .Lfunc1_end: 12170 .size func1, .Lfunc1_end-func1 12171 12172 .globl kern1 12173 .p2align 8 12174 .type kern1,@function 12175 kern1: 12176 // ... 12177 s_getpc_b64 s[4:5] 12178 s_add_u32 s4, s4, func1@rel32@lo+4 12179 s_addc_u32 s5, s5, func1@rel32@lo+4 12180 s_swappc_b64 s[30:31], s[4:5] 12181 // ... 12182 s_endpgm 12183 .Lkern1_end: 12184 .size kern1, .Lkern1_end-kern1 12185 12186 .rodata 12187 .p2align 6 12188 .amdhsa_kernel kern1 12189 // ... 12190 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12191 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12192 .end_amdhsa_kernel 12193 12194These symbols cannot identify connected components in order to automatically 12195track the usage for each kernel. However, in some cases careful organization of 12196the kernels and functions in the source file means there is minimal additional 12197effort required to accurately calculate GPR usage. 12198 12199Additional Documentation 12200======================== 12201 12202.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 12203.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 12204.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 12205.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 12206.. [AMD-GCN-GFX10-RDNA1] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 12207.. [AMD-GCN-GFX10-RDNA2] `AMD "RDNA 2" Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 12208.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 12209.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 12210.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 12211.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 12212.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 12213.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12214.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12215.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12216.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12217.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12218.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12219.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12220.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12221.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12222.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12223.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12224