1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 42to specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as: 73 74 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 75 loader on Linux. See *AMD ROCm Platform Release Notes* 76 [AMD-ROCm-Release-Notes]_ for supported hardware and 77 software. 78 - AMD's PAL runtime using the *pal-amdhsa* loader on 79 Windows. 80 81 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 82 runtime using the *pal-amdpal* loader on Windows and Linux 83 Pro. 84 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 85 3D runtime using the *mesa-mesa3d* loader on Linux. 86 ============== ============================================================ 87 88 .. table:: AMDGPU Environments 89 :name: amdgpu-environment-table 90 91 ============ ============================================================== 92 Environment Description 93 ============ ============================================================== 94 *<empty>* Default. 95 ============ ============================================================== 96 97.. _amdgpu-processors: 98 99Processors 100---------- 101 102Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 103specify the AMDGPU processor together with optional target features. See 104:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 105specific information. 106 107Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 108 109* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 110 111 112 .. table:: AMDGPU Processors 113 :name: amdgpu-processor-table 114 115 =========== =============== ============ ===== ================= =============== =============== ====================== 116 Processor Alternative Target dGPU/ Target Target OS Support Example 117 Processor Triple APU Features Properties *(see* Products 118 Architecture Supported `amdgpu-os`_ 119 *and 120 corresponding 121 runtime release 122 notes for 123 current 124 information and 125 level of 126 support)* 127 =========== =============== ============ ===== ================= =============== =============== ====================== 128 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 129 ----------------------------------------------------------------------------------------------------------------------- 130 ``r600`` ``r600`` dGPU - Does not 131 support 132 generic 133 address 134 space 135 ``r630`` ``r600`` dGPU - Does not 136 support 137 generic 138 address 139 space 140 ``rs880`` ``r600`` dGPU - Does not 141 support 142 generic 143 address 144 space 145 ``rv670`` ``r600`` dGPU - Does not 146 support 147 generic 148 address 149 space 150 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 151 ----------------------------------------------------------------------------------------------------------------------- 152 ``rv710`` ``r600`` dGPU - Does not 153 support 154 generic 155 address 156 space 157 ``rv730`` ``r600`` dGPU - Does not 158 support 159 generic 160 address 161 space 162 ``rv770`` ``r600`` dGPU - Does not 163 support 164 generic 165 address 166 space 167 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 168 ----------------------------------------------------------------------------------------------------------------------- 169 ``cedar`` ``r600`` dGPU - Does not 170 support 171 generic 172 address 173 space 174 ``cypress`` ``r600`` dGPU - Does not 175 support 176 generic 177 address 178 space 179 ``juniper`` ``r600`` dGPU - Does not 180 support 181 generic 182 address 183 space 184 ``redwood`` ``r600`` dGPU - Does not 185 support 186 generic 187 address 188 space 189 ``sumo`` ``r600`` dGPU - Does not 190 support 191 generic 192 address 193 space 194 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 195 ----------------------------------------------------------------------------------------------------------------------- 196 ``barts`` ``r600`` dGPU - Does not 197 support 198 generic 199 address 200 space 201 ``caicos`` ``r600`` dGPU - Does not 202 support 203 generic 204 address 205 space 206 ``cayman`` ``r600`` dGPU - Does not 207 support 208 generic 209 address 210 space 211 ``turks`` ``r600`` dGPU - Does not 212 support 213 generic 214 address 215 space 216 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 217 ----------------------------------------------------------------------------------------------------------------------- 218 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 219 support 220 generic 221 address 222 space 223 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 224 - ``verde`` support 225 generic 226 address 227 space 228 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 229 - ``oland`` support 230 generic 231 address 232 space 233 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 234 ----------------------------------------------------------------------------------------------------------------------- 235 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 236 flat - *pal-amdhsa* - A6 Pro-7050B 237 scratch - *pal-amdpal* - A8-7100 238 - A8 Pro-7150B 239 - A10-7300 240 - A10 Pro-7350B 241 - FX-7500 242 - A8-7200P 243 - A10-7400P 244 - FX-7600P 245 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 246 flat - *pal-amdhsa* - FirePro W9100 247 scratch - *pal-amdpal* - FirePro S9150 248 - FirePro S9170 249 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 250 flat - *pal-amdhsa* - Radeon R9 290x 251 scratch - *pal-amdpal* - Radeon R390 252 - Radeon R390x 253 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 254 - ``mullins`` flat - *pal-amdpal* - E1-2200 255 scratch - E1-2500 256 - E2-3000 257 - E2-3800 258 - A4-5000 259 - A4-5100 260 - A6-5200 261 - A4 Pro-3340B 262 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 263 flat - *pal-amdpal* - Radeon HD 8770 264 scratch - R7 260 265 - R7 260X 266 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 267 flat - *pal-amdpal* 268 scratch .. TODO:: 269 270 Add product 271 names. 272 273 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 274 ----------------------------------------------------------------------------------------------------------------------- 275 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 276 flat - *pal-amdhsa* - Pro A6-8500B 277 scratch - *pal-amdpal* - A8-8600P 278 - Pro A8-8600B 279 - FX-8800P 280 - Pro A12-8800B 281 - A10-8700P 282 - Pro A10-8700B 283 - A10-8780P 284 - A10-9600P 285 - A10-9630P 286 - A12-9700P 287 - A12-9730P 288 - FX-9800P 289 - FX-9830P 290 - E2-9010 291 - A6-9210 292 - A9-9410 293 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 294 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 295 scratch - *pal-amdpal* - Radeon R9 385 296 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 297 - *pal-amdhsa* - Radeon R9 Fury 298 - *pal-amdpal* - Radeon R9 FuryX 299 - Radeon Pro Duo 300 - FirePro S9300x2 301 - Radeon Instinct MI8 302 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 303 flat - *pal-amdhsa* - Radeon RX 480 304 scratch - *pal-amdpal* - Radeon Instinct MI6 305 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 306 flat - *pal-amdhsa* 307 scratch - *pal-amdpal* 308 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 309 flat - *pal-amdhsa* - FirePro S7100 310 scratch - *pal-amdpal* - FirePro W7100 311 - Mobile FirePro 312 M7170 313 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 314 flat - *pal-amdhsa* 315 scratch - *pal-amdpal* .. TODO:: 316 317 Add product 318 names. 319 320 **GCN GFX9 (Vega)** [AMD-GCN-GFX9]_ 321 ----------------------------------------------------------------------------------------------------------------------- 322 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 323 flat - *pal-amdhsa* Frontier Edition 324 scratch - *pal-amdpal* - Radeon RX Vega 56 325 - Radeon RX Vega 64 326 - Radeon RX Vega 64 327 Liquid 328 - Radeon Instinct MI25 329 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 330 flat - *pal-amdhsa* - Ryzen 5 2400G 331 scratch - *pal-amdpal* 332 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 333 - *pal-amdhsa* 334 - *pal-amdpal* .. TODO:: 335 336 Add product 337 names. 338 339 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 340 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 341 scratch - *pal-amdpal* - Radeon VII 342 - Radeon Pro VII 343 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* *TBA* 344 - xnack - Absolute 345 flat .. TODO:: 346 scratch 347 Add product 348 names. 349 350 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 351 flat 352 scratch .. TODO:: 353 354 Add product 355 names. 356 357 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 358 - tgsplit flat 359 - xnack scratch .. TODO:: 360 - Packed 361 work-item Add product 362 IDs names. 363 364 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 365 flat - Ryzen 7 4700GE 366 scratch - Ryzen 5 4600G 367 - Ryzen 5 4600GE 368 - Ryzen 3 4300G 369 - Ryzen 3 4300GE 370 - Ryzen Pro 4000G 371 - Ryzen 7 Pro 4700G 372 - Ryzen 7 Pro 4750GE 373 - Ryzen 5 Pro 4650G 374 - Ryzen 5 Pro 4650GE 375 - Ryzen 3 Pro 4350G 376 - Ryzen 3 Pro 4350GE 377 378 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 379 ----------------------------------------------------------------------------------------------------------------------- 380 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 381 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 382 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 383 - Radeon Pro 5600M 384 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* *TBA* 385 - wavefrontsize64 - Absolute - *pal-amdhsa* 386 - xnack flat - *pal-amdpal* 387 scratch .. TODO:: 388 389 Add product 390 names. 391 392 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 393 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 394 - xnack scratch - *pal-amdpal* 395 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 396 ----------------------------------------------------------------------------------------------------------------------- 397 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 398 - wavefrontsize64 flat - *pal-amdhsa* 399 scratch - *pal-amdpal* .. TODO:: 400 401 Add product 402 names. 403 404 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 405 - wavefrontsize64 flat - *pal-amdhsa* 406 scratch - *pal-amdpal* .. TODO:: 407 408 Add product 409 names. 410 411 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 412 - wavefrontsize64 flat - *pal-amdhsa* 413 scratch - *pal-amdpal* .. TODO:: 414 415 Add product 416 names. 417 418 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 419 - wavefrontsize64 flat 420 scratch .. TODO:: 421 422 Add product 423 names. 424 425 =========== =============== ============ ===== ================= =============== =============== ====================== 426 427.. _amdgpu-target-features: 428 429Target Features 430--------------- 431 432Target features control how code is generated to support certain 433processor specific features. Not all target features are supported by 434all processors. The runtime must ensure that the features supported by 435the device used to execute the code match the features enabled when 436generating the code. A mismatch of features may result in incorrect 437execution, or a reduction in performance. 438 439The target features supported by each processor is listed in 440:ref:`amdgpu-processor-table`. 441 442Target features are controlled by exactly one of the following Clang 443options: 444 445``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 446 447 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 448 optional components of the target ID. If omitted, the target feature has the 449 ``any`` value. See :ref:`amdgpu-target-id`. 450 451``-m[no-]<target-feature>`` 452 453 Target features not specified by the target ID are specified using a 454 separate option. These target features can have an ``on`` or ``off`` 455 value. ``on`` is specified by omitting the ``no-`` prefix, and 456 ``off`` is specified by including the ``no-`` prefix. The default 457 if not specified is ``off``. 458 459For example: 460 461``-mcpu=gfx908:xnack+`` 462 Enable the ``xnack`` feature. 463``-mcpu=gfx908:xnack-`` 464 Disable the ``xnack`` feature. 465``-mcumode`` 466 Enable the ``cumode`` feature. 467``-mno-cumode`` 468 Disable the ``cumode`` feature. 469 470 .. table:: AMDGPU Target Features 471 :name: amdgpu-target-features-table 472 473 =============== ============================ ================================================== 474 Target Feature Clang Option to Control Description 475 Name 476 =============== ============================ ================================================== 477 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 478 when generating code for kernels. When disabled 479 native WGP wavefront execution mode is used, 480 when enabled CU wavefront execution mode is used 481 (see :ref:`amdgpu-amdhsa-memory-model`). 482 483 sramecc - ``-mcpu`` If specified, generate code that can only be 484 - ``--offload-arch`` loaded and executed in a process that has a 485 matching setting for SRAMECC. 486 487 If not specified for code object V2 to V3, generate 488 code that can be loaded and executed in a process 489 with SRAMECC enabled. 490 491 If not specified for code object V4, generate 492 code that can be loaded and executed in a process 493 with either setting of SRAMECC. 494 495 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 496 work-groups are launched in threadgroup split mode. 497 When enabled the waves of a work-group may be 498 launched in different CUs. 499 500 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 501 generating code for kernels. When disabled 502 native wavefront size 32 is used, when enabled 503 wavefront size 64 is used. 504 505 xnack - ``-mcpu`` If specified, generate code that can only be 506 - ``--offload-arch`` loaded and executed in a process that has a 507 matching setting for XNACK replay. 508 509 If not specified for code object V2 to V3, generate 510 code that can be loaded and executed in a process 511 with XNACK replay enabled. 512 513 If not specified for code object V4, generate 514 code that can be loaded and executed in a process 515 with either setting of XNACK replay. 516 517 XNACK replay can be used for demand paging and 518 page migration. If enabled in the device, then if 519 a page fault occurs the code may execute 520 incorrectly unless generated with XNACK replay 521 enabled, or generated for code object V4 without 522 specifying XNACK replay. Executing code that was 523 generated with XNACK replay enabled, or generated 524 for code object V4 without specifying XNACK replay, 525 on a device that does not have XNACK replay 526 enabled will execute correctly but may be less 527 performant than code generated for XNACK replay 528 disabled. 529 =============== ============================ ================================================== 530 531.. _amdgpu-target-id: 532 533Target ID 534--------- 535 536AMDGPU supports target IDs. See `Clang Offload Bundler 537<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 538description. The AMDGPU target specific information is: 539 540**processor** 541 Is an AMDGPU processor or alternative processor name specified in 542 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 543 the primary processor and alternative processor names. The canonical form 544 target ID only allow the primary processor name. 545 546**target-feature** 547 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 548 is supported by the processor. The target features supported by each processor 549 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 550 a target ID are marked as being controlled by ``-mcpu`` and 551 ``--offload-arch``. Each target feature must appear at most once in a target 552 ID. The non-canonical form target ID allows the target features to be 553 specified in any order. The canonical form target ID requires the target 554 features to be specified in alphabetic order. 555 556.. _amdgpu-target-id-v2-v3: 557 558Code Object V2 to V3 Target ID 559~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 560 561The target ID syntax for code object V2 to V3 is the same as defined in `Clang 562Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 563when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 564directive and the bundle entry ID. In those cases it has the following BNF 565syntax: 566 567.. code:: 568 569 <target-id> ::== <processor> ( "+" <target-feature> )* 570 571Where a target feature is omitted if *Off* and present if *On* or *Any*. 572 573.. note:: 574 575 The code object V2 to V3 cannot represent *Any* and treats it the same as 576 *On*. 577 578.. _amdgpu-embedding-bundled-objects: 579 580Embedding Bundled Code Objects 581------------------------------ 582 583AMDGPU supports the HIP and OpenMP languages that perform code object embedding 584as described in `Clang Offload Bundler 585<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 586 587.. note:: 588 589 The target ID syntax used for code object V2 to V3 for a bundle entry ID 590 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 591 592.. _amdgpu-address-spaces: 593 594Address Spaces 595-------------- 596 597The AMDGPU architecture supports a number of memory address spaces. The address 598space names use the OpenCL standard names, with some additions. 599 600The AMDGPU address spaces correspond to target architecture specific LLVM 601address space numbers used in LLVM IR. 602 603The AMDGPU address spaces are described in 604:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 605supported for the ``amdgcn`` target. 606 607 .. table:: AMDGPU Address Spaces 608 :name: amdgpu-address-spaces-table 609 610 ================================= =============== =========== ================ ======= ============================ 611 .. 64-Bit Process Address Space 612 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 613 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 614 Space Number Name Name Size 615 ================================= =============== =========== ================ ======= ============================ 616 Generic 0 flat flat 64 0x0000000000000000 617 Global 1 global global 64 0x0000000000000000 618 Region 2 N/A GDS 32 *not implemented for AMDHSA* 619 Local 3 group LDS 32 0xFFFFFFFF 620 Constant 4 constant *same as global* 64 0x0000000000000000 621 Private 5 private scratch 32 0xFFFFFFFF 622 Constant 32-bit 6 *TODO* 0x00000000 623 Buffer Fat Pointer (experimental) 7 *TODO* 624 ================================= =============== =========== ================ ======= ============================ 625 626**Generic** 627 The generic address space is supported unless the *Target Properties* column 628 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 629 space*. 630 631 The generic address space uses the hardware flat address support for two fixed 632 ranges of virtual addresses (the private and local apertures), that are 633 outside the range of addressable global memory, to map from a flat address to 634 a private or local address. This uses FLAT instructions that can take a flat 635 address and access global, private (scratch), and group (LDS) memory depending 636 on if the address is within one of the aperture ranges. 637 638 Flat access to scratch requires hardware aperture setup and setup in the 639 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 640 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 641 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 642 643 To convert between a private or group address space address (termed a segment 644 address) and a flat address the base address of the corresponding aperture 645 can be used. For GFX7-GFX8 these are available in the 646 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 647 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 648 GFX9-GFX10 the aperture base addresses are directly available as inline 649 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 650 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 651 aligned to 2^32 which makes it easier to convert from flat to segment or 652 segment to flat. 653 654 A global address space address has the same value when used as a flat address 655 so no conversion is needed. 656 657**Global and Constant** 658 The global and constant address spaces both use global virtual addresses, 659 which are the same virtual address space used by the CPU. However, some 660 virtual addresses may only be accessible to the CPU, some only accessible 661 by the GPU, and some by both. 662 663 Using the constant address space indicates that the data will not change 664 during the execution of the kernel. This allows scalar read instructions to 665 be used. As the constant address space could only be modified on the host 666 side, a generic pointer loaded from the constant address space is safe to be 667 assumed as a global pointer since only the device global memory is visible 668 and managed on the host side. The vector and scalar L1 caches are invalidated 669 of volatile data before each kernel dispatch execution to allow constant 670 memory to change values between kernel dispatches. 671 672**Region** 673 The region address space uses the hardware Global Data Store (GDS). All 674 wavefronts executing on the same device will access the same memory for any 675 given region address. However, the same region address accessed by wavefronts 676 executing on different devices will access different memory. It is higher 677 performance than global memory. It is allocated by the runtime. The data 678 store (DS) instructions can be used to access it. 679 680**Local** 681 The local address space uses the hardware Local Data Store (LDS) which is 682 automatically allocated when the hardware creates the wavefronts of a 683 work-group, and freed when all the wavefronts of a work-group have 684 terminated. All wavefronts belonging to the same work-group will access the 685 same memory for any given local address. However, the same local address 686 accessed by wavefronts belonging to different work-groups will access 687 different memory. It is higher performance than global memory. The data store 688 (DS) instructions can be used to access it. 689 690**Private** 691 The private address space uses the hardware scratch memory support which 692 automatically allocates memory when it creates a wavefront and frees it when 693 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 694 given private address will be different to the memory accessed by another lane 695 of the same or different wavefront for the same private address. 696 697 If a kernel dispatch uses scratch, then the hardware allocates memory from a 698 pool of backing memory allocated by the runtime for each wavefront. The lanes 699 of the wavefront access this using dword (4 byte) interleaving. The mapping 700 used from private address to backing memory address is: 701 702 ``wavefront-scratch-base + 703 ((private-address / 4) * wavefront-size * 4) + 704 (wavefront-lane-id * 4) + (private-address % 4)`` 705 706 If each lane of a wavefront accesses the same private address, the 707 interleaving results in adjacent dwords being accessed and hence requires 708 fewer cache lines to be fetched. 709 710 There are different ways that the wavefront scratch base address is 711 determined by a wavefront (see 712 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 713 714 Scratch memory can be accessed in an interleaved manner using buffer 715 instructions with the scratch buffer descriptor and per wavefront scratch 716 offset, by the scratch instructions, or by flat instructions. Multi-dword 717 access is not supported except by flat and scratch instructions in 718 GFX9-GFX10. 719 720**Constant 32-bit** 721 *TODO* 722 723**Buffer Fat Pointer** 724 The buffer fat pointer is an experimental address space that is currently 725 unsupported in the backend. It exposes a non-integral pointer that is in 726 the future intended to support the modelling of 128-bit buffer descriptors 727 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 728 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 729 model the buffer descriptors used heavily in graphics workloads targeting 730 the backend. 731 732.. _amdgpu-memory-scopes: 733 734Memory Scopes 735------------- 736 737This section provides LLVM memory synchronization scopes supported by the AMDGPU 738backend memory model when the target triple OS is ``amdhsa`` (see 739:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 740 741The memory model supported is based on the HSA memory model [HSA]_ which is 742based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 743relation is transitive over the synchronizes-with relation independent of scope 744and synchronizes-with allows the memory scope instances to be inclusive (see 745table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 746 747This is different to the OpenCL [OpenCL]_ memory model which does not have scope 748inclusion and requires the memory scopes to exactly match. However, this 749is conservatively correct for OpenCL. 750 751 .. table:: AMDHSA LLVM Sync Scopes 752 :name: amdgpu-amdhsa-llvm-sync-scopes-table 753 754 ======================= =================================================== 755 LLVM Sync Scope Description 756 ======================= =================================================== 757 *none* The default: ``system``. 758 759 Synchronizes with, and participates in modification 760 and seq_cst total orderings with, other operations 761 (except image operations) for all address spaces 762 (except private, or generic that accesses private) 763 provided the other operation's sync scope is: 764 765 - ``system``. 766 - ``agent`` and executed by a thread on the same 767 agent. 768 - ``workgroup`` and executed by a thread in the 769 same work-group. 770 - ``wavefront`` and executed by a thread in the 771 same wavefront. 772 773 ``agent`` Synchronizes with, and participates in modification 774 and seq_cst total orderings with, other operations 775 (except image operations) for all address spaces 776 (except private, or generic that accesses private) 777 provided the other operation's sync scope is: 778 779 - ``system`` or ``agent`` and executed by a thread 780 on the same agent. 781 - ``workgroup`` and executed by a thread in the 782 same work-group. 783 - ``wavefront`` and executed by a thread in the 784 same wavefront. 785 786 ``workgroup`` Synchronizes with, and participates in modification 787 and seq_cst total orderings with, other operations 788 (except image operations) for all address spaces 789 (except private, or generic that accesses private) 790 provided the other operation's sync scope is: 791 792 - ``system``, ``agent`` or ``workgroup`` and 793 executed by a thread in the same work-group. 794 - ``wavefront`` and executed by a thread in the 795 same wavefront. 796 797 ``wavefront`` Synchronizes with, and participates in modification 798 and seq_cst total orderings with, other operations 799 (except image operations) for all address spaces 800 (except private, or generic that accesses private) 801 provided the other operation's sync scope is: 802 803 - ``system``, ``agent``, ``workgroup`` or 804 ``wavefront`` and executed by a thread in the 805 same wavefront. 806 807 ``singlethread`` Only synchronizes with and participates in 808 modification and seq_cst total orderings with, 809 other operations (except image operations) running 810 in the same thread for all address spaces (for 811 example, in signal handlers). 812 813 ``one-as`` Same as ``system`` but only synchronizes with other 814 operations within the same address space. 815 816 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 817 operations within the same address space. 818 819 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 820 other operations within the same address space. 821 822 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 823 other operations within the same address space. 824 825 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 826 other operations within the same address space. 827 ======================= =================================================== 828 829LLVM IR Intrinsics 830------------------ 831 832The AMDGPU backend implements the following LLVM IR intrinsics. 833 834*This section is WIP.* 835 836.. TODO:: 837 838 List AMDGPU intrinsics. 839 840LLVM IR Attributes 841------------------ 842 843The AMDGPU backend supports the following LLVM IR attributes. 844 845 .. table:: AMDGPU LLVM IR Attributes 846 :name: amdgpu-llvm-ir-attributes-table 847 848 ======================================= ========================================================== 849 LLVM Attribute Description 850 ======================================= ========================================================== 851 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 852 will be specified when the kernel is dispatched. Generated 853 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 854 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 855 argument block size for the implicit arguments. This 856 varies by OS and language (for OpenCL see 857 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 858 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 859 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 860 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 861 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 862 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 863 execution unit. Generated by the ``amdgpu_waves_per_eu`` 864 CLANG attribute [CLANG-ATTR]_. 865 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 866 mode register to be set on entry. Overrides the default for 867 the calling convention. 868 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 869 the mode register to be set on entry. Overrides the default 870 for the calling convention. 871 ======================================= ========================================================== 872 873.. _amdgpu-elf-code-object: 874 875ELF Code Object 876=============== 877 878The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 879can be linked by ``lld`` to produce a standard ELF shared code object which can 880be loaded and executed on an AMDGPU target. 881 882.. _amdgpu-elf-header: 883 884Header 885------ 886 887The AMDGPU backend uses the following ELF header: 888 889 .. table:: AMDGPU ELF Header 890 :name: amdgpu-elf-header-table 891 892 ========================== =============================== 893 Field Value 894 ========================== =============================== 895 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 896 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 897 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 898 - ``ELFOSABI_AMDGPU_HSA`` 899 - ``ELFOSABI_AMDGPU_PAL`` 900 - ``ELFOSABI_AMDGPU_MESA3D`` 901 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 902 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 903 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 904 - ``ELFABIVERSION_AMDGPU_PAL`` 905 - ``ELFABIVERSION_AMDGPU_MESA3D`` 906 ``e_type`` - ``ET_REL`` 907 - ``ET_DYN`` 908 ``e_machine`` ``EM_AMDGPU`` 909 ``e_entry`` 0 910 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 911 :ref:`amdgpu-elf-header-e_flags-table-v3`, 912 and :ref:`amdgpu-elf-header-e_flags-table-v4` 913 ========================== =============================== 914 915.. 916 917 .. table:: AMDGPU ELF Header Enumeration Values 918 :name: amdgpu-elf-header-enumeration-values-table 919 920 =============================== ===== 921 Name Value 922 =============================== ===== 923 ``EM_AMDGPU`` 224 924 ``ELFOSABI_NONE`` 0 925 ``ELFOSABI_AMDGPU_HSA`` 64 926 ``ELFOSABI_AMDGPU_PAL`` 65 927 ``ELFOSABI_AMDGPU_MESA3D`` 66 928 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 929 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 930 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 931 ``ELFABIVERSION_AMDGPU_PAL`` 0 932 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 933 =============================== ===== 934 935``e_ident[EI_CLASS]`` 936 The ELF class is: 937 938 * ``ELFCLASS32`` for ``r600`` architecture. 939 940 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 941 process address space applications. 942 943``e_ident[EI_DATA]`` 944 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 945 946``e_ident[EI_OSABI]`` 947 One of the following AMDGPU target architecture specific OS ABIs 948 (see :ref:`amdgpu-os`): 949 950 * ``ELFOSABI_NONE`` for *unknown* OS. 951 952 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 953 954 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 955 956 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 957 958``e_ident[EI_ABIVERSION]`` 959 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 960 object conforms: 961 962 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 963 runtime ABI for code object V2. Specify using the Clang option 964 ``-mcode-object-version=2``. 965 966 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 967 runtime ABI for code object V3. Specify using the Clang option 968 ``-mcode-object-version=3``. This is the default code object 969 version if not specified. 970 971 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 972 runtime ABI for code object V4. Specify using the Clang option 973 ``-mcode-object-version=4``. 974 975 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 976 runtime ABI. 977 978 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 979 3D runtime ABI. 980 981``e_type`` 982 Can be one of the following values: 983 984 985 ``ET_REL`` 986 The type produced by the AMDGPU backend compiler as it is relocatable code 987 object. 988 989 ``ET_DYN`` 990 The type produced by the linker as it is a shared code object. 991 992 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 993 994``e_machine`` 995 The value ``EM_AMDGPU`` is used for the machine for all processors supported 996 by the ``r600`` and ``amdgcn`` architectures (see 997 :ref:`amdgpu-processor-table`). The specific processor is specified in the 998 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 999 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1000 ``e_flags`` for code object V3 to V4 (see 1001 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1002 :ref:`amdgpu-elf-header-e_flags-table-v4`). 1003 1004``e_entry`` 1005 The entry point is 0 as the entry points for individual kernels must be 1006 selected in order to invoke them through AQL packets. 1007 1008``e_flags`` 1009 The AMDGPU backend uses the following ELF header flags: 1010 1011 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1012 :name: amdgpu-elf-header-e_flags-v2-table 1013 1014 ===================================== ===== ============================= 1015 Name Value Description 1016 ===================================== ===== ============================= 1017 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1018 target feature is 1019 enabled for all code 1020 contained in the code object. 1021 If the processor 1022 does not support the 1023 ``xnack`` target 1024 feature then must 1025 be 0. 1026 See 1027 :ref:`amdgpu-target-features`. 1028 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1029 handler is enabled for all 1030 code contained in the code 1031 object. If the processor 1032 does not support a trap 1033 handler then must be 0. 1034 See 1035 :ref:`amdgpu-target-features`. 1036 ===================================== ===== ============================= 1037 1038 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1039 :name: amdgpu-elf-header-e_flags-table-v3 1040 1041 ================================= ===== ============================= 1042 Name Value Description 1043 ================================= ===== ============================= 1044 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1045 mask for 1046 ``EF_AMDGPU_MACH_xxx`` values 1047 defined in 1048 :ref:`amdgpu-ef-amdgpu-mach-table`. 1049 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1050 target feature is 1051 enabled for all code 1052 contained in the code object. 1053 If the processor 1054 does not support the 1055 ``xnack`` target 1056 feature then must 1057 be 0. 1058 See 1059 :ref:`amdgpu-target-features`. 1060 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1061 target feature is 1062 enabled for all code 1063 contained in the code object. 1064 If the processor 1065 does not support the 1066 ``sramecc`` target 1067 feature then must 1068 be 0. 1069 See 1070 :ref:`amdgpu-target-features`. 1071 ================================= ===== ============================= 1072 1073 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1074 :name: amdgpu-elf-header-e_flags-table-v4 1075 1076 ============================================ ===== =================================== 1077 Name Value Description 1078 ============================================ ===== =================================== 1079 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1080 mask for 1081 ``EF_AMDGPU_MACH_xxx`` values 1082 defined in 1083 :ref:`amdgpu-ef-amdgpu-mach-table`. 1084 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1085 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1086 values. 1087 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1088 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1089 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1090 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1091 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1092 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1093 values. 1094 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1095 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1096 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1097 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1098 ============================================ ===== =================================== 1099 1100 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1101 :name: amdgpu-ef-amdgpu-mach-table 1102 1103 ==================================== ========== ============================= 1104 Name Value Description (see 1105 :ref:`amdgpu-processor-table`) 1106 ==================================== ========== ============================= 1107 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1108 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1109 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1110 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1111 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1112 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1113 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1114 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1115 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1116 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1117 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1118 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1119 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1120 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1121 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1122 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1123 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1124 *reserved* 0x011 - Reserved for ``r600`` 1125 0x01f architecture processors. 1126 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1127 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1128 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1129 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1130 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1131 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1132 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1133 *reserved* 0x027 Reserved. 1134 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1135 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1136 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1137 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1138 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1139 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1140 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1141 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1142 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1143 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1144 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1145 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1146 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1147 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1148 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1149 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1150 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1151 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1152 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1153 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1154 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1155 *reserved* 0x03d Reserved. 1156 *reserved* 0x03e Reserved. 1157 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1158 *reserved* 0x040 Reserved. 1159 *reserved* 0x041 Reserved. 1160 ==================================== ========== ============================= 1161 1162Sections 1163-------- 1164 1165An AMDGPU target ELF code object has the standard ELF sections which include: 1166 1167 .. table:: AMDGPU ELF Sections 1168 :name: amdgpu-elf-sections-table 1169 1170 ================== ================ ================================= 1171 Name Type Attributes 1172 ================== ================ ================================= 1173 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1174 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1175 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1176 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1177 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1178 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1179 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1180 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1181 ``.note`` ``SHT_NOTE`` *none* 1182 ``.rela``\ *name* ``SHT_RELA`` *none* 1183 ``.rela.dyn`` ``SHT_RELA`` *none* 1184 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1185 ``.shstrtab`` ``SHT_STRTAB`` *none* 1186 ``.strtab`` ``SHT_STRTAB`` *none* 1187 ``.symtab`` ``SHT_SYMTAB`` *none* 1188 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1189 ================== ================ ================================= 1190 1191These sections have their standard meanings (see [ELF]_) and are only generated 1192if needed. 1193 1194``.debug``\ *\** 1195 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1196 information on the DWARF produced by the AMDGPU backend. 1197 1198``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1199 The standard sections used by a dynamic loader. 1200 1201``.note`` 1202 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1203 backend. 1204 1205``.rela``\ *name*, ``.rela.dyn`` 1206 For relocatable code objects, *name* is the name of the section that the 1207 relocation records apply. For example, ``.rela.text`` is the section name for 1208 relocation records associated with the ``.text`` section. 1209 1210 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1211 records from each of the relocatable code object's ``.rela``\ *name* sections. 1212 1213 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1214 the AMDGPU backend. 1215 1216``.text`` 1217 The executable machine code for the kernels and functions they call. Generated 1218 as position independent code. See :ref:`amdgpu-code-conventions` for 1219 information on conventions used in the isa generation. 1220 1221.. _amdgpu-note-records: 1222 1223Note Records 1224------------ 1225 1226The AMDGPU backend code object contains ELF note records in the ``.note`` 1227section. The set of generated notes and their semantics depend on the code 1228object version; see :ref:`amdgpu-note-records-v2` and 1229:ref:`amdgpu-note-records-v3-v4`. 1230 1231As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1232must be generated after the ``name`` field to ensure the ``desc`` field is 4 1233byte aligned. In addition, minimal zero-byte padding must be generated to 1234ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1235field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1236alignment. 1237 1238.. _amdgpu-note-records-v2: 1239 1240Code Object V2 Note Records 1241~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1242 1243.. warning:: 1244 Code object V2 is not the default code object version emitted by 1245 this version of LLVM. 1246 1247The AMDGPU backend code object uses the following ELF note record in the 1248``.note`` section when compiling for code object V2. 1249 1250The note record vendor field is "AMD". 1251 1252Additional note records may be present, but any which are not documented here 1253are deprecated and should not be used. 1254 1255 .. table:: AMDGPU Code Object V2 ELF Note Records 1256 :name: amdgpu-elf-note-records-v2-table 1257 1258 ===== ===================================== ====================================== 1259 Name Type Description 1260 ===== ===================================== ====================================== 1261 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1262 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1263 Finalizer and not the LLVM compiler. 1264 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1265 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1266 YAML [YAML]_ textual format. 1267 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1268 ===== ===================================== ====================================== 1269 1270.. 1271 1272 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1273 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1274 1275 ===================================== ===== 1276 Name Value 1277 ===================================== ===== 1278 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1279 ``NT_AMD_HSA_HSAIL`` 2 1280 ``NT_AMD_HSA_ISA_VERSION`` 3 1281 *reserved* 4-9 1282 ``NT_AMD_HSA_METADATA`` 10 1283 ``NT_AMD_HSA_ISA_NAME`` 11 1284 ===================================== ===== 1285 1286``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1287 Specifies the code object version number. The description field has the 1288 following layout: 1289 1290 .. code:: 1291 1292 struct amdgpu_hsa_note_code_object_version_s { 1293 uint32_t major_version; 1294 uint32_t minor_version; 1295 }; 1296 1297 The ``major_version`` has a value less than or equal to 2. 1298 1299``NT_AMD_HSA_HSAIL`` 1300 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1301 field has the following layout: 1302 1303 .. code:: 1304 1305 struct amdgpu_hsa_note_hsail_s { 1306 uint32_t hsail_major_version; 1307 uint32_t hsail_minor_version; 1308 uint8_t profile; 1309 uint8_t machine_model; 1310 uint8_t default_float_round; 1311 }; 1312 1313``NT_AMD_HSA_ISA_VERSION`` 1314 Specifies the target ISA version. The description field has the following layout: 1315 1316 .. code:: 1317 1318 struct amdgpu_hsa_note_isa_s { 1319 uint16_t vendor_name_size; 1320 uint16_t architecture_name_size; 1321 uint32_t major; 1322 uint32_t minor; 1323 uint32_t stepping; 1324 char vendor_and_architecture_name[1]; 1325 }; 1326 1327 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1328 vendor and architecture names respectively, including the NUL character. 1329 1330 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1331 vendor, immediately followed by the NUL terminated string for the 1332 architecture. 1333 1334 This note record is used by the HSA runtime loader. 1335 1336 Code object V2 only supports a limited number of processors and has fixed 1337 settings for target features. See 1338 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1339 processors and the corresponding target ID. In the table the note record ISA 1340 name is a concatenation of the vendor name, architecture name, major, minor, 1341 and stepping separated by a ":". 1342 1343 The target ID column shows the processor name and fixed target features used 1344 by the LLVM compiler. The LLVM compiler does not generate a 1345 ``NT_AMD_HSA_HSAIL`` note record. 1346 1347 A code object generated by the Finalizer also uses code object V2 and always 1348 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1349 ``sramecc`` target feature is as shown in 1350 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1351 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1352 bit. 1353 1354``NT_AMD_HSA_ISA_NAME`` 1355 Specifies the target ISA name as a non-NUL terminated string. 1356 1357 This note record is not used by the HSA runtime loader. 1358 1359 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1360 V2's limited support of processors and fixed settings for target features. 1361 1362 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1363 from the string to the corresponding target ID. If the ``xnack`` target 1364 feature is supported and enabled, the string produced by the LLVM compiler 1365 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1366 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1367 1368``NT_AMD_HSA_METADATA`` 1369 Specifies extensible metadata associated with the code objects executed on HSA 1370 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1371 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1372 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1373 metadata string. 1374 1375 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1376 :name: amdgpu-elf-note-record-supported_processors-v2-table 1377 1378 ==================== ========================== 1379 Note Record ISA Name Target ID 1380 ==================== ========================== 1381 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1382 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1383 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1384 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1385 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1386 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1387 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1388 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1389 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1390 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1391 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1392 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1393 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1394 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1395 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1396 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1397 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1398 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1399 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1400 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1401 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1402 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1403 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1404 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1405 ==================== ========================== 1406 1407.. _amdgpu-note-records-v3-v4: 1408 1409Code Object V3 to V4 Note Records 1410~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1411 1412The AMDGPU backend code object uses the following ELF note record in the 1413``.note`` section when compiling for code object V3 to V4. 1414 1415The note record vendor field is "AMDGPU". 1416 1417Additional note records may be present, but any which are not documented here 1418are deprecated and should not be used. 1419 1420 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1421 :name: amdgpu-elf-note-records-table-v3-v4 1422 1423 ======== ============================== ====================================== 1424 Name Type Description 1425 ======== ============================== ====================================== 1426 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1427 binary format. 1428 ======== ============================== ====================================== 1429 1430.. 1431 1432 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1433 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1434 1435 ============================== ===== 1436 Name Value 1437 ============================== ===== 1438 *reserved* 0-31 1439 ``NT_AMDGPU_METADATA`` 32 1440 ============================== ===== 1441 1442``NT_AMDGPU_METADATA`` 1443 Specifies extensible metadata associated with an AMDGPU code object. It is 1444 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1445 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1446 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1447 ``amdhsa`` OS. 1448 1449.. _amdgpu-symbols: 1450 1451Symbols 1452------- 1453 1454Symbols include the following: 1455 1456 .. table:: AMDGPU ELF Symbols 1457 :name: amdgpu-elf-symbols-table 1458 1459 ===================== ================== ================ ================== 1460 Name Type Section Description 1461 ===================== ================== ================ ================== 1462 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1463 - ``.rodata`` 1464 - ``.bss`` 1465 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1466 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1467 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1468 ===================== ================== ================ ================== 1469 1470Global variable 1471 Global variables both used and defined by the compilation unit. 1472 1473 If the symbol is defined in the compilation unit then it is allocated in the 1474 appropriate section according to if it has initialized data or is readonly. 1475 1476 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1477 will resolve relocations using the definition provided by another code object 1478 or explicitly defined by the runtime. 1479 1480 If the symbol resides in local/group memory (LDS) then its section is the 1481 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1482 ``st_value`` field describes alignment requirements as it does for common 1483 symbols. 1484 1485 .. TODO:: 1486 1487 Add description of linked shared object symbols. Seems undefined symbols 1488 are marked as STT_NOTYPE. 1489 1490Kernel descriptor 1491 Every HSA kernel has an associated kernel descriptor. It is the address of the 1492 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1493 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1494 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1495 1496Kernel entry point 1497 Every HSA kernel also has a symbol for its machine code entry point. 1498 1499.. _amdgpu-relocation-records: 1500 1501Relocation Records 1502------------------ 1503 1504AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1505relocatable fields are: 1506 1507``word32`` 1508 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1509 alignment. These values use the same byte order as other word values in the 1510 AMDGPU architecture. 1511 1512``word64`` 1513 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1514 alignment. These values use the same byte order as other word values in the 1515 AMDGPU architecture. 1516 1517Following notations are used for specifying relocation calculations: 1518 1519**A** 1520 Represents the addend used to compute the value of the relocatable field. 1521 1522**G** 1523 Represents the offset into the global offset table at which the relocation 1524 entry's symbol will reside during execution. 1525 1526**GOT** 1527 Represents the address of the global offset table. 1528 1529**P** 1530 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1531 of the storage unit being relocated (computed using ``r_offset``). 1532 1533**S** 1534 Represents the value of the symbol whose index resides in the relocation 1535 entry. Relocations not using this must specify a symbol index of 1536 ``STN_UNDEF``. 1537 1538**B** 1539 Represents the base address of a loaded executable or shared object which is 1540 the difference between the ELF address and the actual load address. 1541 Relocations using this are only valid in executable or shared objects. 1542 1543The following relocation types are supported: 1544 1545 .. table:: AMDGPU ELF Relocation Records 1546 :name: amdgpu-elf-relocation-records-table 1547 1548 ========================== ======= ===== ========== ============================== 1549 Relocation Type Kind Value Field Calculation 1550 ========================== ======= ===== ========== ============================== 1551 ``R_AMDGPU_NONE`` 0 *none* *none* 1552 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1553 Dynamic 1554 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1555 Dynamic 1556 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1557 Dynamic 1558 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1559 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1560 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1561 Dynamic 1562 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1563 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1564 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1565 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1566 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1567 *reserved* 12 1568 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1569 ========================== ======= ===== ========== ============================== 1570 1571``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1572the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1573 1574There is no current OS loader support for 32-bit programs and so 1575``R_AMDGPU_ABS32`` is not used. 1576 1577.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1578 1579Loaded Code Object Path Uniform Resource Identifier (URI) 1580--------------------------------------------------------- 1581 1582The AMD GPU code object loader represents the path of the ELF shared object from 1583which the code object was loaded as a textual Unifom Resource Identifier (URI). 1584Note that the code object is the in memory loaded relocated form of the ELF 1585shared object. Multiple code objects may be loaded at different memory 1586addresses in the same process from the same ELF shared object. 1587 1588The loaded code object path URI syntax is defined by the following BNF syntax: 1589 1590.. code:: 1591 1592 code_object_uri ::== file_uri | memory_uri 1593 file_uri ::== "file://" file_path [ range_specifier ] 1594 memory_uri ::== "memory://" process_id range_specifier 1595 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1596 file_path ::== URI_ENCODED_OS_FILE_PATH 1597 process_id ::== DECIMAL_NUMBER 1598 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1599 1600**number** 1601 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1602 and octal values by "0". 1603 1604**file_path** 1605 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1606 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1607 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1608 the path are separated by "/". 1609 1610**offset** 1611 Is a 0-based byte offset to the start of the code object. For a file URI, it 1612 is from the start of the file specified by the ``file_path``, and if omitted 1613 defaults to 0. For a memory URI, it is the memory address and is required. 1614 1615**size** 1616 Is the number of bytes in the code object. For a file URI, if omitted it 1617 defaults to the size of the file. It is required for a memory URI. 1618 1619**process_id** 1620 Is the identity of the process owning the memory. For Linux it is the C 1621 unsigned integral decimal literal for the process ID (PID). 1622 1623For example: 1624 1625.. code:: 1626 1627 file:///dir1/dir2/file1 1628 file:///dir3/dir4/file2#offset=0x2000&size=3000 1629 memory://1234#offset=0x20000&size=3000 1630 1631.. _amdgpu-dwarf-debug-information: 1632 1633DWARF Debug Information 1634======================= 1635 1636.. warning:: 1637 1638 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1639 is not currently fully implemented and is subject to change. 1640 1641AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1642:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1643object executable code and data to the source language constructs. It can be 1644used by tools such as debuggers and profilers. It uses features defined in 1645:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1646DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1647 1648This section defines the AMDGPU target architecture specific DWARF mappings. 1649 1650.. _amdgpu-dwarf-register-identifier: 1651 1652Register Identifier 1653------------------- 1654 1655This section defines the AMDGPU target architecture register numbers used in 1656DWARF operation expressions (see DWARF Version 5 section 2.5 and 1657:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1658instructions (see DWARF Version 5 section 6.4 and 1659:ref:`amdgpu-dwarf-call-frame-information`). 1660 1661A single code object can contain code for kernels that have different wavefront 1662sizes. The vector registers and some scalar registers are based on the wavefront 1663size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1664simplifies the consumer of the DWARF so that each register has a fixed size, 1665rather than being dynamic according to the wavefront size mode. Similarly, 1666distinct DWARF registers are defined for those registers that vary in size 1667according to the process address size. This allows a consumer to treat a 1668specific AMDGPU processor as a single architecture regardless of how it is 1669configured at run time. The compiler explicitly specifies the DWARF registers 1670that match the mode in which the code it is generating will be executed. 1671 1672DWARF registers are encoded as numbers, which are mapped to architecture 1673registers. The mapping for AMDGPU is defined in 1674:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1675mapping. 1676 1677.. table:: AMDGPU DWARF Register Mapping 1678 :name: amdgpu-dwarf-register-mapping-table 1679 1680 ============== ================= ======== ================================== 1681 DWARF Register AMDGPU Register Bit Size Description 1682 ============== ================= ======== ================================== 1683 0 PC_32 32 Program Counter (PC) when 1684 executing in a 32-bit process 1685 address space. Used in the CFI to 1686 describe the PC of the calling 1687 frame. 1688 1 EXEC_MASK_32 32 Execution Mask Register when 1689 executing in wavefront 32 mode. 1690 2-15 *Reserved* *Reserved for highly accessed 1691 registers using DWARF shortcut.* 1692 16 PC_64 64 Program Counter (PC) when 1693 executing in a 64-bit process 1694 address space. Used in the CFI to 1695 describe the PC of the calling 1696 frame. 1697 17 EXEC_MASK_64 64 Execution Mask Register when 1698 executing in wavefront 64 mode. 1699 18-31 *Reserved* *Reserved for highly accessed 1700 registers using DWARF shortcut.* 1701 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1702 Registers. 1703 96-127 *Reserved* *Reserved for frequently accessed 1704 registers using DWARF 1-byte ULEB.* 1705 128 STATUS 32 Status Register. 1706 129-511 *Reserved* *Reserved for future Scalar 1707 Architectural Registers.* 1708 512 VCC_32 32 Vector Condition Code Register 1709 when executing in wavefront 32 1710 mode. 1711 513-1023 *Reserved* *Reserved for future Vector 1712 Architectural Registers when 1713 executing in wavefront 32 mode.* 1714 768 VCC_64 64 Vector Condition Code Register 1715 when executing in wavefront 64 1716 mode. 1717 769-1023 *Reserved* *Reserved for future Vector 1718 Architectural Registers when 1719 executing in wavefront 64 mode.* 1720 1024-1087 *Reserved* *Reserved for padding.* 1721 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1722 1130-1535 *Reserved* *Reserved for future Scalar 1723 General Purpose Registers.* 1724 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1725 when executing in wavefront 32 1726 mode. 1727 1792-2047 *Reserved* *Reserved for future Vector 1728 General Purpose Registers when 1729 executing in wavefront 32 mode.* 1730 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1731 when executing in wavefront 32 1732 mode. 1733 2304-2559 *Reserved* *Reserved for future Vector 1734 Accumulation Registers when 1735 executing in wavefront 32 mode.* 1736 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1737 when executing in wavefront 64 1738 mode. 1739 2816-3071 *Reserved* *Reserved for future Vector 1740 General Purpose Registers when 1741 executing in wavefront 64 mode.* 1742 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1743 when executing in wavefront 64 1744 mode. 1745 3328-3583 *Reserved* *Reserved for future Vector 1746 Accumulation Registers when 1747 executing in wavefront 64 mode.* 1748 ============== ================= ======== ================================== 1749 1750The vector registers are represented as the full size for the wavefront. They 1751are organized as consecutive dwords (32-bits), one per lane, with the dword at 1752the least significant bit position corresponding to lane 0 and so forth. DWARF 1753location expressions involving the ``DW_OP_LLVM_offset`` and 1754``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1755register corresponding to the lane that is executing the current thread of 1756execution in languages that are implemented using a SIMD or SIMT execution 1757model. 1758 1759If the wavefront size is 32 lanes then the wavefront 32 mode register 1760definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1761mode register definitions are used. Some AMDGPU targets support executing in 1762both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1763to the wavefront mode of the generated code will be used. 1764 1765If code is generated to execute in a 32-bit process address space, then the 176632-bit process address space register definitions are used. If code is generated 1767to execute in a 64-bit process address space, then the 64-bit process address 1768space register definitions are used. The ``amdgcn`` target only supports the 176964-bit process address space. 1770 1771.. _amdgpu-dwarf-address-class-identifier: 1772 1773Address Class Identifier 1774------------------------ 1775 1776The DWARF address class represents the source language memory space. See DWARF 1777Version 5 section 2.12 which is updated by the *DWARF Extensions For 1778Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1779 1780The DWARF address class mapping used for AMDGPU is defined in 1781:ref:`amdgpu-dwarf-address-class-mapping-table`. 1782 1783.. table:: AMDGPU DWARF Address Class Mapping 1784 :name: amdgpu-dwarf-address-class-mapping-table 1785 1786 ========================= ====== ================= 1787 DWARF AMDGPU 1788 -------------------------------- ----------------- 1789 Address Class Name Value Address Space 1790 ========================= ====== ================= 1791 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1792 ``DW_ADDR_LLVM_global`` 0x0001 Global 1793 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1794 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1795 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1796 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1797 ========================= ====== ================= 1798 1799The DWARF address class values defined in the *DWARF Extensions For 1800Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1801 1802In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1803available for use for the AMD extension for access to the hardware GDS memory 1804which is scratchpad memory allocated per device. 1805 1806For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1807address class of ``DW_ADDR_none`` is used. 1808 1809See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1810mapping of DWARF address classes to DWARF address spaces, including address size 1811and NULL value. 1812 1813.. _amdgpu-dwarf-address-space-identifier: 1814 1815Address Space Identifier 1816------------------------ 1817 1818DWARF address spaces correspond to target architecture specific linear 1819addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1820For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1821 1822The DWARF address space mapping used for AMDGPU is defined in 1823:ref:`amdgpu-dwarf-address-space-mapping-table`. 1824 1825.. table:: AMDGPU DWARF Address Space Mapping 1826 :name: amdgpu-dwarf-address-space-mapping-table 1827 1828 ======================================= ===== ======= ======== ================= ======================= 1829 DWARF AMDGPU Notes 1830 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1831 Address Space Name Value Address Bit Size Address Space 1832 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1833 .. 64-bit 32-bit 1834 process process 1835 address address 1836 space space 1837 ======================================= ===== ======= ======== ================= ======================= 1838 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1839 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1840 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1841 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1842 *Reserved* 0x04 1843 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1844 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1845 ======================================= ===== ======= ======== ================= ======================= 1846 1847See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1848including address size and NULL value. 1849 1850The ``DW_ASPACE_none`` address space is the default target architecture address 1851space used in DWARF operations that do not specify an address space. It 1852therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1853related operations can refer to addresses in the program code. 1854 1855The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1856specify the flat address space. If the address corresponds to an address in the 1857local address space, then it corresponds to the wavefront that is executing the 1858focused thread of execution. If the address corresponds to an address in the 1859private address space, then it corresponds to the lane that is executing the 1860focused thread of execution for languages that are implemented using a SIMD or 1861SIMT execution model. 1862 1863.. note:: 1864 1865 CUDA-like languages such as HIP that do not have address spaces in the 1866 language type system, but do allow variables to be allocated in different 1867 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1868 address space in the DWARF expression operations as the default address space 1869 is the global address space. 1870 1871The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1872specify the local address space corresponding to the wavefront that is executing 1873the focused thread of execution. 1874 1875The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1876to specify the private address space corresponding to the lane that is executing 1877the focused thread of execution for languages that are implemented using a SIMD 1878or SIMT execution model. 1879 1880The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1881to specify the unswizzled private address space corresponding to the wavefront 1882that is executing the focused thread of execution. The wavefront view of private 1883memory is the per wavefront unswizzled backing memory layout defined in 1884:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1885location for the backing memory of the wavefront (namely the address is not 1886offset by ``wavefront-scratch-base``). The following formula can be used to 1887convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1888``DW_ASPACE_AMDGPU_private_wave`` address: 1889 1890:: 1891 1892 private-address-wavefront = 1893 ((private-address-lane / 4) * wavefront-size * 4) + 1894 (wavefront-lane-id * 4) + (private-address-lane % 4) 1895 1896If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1897of the dwords for each lane starting with lane 0 is required, then this 1898simplifies to: 1899 1900:: 1901 1902 private-address-wavefront = 1903 private-address-lane * wavefront-size 1904 1905A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1906complete spilled vector register back into a complete vector register in the 1907CFI. The frame pointer can be a private lane address which is dword aligned, 1908which can be shifted to multiply by the wavefront size, and then used to form a 1909private wavefront address that gives a location for a contiguous set of dwords, 1910one per lane, where the vector register dwords are spilled. The compiler knows 1911the wavefront size since it generates the code. Note that the type of the 1912address may have to be converted as the size of a 1913``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1914``DW_ASPACE_AMDGPU_private_wave`` address. 1915 1916.. _amdgpu-dwarf-lane-identifier: 1917 1918Lane identifier 1919--------------- 1920 1921DWARF lane identifies specify a target architecture lane position for hardware 1922that executes in a SIMD or SIMT manner, and on which a source language maps its 1923threads of execution onto those lanes. The DWARF lane identifier is pushed by 1924the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1925section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1926section :ref:`amdgpu-dwarf-operation-expressions`. 1927 1928For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1929wavefront. It is numbered from 0 to the wavefront size minus 1. 1930 1931Operation Expressions 1932--------------------- 1933 1934DWARF expressions are used to compute program values and the locations of 1935program objects. See DWARF Version 5 section 2.5 and 1936:ref:`amdgpu-dwarf-operation-expressions`. 1937 1938DWARF location descriptions describe how to access storage which includes memory 1939and registers. When accessing storage on AMDGPU, bytes are ordered with least 1940significant bytes first, and bits are ordered within bytes with least 1941significant bits first. 1942 1943For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1944unwinding vector registers that are spilled under the execution mask to memory: 1945the zero-single location description is the vector register, and the one-single 1946location description is the spilled memory location description. The 1947``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1948memory location description. 1949 1950In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1951``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1952controlled by the execution mask. An undefined location description together 1953with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1954to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1955 1956Debugger Information Entry Attributes 1957------------------------------------- 1958 1959This section describes how certain debugger information entry attributes are 1960used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1961by *DWARF Extensions For Heterogeneous Debugging* section 1962:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1963 1964.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1965 1966``DW_AT_LLVM_lane_pc`` 1967~~~~~~~~~~~~~~~~~~~~~~ 1968 1969For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1970location of the separate lanes of a SIMT thread. 1971 1972If the lane is an active lane then this will be the same as the current program 1973location. 1974 1975If the lane is inactive, but was active on entry to the subprogram, then this is 1976the program location in the subprogram at which execution of the lane is 1977conceptual positioned. 1978 1979If the lane was not active on entry to the subprogram, then this will be the 1980undefined location. A client debugger can check if the lane is part of a valid 1981work-group by checking that the lane is in the range of the associated 1982work-group within the grid, accounting for partial work-groups. If it is not, 1983then the debugger can omit any information for the lane. Otherwise, the debugger 1984may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1985calling subprogram until it finds a non-undefined location. Conceptually the 1986lane only has the call frames that it has a non-undefined 1987``DW_AT_LLVM_lane_pc``. 1988 1989The following example illustrates how the AMDGPU backend can generate a DWARF 1990location list expression for the nested ``IF/THEN/ELSE`` structures of the 1991following subprogram pseudo code for a target with 64 lanes per wavefront. 1992 1993.. code:: 1994 :number-lines: 1995 1996 SUBPROGRAM X 1997 BEGIN 1998 a; 1999 IF (c1) THEN 2000 b; 2001 IF (c2) THEN 2002 c; 2003 ELSE 2004 d; 2005 ENDIF 2006 e; 2007 ELSE 2008 f; 2009 ENDIF 2010 g; 2011 END 2012 2013The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2014execution mask (``EXEC``) to linearize the control flow. The condition is 2015evaluated to make a mask of the lanes for which the condition evaluates to true. 2016First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2017logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2018``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2019the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2020region the ``EXEC`` mask is restored to the value it had at the beginning of the 2021region. This is shown below. Other approaches are possible, but the basic 2022concept is the same. 2023 2024.. code:: 2025 :number-lines: 2026 2027 $lex_start: 2028 a; 2029 %1 = EXEC 2030 %2 = c1 2031 $lex_1_start: 2032 EXEC = %1 & %2 2033 $if_1_then: 2034 b; 2035 %3 = EXEC 2036 %4 = c2 2037 $lex_1_1_start: 2038 EXEC = %3 & %4 2039 $lex_1_1_then: 2040 c; 2041 EXEC = ~EXEC & %3 2042 $lex_1_1_else: 2043 d; 2044 EXEC = %3 2045 $lex_1_1_end: 2046 e; 2047 EXEC = ~EXEC & %1 2048 $lex_1_else: 2049 f; 2050 EXEC = %1 2051 $lex_1_end: 2052 g; 2053 $lex_end: 2054 2055To create the DWARF location list expression that defines the location 2056description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2057pseudo instruction can be used to annotate the linearized control flow. This can 2058be done by defining an artificial variable for the lane PC. The DWARF location 2059list expression created for it is used as the value of the 2060``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2061 2062A DWARF procedure is defined for each well nested structured control flow region 2063which provides the conceptual lane program location for a lane if it is not 2064active (namely it is divergent). The DWARF operation expression for each region 2065conceptually inherits the value of the immediately enclosing region and modifies 2066it according to the semantics of the region. 2067 2068For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2069the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2070region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2071region since the ``THEN`` region has completed. 2072 2073The lane PC artificial variable is assigned at each region transition. It uses 2074the immediately enclosing region's DWARF procedure to compute the program 2075location for each lane assuming they are divergent, and then modifies the result 2076by inserting the current program location for each lane that the ``EXEC`` mask 2077indicates is active. 2078 2079By having separate DWARF procedures for each region, they can be reused to 2080define the value for any nested region. This reduces the total size of the DWARF 2081operation expressions. 2082 2083The following provides an example using pseudo LLVM MIR. 2084 2085.. code:: 2086 :number-lines: 2087 2088 $lex_start: 2089 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2090 DW_AT_name = "__uint64"; 2091 DW_AT_byte_size = 8; 2092 DW_AT_encoding = DW_ATE_unsigned; 2093 ]; 2094 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2095 DW_AT_name = "__active_lane_pc"; 2096 DW_AT_location = [ 2097 DW_OP_regx PC; 2098 DW_OP_LLVM_extend 64, 64; 2099 DW_OP_regval_type EXEC, %uint_64; 2100 DW_OP_LLVM_select_bit_piece 64, 64; 2101 ]; 2102 ]; 2103 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2104 DW_AT_name = "__divergent_lane_pc"; 2105 DW_AT_location = [ 2106 DW_OP_LLVM_undefined; 2107 DW_OP_LLVM_extend 64, 64; 2108 ]; 2109 ]; 2110 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2111 DW_OP_call_ref %__divergent_lane_pc; 2112 DW_OP_call_ref %__active_lane_pc; 2113 ]; 2114 a; 2115 %1 = EXEC; 2116 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2117 %2 = c1; 2118 $lex_1_start: 2119 EXEC = %1 & %2; 2120 $lex_1_then: 2121 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2122 DW_AT_name = "__divergent_lane_pc_1_then"; 2123 DW_AT_location = DIExpression[ 2124 DW_OP_call_ref %__divergent_lane_pc; 2125 DW_OP_addrx &lex_1_start; 2126 DW_OP_stack_value; 2127 DW_OP_LLVM_extend 64, 64; 2128 DW_OP_call_ref %__lex_1_save_exec; 2129 DW_OP_deref_type 64, %__uint_64; 2130 DW_OP_LLVM_select_bit_piece 64, 64; 2131 ]; 2132 ]; 2133 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2134 DW_OP_call_ref %__divergent_lane_pc_1_then; 2135 DW_OP_call_ref %__active_lane_pc; 2136 ]; 2137 b; 2138 %3 = EXEC; 2139 DBG_VALUE %3, %__lex_1_1_save_exec; 2140 %4 = c2; 2141 $lex_1_1_start: 2142 EXEC = %3 & %4; 2143 $lex_1_1_then: 2144 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2145 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2146 DW_AT_location = DIExpression[ 2147 DW_OP_call_ref %__divergent_lane_pc_1_then; 2148 DW_OP_addrx &lex_1_1_start; 2149 DW_OP_stack_value; 2150 DW_OP_LLVM_extend 64, 64; 2151 DW_OP_call_ref %__lex_1_1_save_exec; 2152 DW_OP_deref_type 64, %__uint_64; 2153 DW_OP_LLVM_select_bit_piece 64, 64; 2154 ]; 2155 ]; 2156 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2157 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2158 DW_OP_call_ref %__active_lane_pc; 2159 ]; 2160 c; 2161 EXEC = ~EXEC & %3; 2162 $lex_1_1_else: 2163 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2164 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2165 DW_AT_location = DIExpression[ 2166 DW_OP_call_ref %__divergent_lane_pc_1_then; 2167 DW_OP_addrx &lex_1_1_end; 2168 DW_OP_stack_value; 2169 DW_OP_LLVM_extend 64, 64; 2170 DW_OP_call_ref %__lex_1_1_save_exec; 2171 DW_OP_deref_type 64, %__uint_64; 2172 DW_OP_LLVM_select_bit_piece 64, 64; 2173 ]; 2174 ]; 2175 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2176 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2177 DW_OP_call_ref %__active_lane_pc; 2178 ]; 2179 d; 2180 EXEC = %3; 2181 $lex_1_1_end: 2182 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2183 DW_OP_call_ref %__divergent_lane_pc; 2184 DW_OP_call_ref %__active_lane_pc; 2185 ]; 2186 e; 2187 EXEC = ~EXEC & %1; 2188 $lex_1_else: 2189 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2190 DW_AT_name = "__divergent_lane_pc_1_else"; 2191 DW_AT_location = DIExpression[ 2192 DW_OP_call_ref %__divergent_lane_pc; 2193 DW_OP_addrx &lex_1_end; 2194 DW_OP_stack_value; 2195 DW_OP_LLVM_extend 64, 64; 2196 DW_OP_call_ref %__lex_1_save_exec; 2197 DW_OP_deref_type 64, %__uint_64; 2198 DW_OP_LLVM_select_bit_piece 64, 64; 2199 ]; 2200 ]; 2201 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2202 DW_OP_call_ref %__divergent_lane_pc_1_else; 2203 DW_OP_call_ref %__active_lane_pc; 2204 ]; 2205 f; 2206 EXEC = %1; 2207 $lex_1_end: 2208 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2209 DW_OP_call_ref %__divergent_lane_pc; 2210 DW_OP_call_ref %__active_lane_pc; 2211 ]; 2212 g; 2213 $lex_end: 2214 2215The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2216that are active, with the current program location. 2217 2218Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2219the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2220instruction, location list entries will be created that describe where the 2221artificial variables are allocated at any given program location. The compiler 2222may allocate them to registers or spill them to memory. 2223 2224The DWARF procedures for each region use the values of the saved execution mask 2225artificial variables to only update the lanes that are active on entry to the 2226region. All other lanes retain the value of the enclosing region where they were 2227last active. If they were not active on entry to the subprogram, then will have 2228the undefined location description. 2229 2230Other structured control flow regions can be handled similarly. For example, 2231loops would set the divergent program location for the region at the end of the 2232loop. Any lanes active will be in the loop, and any lanes not active must have 2233exited the loop. 2234 2235An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2236``IF/THEN/ELSE`` regions. 2237 2238The DWARF procedures can use the active lane artificial variable described in 2239:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2240``EXEC`` mask in order to support whole or quad wavefront mode. 2241 2242.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2243 2244``DW_AT_LLVM_active_lane`` 2245~~~~~~~~~~~~~~~~~~~~~~~~~~ 2246 2247The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2248entry is used to specify the lanes that are conceptually active for a SIMT 2249thread. 2250 2251The execution mask may be modified to implement whole or quad wavefront mode 2252operations. For example, all lanes may need to temporarily be made active to 2253execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2254update it to enable the necessary lanes, perform the operations, and then 2255restore the ``EXEC`` mask from the saved value. While executing the whole 2256wavefront region, the conceptual execution mask is the saved value, not the 2257``EXEC`` value. 2258 2259This is handled by defining an artificial variable for the active lane mask. The 2260active lane mask artificial variable would be the actual ``EXEC`` mask for 2261normal regions, and the saved execution mask for regions where the mask is 2262temporarily updated. The location list expression created for this artificial 2263variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2264attribute. 2265 2266``DW_AT_LLVM_augmentation`` 2267~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2268 2269For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2270debugger information entry has the following value for the augmentation string: 2271 2272:: 2273 2274 [amdgpu:v0.0] 2275 2276The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2277extensions used in the DWARF of the compilation unit. The version number 2278conforms to [SEMVER]_. 2279 2280Call Frame Information 2281---------------------- 2282 2283DWARF Call Frame Information (CFI) describes how a consumer can virtually 2284*unwind* call frames in a running process or core dump. See DWARF Version 5 2285section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2286 2287For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2288 22891. ``augmentation`` string contains the following null-terminated UTF-8 string: 2290 2291 :: 2292 2293 [amd:v0.0] 2294 2295 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2296 extensions used in this CIE or to the FDEs that use it. The version number 2297 conforms to [SEMVER]_. 2298 22992. ``address_size`` for the ``Global`` address space is defined in 2300 :ref:`amdgpu-dwarf-address-space-identifier`. 2301 23023. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2303 23044. ``code_alignment_factor`` is 4 bytes. 2305 2306 .. TODO:: 2307 2308 Add to :ref:`amdgpu-processor-table` table. 2309 23105. ``data_alignment_factor`` is 4 bytes. 2311 2312 .. TODO:: 2313 2314 Add to :ref:`amdgpu-processor-table` table. 2315 23166. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2317 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2318 23197. ``initial_instructions`` Since a subprogram X with fewer registers can be 2320 called from subprogram Y that has more allocated, X will not change any of 2321 the extra registers as it cannot access them. Therefore, the default rule 2322 for all columns is ``same value``. 2323 2324For AMDGPU the register number follows the numbering defined in 2325:ref:`amdgpu-dwarf-register-identifier`. 2326 2327For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2328the return address to get the address of a byte within the call site 2329instructions. See DWARF Version 5 section 6.4.4. 2330 2331Accelerated Access 2332------------------ 2333 2334See DWARF Version 5 section 6.1. 2335 2336Lookup By Name Section Header 2337~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2338 2339See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2340 2341For AMDGPU the lookup by name section header table: 2342 2343``augmentation_string_size`` (uword) 2344 2345 Set to the length of the ``augmentation_string`` value which is always a 2346 multiple of 4. 2347 2348``augmentation_string`` (sequence of UTF-8 characters) 2349 2350 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2351 2352 :: 2353 2354 [amdgpu:v0.0] 2355 2356 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2357 extensions used in the DWARF of this index. The version number conforms to 2358 [SEMVER]_. 2359 2360 .. note:: 2361 2362 This is different to the DWARF Version 5 definition that requires the first 2363 4 characters to be the vendor ID. But this is consistent with the other 2364 augmentation strings and does allow multiple vendor contributions. However, 2365 backwards compatibility may be more desirable. 2366 2367Lookup By Address Section Header 2368~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2369 2370See DWARF Version 5 section 6.1.2. 2371 2372For AMDGPU the lookup by address section header table: 2373 2374``address_size`` (ubyte) 2375 2376 Match the address size for the ``Global`` address space defined in 2377 :ref:`amdgpu-dwarf-address-space-identifier`. 2378 2379``segment_selector_size`` (ubyte) 2380 2381 AMDGPU does not use a segment selector so this is 0. The entries in the 2382 ``.debug_aranges`` do not have a segment selector. 2383 2384Line Number Information 2385----------------------- 2386 2387See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2388 2389AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2390The instruction set must be obtained from the ELF file header ``e_flags`` field 2391in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2392<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2393 2394.. TODO:: 2395 2396 Should the ``isa`` state machine register be used to indicate if the code is 2397 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2398 2399For AMDGPU the line number program header fields have the following values (see 2400DWARF Version 5 section 6.2.4): 2401 2402``address_size`` (ubyte) 2403 Matches the address size for the ``Global`` address space defined in 2404 :ref:`amdgpu-dwarf-address-space-identifier`. 2405 2406``segment_selector_size`` (ubyte) 2407 AMDGPU does not use a segment selector so this is 0. 2408 2409``minimum_instruction_length`` (ubyte) 2410 For GFX9-GFX10 this is 4. 2411 2412``maximum_operations_per_instruction`` (ubyte) 2413 For GFX9-GFX10 this is 1. 2414 2415Source text for online-compiled programs (for example, those compiled by the 2416OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2417See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2418Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2419<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2420 2421The Clang option used to control source embedding in AMDGPU is defined in 2422:ref:`amdgpu-clang-debug-options-table`. 2423 2424 .. table:: AMDGPU Clang Debug Options 2425 :name: amdgpu-clang-debug-options-table 2426 2427 ==================== ================================================== 2428 Debug Flag Description 2429 ==================== ================================================== 2430 -g[no-]embed-source Enable/disable embedding source text in DWARF 2431 debug sections. Useful for environments where 2432 source cannot be written to disk, such as 2433 when performing online compilation. 2434 ==================== ================================================== 2435 2436For example: 2437 2438``-gembed-source`` 2439 Enable the embedded source. 2440 2441``-gno-embed-source`` 2442 Disable the embedded source. 2443 244432-Bit and 64-Bit DWARF Formats 2445------------------------------- 2446 2447See DWARF Version 5 section 7.4 and 2448:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2449 2450For AMDGPU: 2451 2452* For the ``amdgcn`` target architecture only the 64-bit process address space 2453 is supported. 2454 2455* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2456 the 32-bit DWARF format. 2457 2458Unit Headers 2459------------ 2460 2461For AMDGPU the following values apply for each of the unit headers described in 2462DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2463 2464``address_size`` (ubyte) 2465 Matches the address size for the ``Global`` address space defined in 2466 :ref:`amdgpu-dwarf-address-space-identifier`. 2467 2468.. _amdgpu-code-conventions: 2469 2470Code Conventions 2471================ 2472 2473This section provides code conventions used for each supported target triple OS 2474(see :ref:`amdgpu-target-triples`). 2475 2476AMDHSA 2477------ 2478 2479This section provides code conventions used when the target triple OS is 2480``amdhsa`` (see :ref:`amdgpu-target-triples`). 2481 2482.. _amdgpu-amdhsa-code-object-metadata: 2483 2484Code Object Metadata 2485~~~~~~~~~~~~~~~~~~~~ 2486 2487The code object metadata specifies extensible metadata associated with the code 2488objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2489encoding and semantics of this metadata depends on the code object version; see 2490:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2491:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2492:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2493 2494Code object metadata is specified in a note record (see 2495:ref:`amdgpu-note-records`) and is required when the target triple OS is 2496``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2497information necessary to support the HSA compatible runtime kernel queries. For 2498example, the segment sizes needed in a dispatch packet. In addition, a 2499high-level language runtime may require other information to be included. For 2500example, the AMD OpenCL runtime records kernel argument information. 2501 2502.. _amdgpu-amdhsa-code-object-metadata-v2: 2503 2504Code Object V2 Metadata 2505+++++++++++++++++++++++ 2506 2507.. warning:: 2508 Code object V2 is not the default code object version emitted by this version 2509 of LLVM. 2510 2511Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2512(see :ref:`amdgpu-note-records-v2`). 2513 2514The metadata is specified as a YAML formatted string (see [YAML]_ and 2515:doc:`YamlIO`). 2516 2517.. TODO:: 2518 2519 Is the string null terminated? It probably should not if YAML allows it to 2520 contain null characters, otherwise it should be. 2521 2522The metadata is represented as a single YAML document comprised of the mapping 2523defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2524referenced tables. 2525 2526For boolean values, the string values of ``false`` and ``true`` are used for 2527false and true respectively. 2528 2529Additional information can be added to the mappings. To avoid conflicts, any 2530non-AMD key names should be prefixed by "*vendor-name*.". 2531 2532 .. table:: AMDHSA Code Object V2 Metadata Map 2533 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2534 2535 ========== ============== ========= ======================================= 2536 String Key Value Type Required? Description 2537 ========== ============== ========= ======================================= 2538 "Version" sequence of Required - The first integer is the major 2539 2 integers version. Currently 1. 2540 - The second integer is the minor 2541 version. Currently 0. 2542 "Printf" sequence of Each string is encoded information 2543 strings about a printf function call. The 2544 encoded information is organized as 2545 fields separated by colon (':'): 2546 2547 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2548 2549 where: 2550 2551 ``ID`` 2552 A 32-bit integer as a unique id for 2553 each printf function call 2554 2555 ``N`` 2556 A 32-bit integer equal to the number 2557 of arguments of printf function call 2558 minus 1 2559 2560 ``S[i]`` (where i = 0, 1, ... , N-1) 2561 32-bit integers for the size in bytes 2562 of the i-th FormatString argument of 2563 the printf function call 2564 2565 FormatString 2566 The format string passed to the 2567 printf function call. 2568 "Kernels" sequence of Required Sequence of the mappings for each 2569 mapping kernel in the code object. See 2570 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2571 for the definition of the mapping. 2572 ========== ============== ========= ======================================= 2573 2574.. 2575 2576 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2577 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2578 2579 ================= ============== ========= ================================ 2580 String Key Value Type Required? Description 2581 ================= ============== ========= ================================ 2582 "Name" string Required Source name of the kernel. 2583 "SymbolName" string Required Name of the kernel 2584 descriptor ELF symbol. 2585 "Language" string Source language of the kernel. 2586 Values include: 2587 2588 - "OpenCL C" 2589 - "OpenCL C++" 2590 - "HCC" 2591 - "OpenMP" 2592 2593 "LanguageVersion" sequence of - The first integer is the major 2594 2 integers version. 2595 - The second integer is the 2596 minor version. 2597 "Attrs" mapping Mapping of kernel attributes. 2598 See 2599 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2600 for the mapping definition. 2601 "Args" sequence of Sequence of mappings of the 2602 mapping kernel arguments. See 2603 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2604 for the definition of the mapping. 2605 "CodeProps" mapping Mapping of properties related to 2606 the kernel code. See 2607 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2608 for the mapping definition. 2609 ================= ============== ========= ================================ 2610 2611.. 2612 2613 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2614 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2615 2616 =================== ============== ========= ============================== 2617 String Key Value Type Required? Description 2618 =================== ============== ========= ============================== 2619 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2620 3 integers must be >=1 and the dispatch 2621 work-group size X, Y, Z must 2622 correspond to the specified 2623 values. Defaults to 0, 0, 0. 2624 2625 Corresponds to the OpenCL 2626 ``reqd_work_group_size`` 2627 attribute. 2628 "WorkGroupSizeHint" sequence of The dispatch work-group size 2629 3 integers X, Y, Z is likely to be the 2630 specified values. 2631 2632 Corresponds to the OpenCL 2633 ``work_group_size_hint`` 2634 attribute. 2635 "VecTypeHint" string The name of a scalar or vector 2636 type. 2637 2638 Corresponds to the OpenCL 2639 ``vec_type_hint`` attribute. 2640 2641 "RuntimeHandle" string The external symbol name 2642 associated with a kernel. 2643 OpenCL runtime allocates a 2644 global buffer for the symbol 2645 and saves the kernel's address 2646 to it, which is used for 2647 device side enqueueing. Only 2648 available for device side 2649 enqueued kernels. 2650 =================== ============== ========= ============================== 2651 2652.. 2653 2654 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2655 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2656 2657 ================= ============== ========= ================================ 2658 String Key Value Type Required? Description 2659 ================= ============== ========= ================================ 2660 "Name" string Kernel argument name. 2661 "TypeName" string Kernel argument type name. 2662 "Size" integer Required Kernel argument size in bytes. 2663 "Align" integer Required Kernel argument alignment in 2664 bytes. Must be a power of two. 2665 "ValueKind" string Required Kernel argument kind that 2666 specifies how to set up the 2667 corresponding argument. 2668 Values include: 2669 2670 "ByValue" 2671 The argument is copied 2672 directly into the kernarg. 2673 2674 "GlobalBuffer" 2675 A global address space pointer 2676 to the buffer data is passed 2677 in the kernarg. 2678 2679 "DynamicSharedPointer" 2680 A group address space pointer 2681 to dynamically allocated LDS 2682 is passed in the kernarg. 2683 2684 "Sampler" 2685 A global address space 2686 pointer to a S# is passed in 2687 the kernarg. 2688 2689 "Image" 2690 A global address space 2691 pointer to a T# is passed in 2692 the kernarg. 2693 2694 "Pipe" 2695 A global address space pointer 2696 to an OpenCL pipe is passed in 2697 the kernarg. 2698 2699 "Queue" 2700 A global address space pointer 2701 to an OpenCL device enqueue 2702 queue is passed in the 2703 kernarg. 2704 2705 "HiddenGlobalOffsetX" 2706 The OpenCL grid dispatch 2707 global offset for the X 2708 dimension is passed in the 2709 kernarg. 2710 2711 "HiddenGlobalOffsetY" 2712 The OpenCL grid dispatch 2713 global offset for the Y 2714 dimension is passed in the 2715 kernarg. 2716 2717 "HiddenGlobalOffsetZ" 2718 The OpenCL grid dispatch 2719 global offset for the Z 2720 dimension is passed in the 2721 kernarg. 2722 2723 "HiddenNone" 2724 An argument that is not used 2725 by the kernel. Space needs to 2726 be left for it, but it does 2727 not need to be set up. 2728 2729 "HiddenPrintfBuffer" 2730 A global address space pointer 2731 to the runtime printf buffer 2732 is passed in kernarg. 2733 2734 "HiddenHostcallBuffer" 2735 A global address space pointer 2736 to the runtime hostcall buffer 2737 is passed in kernarg. 2738 2739 "HiddenDefaultQueue" 2740 A global address space pointer 2741 to the OpenCL device enqueue 2742 queue that should be used by 2743 the kernel by default is 2744 passed in the kernarg. 2745 2746 "HiddenCompletionAction" 2747 A global address space pointer 2748 to help link enqueued kernels into 2749 the ancestor tree for determining 2750 when the parent kernel has finished. 2751 2752 "HiddenMultiGridSyncArg" 2753 A global address space pointer for 2754 multi-grid synchronization is 2755 passed in the kernarg. 2756 2757 "ValueType" string Unused and deprecated. This should no longer 2758 be emitted, but is accepted for compatibility. 2759 2760 2761 "PointeeAlign" integer Alignment in bytes of pointee 2762 type for pointer type kernel 2763 argument. Must be a power 2764 of 2. Only present if 2765 "ValueKind" is 2766 "DynamicSharedPointer". 2767 "AddrSpaceQual" string Kernel argument address space 2768 qualifier. Only present if 2769 "ValueKind" is "GlobalBuffer" or 2770 "DynamicSharedPointer". Values 2771 are: 2772 2773 - "Private" 2774 - "Global" 2775 - "Constant" 2776 - "Local" 2777 - "Generic" 2778 - "Region" 2779 2780 .. TODO:: 2781 2782 Is GlobalBuffer only Global 2783 or Constant? Is 2784 DynamicSharedPointer always 2785 Local? Can HCC allow Generic? 2786 How can Private or Region 2787 ever happen? 2788 2789 "AccQual" string Kernel argument access 2790 qualifier. Only present if 2791 "ValueKind" is "Image" or 2792 "Pipe". Values 2793 are: 2794 2795 - "ReadOnly" 2796 - "WriteOnly" 2797 - "ReadWrite" 2798 2799 .. TODO:: 2800 2801 Does this apply to 2802 GlobalBuffer? 2803 2804 "ActualAccQual" string The actual memory accesses 2805 performed by the kernel on the 2806 kernel argument. Only present if 2807 "ValueKind" is "GlobalBuffer", 2808 "Image", or "Pipe". This may be 2809 more restrictive than indicated 2810 by "AccQual" to reflect what the 2811 kernel actual does. If not 2812 present then the runtime must 2813 assume what is implied by 2814 "AccQual" and "IsConst". Values 2815 are: 2816 2817 - "ReadOnly" 2818 - "WriteOnly" 2819 - "ReadWrite" 2820 2821 "IsConst" boolean Indicates if the kernel argument 2822 is const qualified. Only present 2823 if "ValueKind" is 2824 "GlobalBuffer". 2825 2826 "IsRestrict" boolean Indicates if the kernel argument 2827 is restrict qualified. Only 2828 present if "ValueKind" is 2829 "GlobalBuffer". 2830 2831 "IsVolatile" boolean Indicates if the kernel argument 2832 is volatile qualified. Only 2833 present if "ValueKind" is 2834 "GlobalBuffer". 2835 2836 "IsPipe" boolean Indicates if the kernel argument 2837 is pipe qualified. Only present 2838 if "ValueKind" is "Pipe". 2839 2840 .. TODO:: 2841 2842 Can GlobalBuffer be pipe 2843 qualified? 2844 2845 ================= ============== ========= ================================ 2846 2847.. 2848 2849 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2850 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2851 2852 ============================ ============== ========= ===================== 2853 String Key Value Type Required? Description 2854 ============================ ============== ========= ===================== 2855 "KernargSegmentSize" integer Required The size in bytes of 2856 the kernarg segment 2857 that holds the values 2858 of the arguments to 2859 the kernel. 2860 "GroupSegmentFixedSize" integer Required The amount of group 2861 segment memory 2862 required by a 2863 work-group in 2864 bytes. This does not 2865 include any 2866 dynamically allocated 2867 group segment memory 2868 that may be added 2869 when the kernel is 2870 dispatched. 2871 "PrivateSegmentFixedSize" integer Required The amount of fixed 2872 private address space 2873 memory required for a 2874 work-item in 2875 bytes. If the kernel 2876 uses a dynamic call 2877 stack then additional 2878 space must be added 2879 to this value for the 2880 call stack. 2881 "KernargSegmentAlign" integer Required The maximum byte 2882 alignment of 2883 arguments in the 2884 kernarg segment. Must 2885 be a power of 2. 2886 "WavefrontSize" integer Required Wavefront size. Must 2887 be a power of 2. 2888 "NumSGPRs" integer Required Number of scalar 2889 registers used by a 2890 wavefront for 2891 GFX6-GFX10. This 2892 includes the special 2893 SGPRs for VCC, Flat 2894 Scratch (GFX7-GFX10) 2895 and XNACK (for 2896 GFX8-GFX10). It does 2897 not include the 16 2898 SGPR added if a trap 2899 handler is 2900 enabled. It is not 2901 rounded up to the 2902 allocation 2903 granularity. 2904 "NumVGPRs" integer Required Number of vector 2905 registers used by 2906 each work-item for 2907 GFX6-GFX10 2908 "MaxFlatWorkGroupSize" integer Required Maximum flat 2909 work-group size 2910 supported by the 2911 kernel in work-items. 2912 Must be >=1 and 2913 consistent with 2914 ReqdWorkGroupSize if 2915 not 0, 0, 0. 2916 "NumSpilledSGPRs" integer Number of stores from 2917 a scalar register to 2918 a register allocator 2919 created spill 2920 location. 2921 "NumSpilledVGPRs" integer Number of stores from 2922 a vector register to 2923 a register allocator 2924 created spill 2925 location. 2926 ============================ ============== ========= ===================== 2927 2928.. _amdgpu-amdhsa-code-object-metadata-v3: 2929 2930Code Object V3 Metadata 2931+++++++++++++++++++++++ 2932 2933Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2934record (see :ref:`amdgpu-note-records-v3-v4`). 2935 2936The metadata is represented as Message Pack formatted binary data (see 2937[MsgPack]_). The top level is a Message Pack map that includes the 2938keys defined in table 2939:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2940tables. 2941 2942Additional information can be added to the maps. To avoid conflicts, 2943any key names should be prefixed by "*vendor-name*." where 2944``vendor-name`` can be the name of the vendor and specific vendor 2945tool that generates the information. The prefix is abbreviated to 2946simply "." when it appears within a map that has been added by the 2947same *vendor-name*. 2948 2949 .. table:: AMDHSA Code Object V3 Metadata Map 2950 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2951 2952 ================= ============== ========= ======================================= 2953 String Key Value Type Required? Description 2954 ================= ============== ========= ======================================= 2955 "amdhsa.version" sequence of Required - The first integer is the major 2956 2 integers version. Currently 1. 2957 - The second integer is the minor 2958 version. Currently 0. 2959 "amdhsa.printf" sequence of Each string is encoded information 2960 strings about a printf function call. The 2961 encoded information is organized as 2962 fields separated by colon (':'): 2963 2964 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2965 2966 where: 2967 2968 ``ID`` 2969 A 32-bit integer as a unique id for 2970 each printf function call 2971 2972 ``N`` 2973 A 32-bit integer equal to the number 2974 of arguments of printf function call 2975 minus 1 2976 2977 ``S[i]`` (where i = 0, 1, ... , N-1) 2978 32-bit integers for the size in bytes 2979 of the i-th FormatString argument of 2980 the printf function call 2981 2982 FormatString 2983 The format string passed to the 2984 printf function call. 2985 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2986 map kernel in the code object. See 2987 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2988 for the definition of the keys included 2989 in that map. 2990 ================= ============== ========= ======================================= 2991 2992.. 2993 2994 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2995 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2996 2997 =================================== ============== ========= ================================ 2998 String Key Value Type Required? Description 2999 =================================== ============== ========= ================================ 3000 ".name" string Required Source name of the kernel. 3001 ".symbol" string Required Name of the kernel 3002 descriptor ELF symbol. 3003 ".language" string Source language of the kernel. 3004 Values include: 3005 3006 - "OpenCL C" 3007 - "OpenCL C++" 3008 - "HCC" 3009 - "HIP" 3010 - "OpenMP" 3011 - "Assembler" 3012 3013 ".language_version" sequence of - The first integer is the major 3014 2 integers version. 3015 - The second integer is the 3016 minor version. 3017 ".args" sequence of Sequence of maps of the 3018 map kernel arguments. See 3019 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3020 for the definition of the keys 3021 included in that map. 3022 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3023 3 integers must be >=1 and the dispatch 3024 work-group size X, Y, Z must 3025 correspond to the specified 3026 values. Defaults to 0, 0, 0. 3027 3028 Corresponds to the OpenCL 3029 ``reqd_work_group_size`` 3030 attribute. 3031 ".workgroup_size_hint" sequence of The dispatch work-group size 3032 3 integers X, Y, Z is likely to be the 3033 specified values. 3034 3035 Corresponds to the OpenCL 3036 ``work_group_size_hint`` 3037 attribute. 3038 ".vec_type_hint" string The name of a scalar or vector 3039 type. 3040 3041 Corresponds to the OpenCL 3042 ``vec_type_hint`` attribute. 3043 3044 ".device_enqueue_symbol" string The external symbol name 3045 associated with a kernel. 3046 OpenCL runtime allocates a 3047 global buffer for the symbol 3048 and saves the kernel's address 3049 to it, which is used for 3050 device side enqueueing. Only 3051 available for device side 3052 enqueued kernels. 3053 ".kernarg_segment_size" integer Required The size in bytes of 3054 the kernarg segment 3055 that holds the values 3056 of the arguments to 3057 the kernel. 3058 ".group_segment_fixed_size" integer Required The amount of group 3059 segment memory 3060 required by a 3061 work-group in 3062 bytes. This does not 3063 include any 3064 dynamically allocated 3065 group segment memory 3066 that may be added 3067 when the kernel is 3068 dispatched. 3069 ".private_segment_fixed_size" integer Required The amount of fixed 3070 private address space 3071 memory required for a 3072 work-item in 3073 bytes. If the kernel 3074 uses a dynamic call 3075 stack then additional 3076 space must be added 3077 to this value for the 3078 call stack. 3079 ".kernarg_segment_align" integer Required The maximum byte 3080 alignment of 3081 arguments in the 3082 kernarg segment. Must 3083 be a power of 2. 3084 ".wavefront_size" integer Required Wavefront size. Must 3085 be a power of 2. 3086 ".sgpr_count" integer Required Number of scalar 3087 registers required by a 3088 wavefront for 3089 GFX6-GFX9. A register 3090 is required if it is 3091 used explicitly, or 3092 if a higher numbered 3093 register is used 3094 explicitly. This 3095 includes the special 3096 SGPRs for VCC, Flat 3097 Scratch (GFX7-GFX9) 3098 and XNACK (for 3099 GFX8-GFX9). It does 3100 not include the 16 3101 SGPR added if a trap 3102 handler is 3103 enabled. It is not 3104 rounded up to the 3105 allocation 3106 granularity. 3107 ".vgpr_count" integer Required Number of vector 3108 registers required by 3109 each work-item for 3110 GFX6-GFX9. A register 3111 is required if it is 3112 used explicitly, or 3113 if a higher numbered 3114 register is used 3115 explicitly. 3116 ".max_flat_workgroup_size" integer Required Maximum flat 3117 work-group size 3118 supported by the 3119 kernel in work-items. 3120 Must be >=1 and 3121 consistent with 3122 ReqdWorkGroupSize if 3123 not 0, 0, 0. 3124 ".sgpr_spill_count" integer Number of stores from 3125 a scalar register to 3126 a register allocator 3127 created spill 3128 location. 3129 ".vgpr_spill_count" integer Number of stores from 3130 a vector register to 3131 a register allocator 3132 created spill 3133 location. 3134 =================================== ============== ========= ================================ 3135 3136.. 3137 3138 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3139 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3140 3141 ====================== ============== ========= ================================ 3142 String Key Value Type Required? Description 3143 ====================== ============== ========= ================================ 3144 ".name" string Kernel argument name. 3145 ".type_name" string Kernel argument type name. 3146 ".size" integer Required Kernel argument size in bytes. 3147 ".offset" integer Required Kernel argument offset in 3148 bytes. The offset must be a 3149 multiple of the alignment 3150 required by the argument. 3151 ".value_kind" string Required Kernel argument kind that 3152 specifies how to set up the 3153 corresponding argument. 3154 Values include: 3155 3156 "by_value" 3157 The argument is copied 3158 directly into the kernarg. 3159 3160 "global_buffer" 3161 A global address space pointer 3162 to the buffer data is passed 3163 in the kernarg. 3164 3165 "dynamic_shared_pointer" 3166 A group address space pointer 3167 to dynamically allocated LDS 3168 is passed in the kernarg. 3169 3170 "sampler" 3171 A global address space 3172 pointer to a S# is passed in 3173 the kernarg. 3174 3175 "image" 3176 A global address space 3177 pointer to a T# is passed in 3178 the kernarg. 3179 3180 "pipe" 3181 A global address space pointer 3182 to an OpenCL pipe is passed in 3183 the kernarg. 3184 3185 "queue" 3186 A global address space pointer 3187 to an OpenCL device enqueue 3188 queue is passed in the 3189 kernarg. 3190 3191 "hidden_global_offset_x" 3192 The OpenCL grid dispatch 3193 global offset for the X 3194 dimension is passed in the 3195 kernarg. 3196 3197 "hidden_global_offset_y" 3198 The OpenCL grid dispatch 3199 global offset for the Y 3200 dimension is passed in the 3201 kernarg. 3202 3203 "hidden_global_offset_z" 3204 The OpenCL grid dispatch 3205 global offset for the Z 3206 dimension is passed in the 3207 kernarg. 3208 3209 "hidden_none" 3210 An argument that is not used 3211 by the kernel. Space needs to 3212 be left for it, but it does 3213 not need to be set up. 3214 3215 "hidden_printf_buffer" 3216 A global address space pointer 3217 to the runtime printf buffer 3218 is passed in kernarg. 3219 3220 "hidden_hostcall_buffer" 3221 A global address space pointer 3222 to the runtime hostcall buffer 3223 is passed in kernarg. 3224 3225 "hidden_default_queue" 3226 A global address space pointer 3227 to the OpenCL device enqueue 3228 queue that should be used by 3229 the kernel by default is 3230 passed in the kernarg. 3231 3232 "hidden_completion_action" 3233 A global address space pointer 3234 to help link enqueued kernels into 3235 the ancestor tree for determining 3236 when the parent kernel has finished. 3237 3238 "hidden_multigrid_sync_arg" 3239 A global address space pointer for 3240 multi-grid synchronization is 3241 passed in the kernarg. 3242 3243 ".value_type" string Unused and deprecated. This should no longer 3244 be emitted, but is accepted for compatibility. 3245 3246 ".pointee_align" integer Alignment in bytes of pointee 3247 type for pointer type kernel 3248 argument. Must be a power 3249 of 2. Only present if 3250 ".value_kind" is 3251 "dynamic_shared_pointer". 3252 ".address_space" string Kernel argument address space 3253 qualifier. Only present if 3254 ".value_kind" is "global_buffer" or 3255 "dynamic_shared_pointer". Values 3256 are: 3257 3258 - "private" 3259 - "global" 3260 - "constant" 3261 - "local" 3262 - "generic" 3263 - "region" 3264 3265 .. TODO:: 3266 3267 Is "global_buffer" only "global" 3268 or "constant"? Is 3269 "dynamic_shared_pointer" always 3270 "local"? Can HCC allow "generic"? 3271 How can "private" or "region" 3272 ever happen? 3273 3274 ".access" string Kernel argument access 3275 qualifier. Only present if 3276 ".value_kind" is "image" or 3277 "pipe". Values 3278 are: 3279 3280 - "read_only" 3281 - "write_only" 3282 - "read_write" 3283 3284 .. TODO:: 3285 3286 Does this apply to 3287 "global_buffer"? 3288 3289 ".actual_access" string The actual memory accesses 3290 performed by the kernel on the 3291 kernel argument. Only present if 3292 ".value_kind" is "global_buffer", 3293 "image", or "pipe". This may be 3294 more restrictive than indicated 3295 by ".access" to reflect what the 3296 kernel actual does. If not 3297 present then the runtime must 3298 assume what is implied by 3299 ".access" and ".is_const" . Values 3300 are: 3301 3302 - "read_only" 3303 - "write_only" 3304 - "read_write" 3305 3306 ".is_const" boolean Indicates if the kernel argument 3307 is const qualified. Only present 3308 if ".value_kind" is 3309 "global_buffer". 3310 3311 ".is_restrict" boolean Indicates if the kernel argument 3312 is restrict qualified. Only 3313 present if ".value_kind" is 3314 "global_buffer". 3315 3316 ".is_volatile" boolean Indicates if the kernel argument 3317 is volatile qualified. Only 3318 present if ".value_kind" is 3319 "global_buffer". 3320 3321 ".is_pipe" boolean Indicates if the kernel argument 3322 is pipe qualified. Only present 3323 if ".value_kind" is "pipe". 3324 3325 .. TODO:: 3326 3327 Can "global_buffer" be pipe 3328 qualified? 3329 3330 ====================== ============== ========= ================================ 3331 3332.. _amdgpu-amdhsa-code-object-metadata-v4: 3333 3334Code Object V4 Metadata 3335+++++++++++++++++++++++ 3336 3337.. warning:: 3338 Code object V4 is not the default code object version emitted by this version 3339 of LLVM. 3340 3341Code object V4 metadata is the same as 3342:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3343defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3344 3345 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3346 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3347 3348 ================= ============== ========= ======================================= 3349 String Key Value Type Required? Description 3350 ================= ============== ========= ======================================= 3351 "amdhsa.version" sequence of Required - The first integer is the major 3352 2 integers version. Currently 1. 3353 - The second integer is the minor 3354 version. Currently 1. 3355 "amdhsa.target" string Required The target name of the code using the syntax: 3356 3357 .. code:: 3358 3359 <target-triple> [ "-" <target-id> ] 3360 3361 A canonical target ID must be 3362 used. See :ref:`amdgpu-target-triples` 3363 and :ref:`amdgpu-target-id`. 3364 ================= ============== ========= ======================================= 3365 3366.. 3367 3368Kernel Dispatch 3369~~~~~~~~~~~~~~~ 3370 3371The HSA architected queuing language (AQL) defines a user space memory interface 3372that can be used to control the dispatch of kernels, in an agent independent 3373way. An agent can have zero or more AQL queues created for it using an HSA 3374compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3375are 64 bytes) can be placed. See the *HSA Platform System Architecture 3376Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3377 3378The packet processor of a kernel agent is responsible for detecting and 3379dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3380packet processor is implemented by the hardware command processor (CP), 3381asynchronous dispatch controller (ADC) and shader processor input controller 3382(SPI). 3383 3384An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3385the kernel mode driver to initialize and register the AQL queue with CP. 3386 3387To dispatch a kernel the following actions are performed. This can occur in the 3388CPU host program, or from an HSA kernel executing on a GPU. 3389 33901. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3391 executed is obtained. 33922. A pointer to the kernel descriptor (see 3393 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3394 It must be for a kernel that is contained in a code object that that was 3395 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3396 queue is associated. 33973. Space is allocated for the kernel arguments using the HSA compatible runtime 3398 allocator for a memory region with the kernarg property for the kernel agent 3399 that will execute the kernel. It must be at least 16-byte aligned. 34004. Kernel argument values are assigned to the kernel argument memory 3401 allocation. The layout is defined in the *HSA Programmer's Language 3402 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3403 kernel argument memory in the same way constant memory is accessed. (Note 3404 that the HSA specification allows an implementation to copy the kernel 3405 argument contents to another location that is accessed by the kernel.) 34065. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3407 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3408 for the packet. The packet must be set up, and the final write must use an 3409 atomic store release to set the packet kind to ensure the packet contents are 3410 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3411 notify the kernel agent that the AQL queue has been updated. These rules, and 3412 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3413 System Architecture Specification* [HSA]_. 34146. A kernel dispatch packet includes information about the actual dispatch, 3415 such as grid and work-group size, together with information from the code 3416 object about the kernel, such as segment sizes. The HSA compatible runtime 3417 queries on the kernel symbol can be used to obtain the code object values 3418 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 34197. CP executes micro-code and is responsible for detecting and setting up the 3420 GPU to execute the wavefronts of a kernel dispatch. 34218. CP ensures that when the a wavefront starts executing the kernel machine 3422 code, the scalar general purpose registers (SGPR) and vector general purpose 3423 registers (VGPR) are set up as required by the machine code. The required 3424 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3425 register state is defined in 3426 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 34279. The prolog of the kernel machine code (see 3428 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3429 before continuing executing the machine code that corresponds to the kernel. 343010. When the kernel dispatch has completed execution, CP signals the completion 3431 signal specified in the kernel dispatch packet if not 0. 3432 3433.. _amdgpu-amdhsa-memory-spaces: 3434 3435Memory Spaces 3436~~~~~~~~~~~~~ 3437 3438The memory space properties are: 3439 3440 .. table:: AMDHSA Memory Spaces 3441 :name: amdgpu-amdhsa-memory-spaces-table 3442 3443 ================= =========== ======== ======= ================== 3444 Memory Space Name HSA Segment Hardware Address NULL Value 3445 Name Name Size 3446 ================= =========== ======== ======= ================== 3447 Private private scratch 32 0x00000000 3448 Local group LDS 32 0xFFFFFFFF 3449 Global global global 64 0x0000000000000000 3450 Constant constant *same as 64 0x0000000000000000 3451 global* 3452 Generic flat flat 64 0x0000000000000000 3453 Region N/A GDS 32 *not implemented 3454 for AMDHSA* 3455 ================= =========== ======== ======= ================== 3456 3457The global and constant memory spaces both use global virtual addresses, which 3458are the same virtual address space used by the CPU. However, some virtual 3459addresses may only be accessible to the CPU, some only accessible by the GPU, 3460and some by both. 3461 3462Using the constant memory space indicates that the data will not change during 3463the execution of the kernel. This allows scalar read instructions to be 3464used. The vector and scalar L1 caches are invalidated of volatile data before 3465each kernel dispatch execution to allow constant memory to change values between 3466kernel dispatches. 3467 3468The local memory space uses the hardware Local Data Store (LDS) which is 3469automatically allocated when the hardware creates work-groups of wavefronts, and 3470freed when all the wavefronts of a work-group have terminated. The data store 3471(DS) instructions can be used to access it. 3472 3473The private memory space uses the hardware scratch memory support. If the kernel 3474uses scratch, then the hardware allocates memory that is accessed using 3475wavefront lane dword (4 byte) interleaving. The mapping used from private 3476address to physical address is: 3477 3478 ``wavefront-scratch-base + 3479 (private-address * wavefront-size * 4) + 3480 (wavefront-lane-id * 4)`` 3481 3482There are different ways that the wavefront scratch base address is determined 3483by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3484memory can be accessed in an interleaved manner using buffer instruction with 3485the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3486instructions, or by flat instructions. If each lane of a wavefront accesses the 3487same private address, the interleaving results in adjacent dwords being accessed 3488and hence requires fewer cache lines to be fetched. Multi-dword access is not 3489supported except by flat and scratch instructions in GFX9-GFX10. 3490 3491The generic address space uses the hardware flat address support available in 3492GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3493local apertures), that are outside the range of addressible global memory, to 3494map from a flat address to a private or local address. 3495 3496FLAT instructions can take a flat address and access global, private (scratch) 3497and group (LDS) memory depending in if the address is within one of the 3498aperture ranges. Flat access to scratch requires hardware aperture setup and 3499setup in the kernel prologue (see 3500:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3501hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3502:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3503 3504To convert between a segment address and a flat address the base address of the 3505apertures address can be used. For GFX7-GFX8 these are available in the 3506:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3507Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3508GFX9-GFX10 the aperture base addresses are directly available as inline constant 3509registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3510address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3511which makes it easier to convert from flat to segment or segment to flat. 3512 3513Image and Samplers 3514~~~~~~~~~~~~~~~~~~ 3515 3516Image and sample handles created by an HSA compatible runtime (see 3517:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3518object respectively. In order to support the HSA ``query_sampler`` operations 3519two extra dwords are used to store the HSA BRIG enumeration values for the 3520queries that are not trivially deducible from the S# representation. 3521 3522HSA Signals 3523~~~~~~~~~~~ 3524 3525HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3526are 64-bit addresses of a structure allocated in memory accessible from both the 3527CPU and GPU. The structure is defined by the runtime and subject to change 3528between releases. For example, see [AMD-ROCm-github]_. 3529 3530.. _amdgpu-amdhsa-hsa-aql-queue: 3531 3532HSA AQL Queue 3533~~~~~~~~~~~~~ 3534 3535The HSA AQL queue structure is defined by an HSA compatible runtime (see 3536:ref:`amdgpu-os`) and subject to change between releases. For example, see 3537[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3538certain language features such as the flat address aperture bases. It also 3539contains fields used by CP such as managing the allocation of scratch memory. 3540 3541.. _amdgpu-amdhsa-kernel-descriptor: 3542 3543Kernel Descriptor 3544~~~~~~~~~~~~~~~~~ 3545 3546A kernel descriptor consists of the information needed by CP to initiate the 3547execution of a kernel, including the entry point address of the machine code 3548that implements the kernel. 3549 3550Code Object V3 Kernel Descriptor 3551++++++++++++++++++++++++++++++++ 3552 3553CP microcode requires the Kernel descriptor to be allocated on 64-byte 3554alignment. 3555 3556The fields used by CP for code objects before V3 also match those specified in 3557:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3558 3559 .. table:: Code Object V3 Kernel Descriptor 3560 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3561 3562 ======= ======= =============================== ============================ 3563 Bits Size Field Name Description 3564 ======= ======= =============================== ============================ 3565 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3566 address space memory 3567 required for a work-group 3568 in bytes. This does not 3569 include any dynamically 3570 allocated local address 3571 space memory that may be 3572 added when the kernel is 3573 dispatched. 3574 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3575 private address space 3576 memory required for a 3577 work-item in bytes. 3578 Additional space may need to 3579 be added to this value if 3580 the call stack has 3581 non-inlined function calls. 3582 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3583 memory pointed to by the 3584 AQL dispatch packet. The 3585 kernarg memory is used to 3586 pass arguments to the 3587 kernel. 3588 3589 * If the kernarg pointer in 3590 the dispatch packet is NULL 3591 then there are no kernel 3592 arguments. 3593 * If the kernarg pointer in 3594 the dispatch packet is 3595 not NULL and this value 3596 is 0 then the kernarg 3597 memory size is 3598 unspecified. 3599 * If the kernarg pointer in 3600 the dispatch packet is 3601 not NULL and this value 3602 is not 0 then the value 3603 specifies the kernarg 3604 memory size in bytes. It 3605 is recommended to provide 3606 a value as it may be used 3607 by CP to optimize making 3608 the kernarg memory 3609 visible to the kernel 3610 code. 3611 3612 127:96 4 bytes Reserved, must be 0. 3613 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3614 negative) from base 3615 address of kernel 3616 descriptor to kernel's 3617 entry point instruction 3618 which must be 256 byte 3619 aligned. 3620 351:272 20 Reserved, must be 0. 3621 bytes 3622 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3623 Reserved, must be 0. 3624 GFX90A 3625 Compute Shader (CS) 3626 program settings used by 3627 CP to set up 3628 ``COMPUTE_PGM_RSRC3`` 3629 configuration 3630 register. See 3631 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3632 GFX10 3633 Compute Shader (CS) 3634 program settings used by 3635 CP to set up 3636 ``COMPUTE_PGM_RSRC3`` 3637 configuration 3638 register. See 3639 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3640 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3641 program settings used by 3642 CP to set up 3643 ``COMPUTE_PGM_RSRC1`` 3644 configuration 3645 register. See 3646 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3647 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3648 program settings used by 3649 CP to set up 3650 ``COMPUTE_PGM_RSRC2`` 3651 configuration 3652 register. See 3653 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3654 458:448 7 bits *See separate bits below.* Enable the setup of the 3655 SGPR user data registers 3656 (see 3657 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3658 3659 The total number of SGPR 3660 user data registers 3661 requested must not exceed 3662 16 and match value in 3663 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3664 Any requests beyond 16 3665 will be ignored. 3666 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3667 _BUFFER 3668 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3669 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3670 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3671 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3672 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT 3673 3674 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3675 _SIZE 3676 457:455 3 bits Reserved, must be 0. 3677 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3678 Reserved, must be 0. 3679 GFX10 3680 - If 0 execute in 3681 wavefront size 64 mode. 3682 - If 1 execute in 3683 native wavefront size 3684 32 mode. 3685 463:459 1 bit Reserved, must be 0. 3686 464 1 bit RESERVED_464 Deprecated, must be 0. 3687 467:465 3 bits Reserved, must be 0. 3688 468 1 bit RESERVED_468 Deprecated, must be 0. 3689 469:471 3 bits Reserved, must be 0. 3690 511:472 5 bytes Reserved, must be 0. 3691 512 **Total size 64 bytes.** 3692 ======= ==================================================================== 3693 3694.. 3695 3696 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3697 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3698 3699 ======= ======= =============================== =========================================================================== 3700 Bits Size Field Name Description 3701 ======= ======= =============================== =========================================================================== 3702 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3703 blocks used by each work-item; 3704 granularity is device 3705 specific: 3706 3707 GFX6-GFX9 3708 - vgprs_used 0..256 3709 - max(0, ceil(vgprs_used / 4) - 1) 3710 GFX90A 3711 - vgprs_used 0..512 3712 - vgprs_used = align(arch_vgprs, 4) 3713 + acc_vgprs 3714 - max(0, ceil(vgprs_used / 8) - 1) 3715 GFX10 (wavefront size 64) 3716 - max_vgpr 1..256 3717 - max(0, ceil(vgprs_used / 4) - 1) 3718 GFX10 (wavefront size 32) 3719 - max_vgpr 1..256 3720 - max(0, ceil(vgprs_used / 8) - 1) 3721 3722 Where vgprs_used is defined 3723 as the highest VGPR number 3724 explicitly referenced plus 3725 one. 3726 3727 Used by CP to set up 3728 ``COMPUTE_PGM_RSRC1.VGPRS``. 3729 3730 The 3731 :ref:`amdgpu-assembler` 3732 calculates this 3733 automatically for the 3734 selected processor from 3735 values provided to the 3736 `.amdhsa_kernel` directive 3737 by the 3738 `.amdhsa_next_free_vgpr` 3739 nested directive (see 3740 :ref:`amdhsa-kernel-directives-table`). 3741 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3742 blocks used by a wavefront; 3743 granularity is device 3744 specific: 3745 3746 GFX6-GFX8 3747 - sgprs_used 0..112 3748 - max(0, ceil(sgprs_used / 8) - 1) 3749 GFX9 3750 - sgprs_used 0..112 3751 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3752 GFX10 3753 Reserved, must be 0. 3754 (128 SGPRs always 3755 allocated.) 3756 3757 Where sgprs_used is 3758 defined as the highest 3759 SGPR number explicitly 3760 referenced plus one, plus 3761 a target specific number 3762 of additional special 3763 SGPRs for VCC, 3764 FLAT_SCRATCH (GFX7+) and 3765 XNACK_MASK (GFX8+), and 3766 any additional 3767 target specific 3768 limitations. It does not 3769 include the 16 SGPRs added 3770 if a trap handler is 3771 enabled. 3772 3773 The target specific 3774 limitations and special 3775 SGPR layout are defined in 3776 the hardware 3777 documentation, which can 3778 be found in the 3779 :ref:`amdgpu-processors` 3780 table. 3781 3782 Used by CP to set up 3783 ``COMPUTE_PGM_RSRC1.SGPRS``. 3784 3785 The 3786 :ref:`amdgpu-assembler` 3787 calculates this 3788 automatically for the 3789 selected processor from 3790 values provided to the 3791 `.amdhsa_kernel` directive 3792 by the 3793 `.amdhsa_next_free_sgpr` 3794 and `.amdhsa_reserve_*` 3795 nested directives (see 3796 :ref:`amdhsa-kernel-directives-table`). 3797 11:10 2 bits PRIORITY Must be 0. 3798 3799 Start executing wavefront 3800 at the specified priority. 3801 3802 CP is responsible for 3803 filling in 3804 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3805 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3806 with specified rounding 3807 mode for single (32 3808 bit) floating point 3809 precision floating point 3810 operations. 3811 3812 Floating point rounding 3813 mode values are defined in 3814 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3815 3816 Used by CP to set up 3817 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3818 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3819 with specified rounding 3820 denorm mode for half/double (16 3821 and 64-bit) floating point 3822 precision floating point 3823 operations. 3824 3825 Floating point rounding 3826 mode values are defined in 3827 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3828 3829 Used by CP to set up 3830 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3831 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3832 with specified denorm mode 3833 for single (32 3834 bit) floating point 3835 precision floating point 3836 operations. 3837 3838 Floating point denorm mode 3839 values are defined in 3840 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3841 3842 Used by CP to set up 3843 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3844 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3845 with specified denorm mode 3846 for half/double (16 3847 and 64-bit) floating point 3848 precision floating point 3849 operations. 3850 3851 Floating point denorm mode 3852 values are defined in 3853 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3854 3855 Used by CP to set up 3856 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3857 20 1 bit PRIV Must be 0. 3858 3859 Start executing wavefront 3860 in privilege trap handler 3861 mode. 3862 3863 CP is responsible for 3864 filling in 3865 ``COMPUTE_PGM_RSRC1.PRIV``. 3866 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3867 with DX10 clamp mode 3868 enabled. Used by the vector 3869 ALU to force DX10 style 3870 treatment of NaN's (when 3871 set, clamp NaN to zero, 3872 otherwise pass NaN 3873 through). 3874 3875 Used by CP to set up 3876 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3877 22 1 bit DEBUG_MODE Must be 0. 3878 3879 Start executing wavefront 3880 in single step mode. 3881 3882 CP is responsible for 3883 filling in 3884 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3885 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3886 with IEEE mode 3887 enabled. Floating point 3888 opcodes that support 3889 exception flag gathering 3890 will quiet and propagate 3891 signaling-NaN inputs per 3892 IEEE 754-2008. Min_dx10 and 3893 max_dx10 become IEEE 3894 754-2008 compliant due to 3895 signaling-NaN propagation 3896 and quieting. 3897 3898 Used by CP to set up 3899 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3900 24 1 bit BULKY Must be 0. 3901 3902 Only one work-group allowed 3903 to execute on a compute 3904 unit. 3905 3906 CP is responsible for 3907 filling in 3908 ``COMPUTE_PGM_RSRC1.BULKY``. 3909 25 1 bit CDBG_USER Must be 0. 3910 3911 Flag that can be used to 3912 control debugging code. 3913 3914 CP is responsible for 3915 filling in 3916 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3917 26 1 bit FP16_OVFL GFX6-GFX8 3918 Reserved, must be 0. 3919 GFX9-GFX10 3920 Wavefront starts execution 3921 with specified fp16 overflow 3922 mode. 3923 3924 - If 0, fp16 overflow generates 3925 +/-INF values. 3926 - If 1, fp16 overflow that is the 3927 result of an +/-INF input value 3928 or divide by 0 produces a +/-INF, 3929 otherwise clamps computed 3930 overflow to +/-MAX_FP16 as 3931 appropriate. 3932 3933 Used by CP to set up 3934 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3935 28:27 2 bits Reserved, must be 0. 3936 29 1 bit WGP_MODE GFX6-GFX9 3937 Reserved, must be 0. 3938 GFX10 3939 - If 0 execute work-groups in 3940 CU wavefront execution mode. 3941 - If 1 execute work-groups on 3942 in WGP wavefront execution mode. 3943 3944 See :ref:`amdgpu-amdhsa-memory-model`. 3945 3946 Used by CP to set up 3947 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3948 30 1 bit MEM_ORDERED GFX6-GFX9 3949 Reserved, must be 0. 3950 GFX10 3951 Controls the behavior of the 3952 s_waitcnt's vmcnt and vscnt 3953 counters. 3954 3955 - If 0 vmcnt reports completion 3956 of load and atomic with return 3957 out of order with sample 3958 instructions, and the vscnt 3959 reports the completion of 3960 store and atomic without 3961 return in order. 3962 - If 1 vmcnt reports completion 3963 of load, atomic with return 3964 and sample instructions in 3965 order, and the vscnt reports 3966 the completion of store and 3967 atomic without return in order. 3968 3969 Used by CP to set up 3970 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3971 31 1 bit FWD_PROGRESS GFX6-GFX9 3972 Reserved, must be 0. 3973 GFX10 3974 - If 0 execute SIMD wavefronts 3975 using oldest first policy. 3976 - If 1 execute SIMD wavefronts to 3977 ensure wavefronts will make some 3978 forward progress. 3979 3980 Used by CP to set up 3981 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3982 32 **Total size 4 bytes** 3983 ======= =================================================================================================================== 3984 3985.. 3986 3987 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3988 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3989 3990 ======= ======= =============================== =========================================================================== 3991 Bits Size Field Name Description 3992 ======= ======= =============================== =========================================================================== 3993 0 1 bit ENABLE_PRIVATE_SEGMENT Enable the setup of the 3994 private segment. 3995 3996 In addition, enable the 3997 setup of the SGPR 3998 wavefront scratch offset 3999 system register (see 4000 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4001 4002 Used by CP to set up 4003 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4004 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4005 user data registers 4006 requested. This number must 4007 match the number of user 4008 data registers enabled. 4009 4010 Used by CP to set up 4011 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4012 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4013 4014 This bit represents 4015 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4016 which is set by the CP if 4017 the runtime has installed a 4018 trap handler. 4019 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4020 system SGPR register for 4021 the work-group id in the X 4022 dimension (see 4023 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4024 4025 Used by CP to set up 4026 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4027 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4028 system SGPR register for 4029 the work-group id in the Y 4030 dimension (see 4031 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4032 4033 Used by CP to set up 4034 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4035 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4036 system SGPR register for 4037 the work-group id in the Z 4038 dimension (see 4039 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4040 4041 Used by CP to set up 4042 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4043 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4044 system SGPR register for 4045 work-group information (see 4046 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4047 4048 Used by CP to set up 4049 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4050 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4051 VGPR system registers used 4052 for the work-item ID. 4053 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4054 defines the values. 4055 4056 Used by CP to set up 4057 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4058 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4059 4060 Wavefront starts execution 4061 with address watch 4062 exceptions enabled which 4063 are generated when L1 has 4064 witnessed a thread access 4065 an *address of 4066 interest*. 4067 4068 CP is responsible for 4069 filling in the address 4070 watch bit in 4071 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4072 according to what the 4073 runtime requests. 4074 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4075 4076 Wavefront starts execution 4077 with memory violation 4078 exceptions exceptions 4079 enabled which are generated 4080 when a memory violation has 4081 occurred for this wavefront from 4082 L1 or LDS 4083 (write-to-read-only-memory, 4084 mis-aligned atomic, LDS 4085 address out of range, 4086 illegal address, etc.). 4087 4088 CP sets the memory 4089 violation bit in 4090 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4091 according to what the 4092 runtime requests. 4093 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4094 4095 CP uses the rounded value 4096 from the dispatch packet, 4097 not this value, as the 4098 dispatch may contain 4099 dynamically allocated group 4100 segment memory. CP writes 4101 directly to 4102 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4103 4104 Amount of group segment 4105 (LDS) to allocate for each 4106 work-group. Granularity is 4107 device specific: 4108 4109 GFX6 4110 roundup(lds-size / (64 * 4)) 4111 GFX7-GFX10 4112 roundup(lds-size / (128 * 4)) 4113 4114 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4115 _INVALID_OPERATION with specified exceptions 4116 enabled. 4117 4118 Used by CP to set up 4119 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4120 (set from bits 0..6). 4121 4122 IEEE 754 FP Invalid 4123 Operation 4124 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4125 _SOURCE input operands is a 4126 denormal number 4127 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4128 _DIVISION_BY_ZERO Zero 4129 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4130 _OVERFLOW 4131 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4132 _UNDERFLOW 4133 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4134 _INEXACT 4135 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4136 _ZERO (rcp_iflag_f32 instruction 4137 only) 4138 31 1 bit Reserved, must be 0. 4139 32 **Total size 4 bytes.** 4140 ======= =================================================================================================================== 4141 4142.. 4143 4144 .. table:: compute_pgm_rsrc3 for GFX90A 4145 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4146 4147 ======= ======= =============================== =========================================================================== 4148 Bits Size Field Name Description 4149 ======= ======= =============================== =========================================================================== 4150 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4151 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4152 63 - accum-offset = 256. 4153 6:15 10 Reserved, must be 0. 4154 bits 4155 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4156 launched in the same CU. 4157 - If 1 the waves of a work-group can be 4158 launched in different CUs. The waves 4159 cannot use S_BARRIER or LDS. 4160 17:31 15 Reserved, must be 0. 4161 bits 4162 32 **Total size 4 bytes.** 4163 ======= =================================================================================================================== 4164 4165.. 4166 4167 .. table:: compute_pgm_rsrc3 for GFX10 4168 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4169 4170 ======= ======= =============================== =========================================================================== 4171 Bits Size Field Name Description 4172 ======= ======= =============================== =========================================================================== 4173 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4174 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4175 31:4 28 Reserved, must be 0. 4176 bits 4177 32 **Total size 4 bytes.** 4178 ======= =================================================================================================================== 4179 4180.. 4181 4182 .. table:: Floating Point Rounding Mode Enumeration Values 4183 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4184 4185 ====================================== ===== ============================== 4186 Enumeration Name Value Description 4187 ====================================== ===== ============================== 4188 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4189 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4190 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4191 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4192 ====================================== ===== ============================== 4193 4194.. 4195 4196 .. table:: Floating Point Denorm Mode Enumeration Values 4197 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4198 4199 ====================================== ===== ============================== 4200 Enumeration Name Value Description 4201 ====================================== ===== ============================== 4202 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4203 Denorms 4204 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4205 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4206 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4207 ====================================== ===== ============================== 4208 4209.. 4210 4211 .. table:: System VGPR Work-Item ID Enumeration Values 4212 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4213 4214 ======================================== ===== ============================ 4215 Enumeration Name Value Description 4216 ======================================== ===== ============================ 4217 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4218 ID. 4219 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4220 dimensions ID. 4221 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4222 dimensions ID. 4223 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4224 ======================================== ===== ============================ 4225 4226.. _amdgpu-amdhsa-initial-kernel-execution-state: 4227 4228Initial Kernel Execution State 4229~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4230 4231This section defines the register state that will be set up by the packet 4232processor prior to the start of execution of every wavefront. This is limited by 4233the constraints of the hardware controllers of CP/ADC/SPI. 4234 4235The order of the SGPR registers is defined, but the compiler can specify which 4236ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4237fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4238for enabled registers are dense starting at SGPR0: the first enabled register is 4239SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4240an SGPR number. 4241 4242The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4243all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4244using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4245actually initialized. These are then immediately followed by the System SGPRs 4246that are set up by ADC/SPI and can have different values for each wavefront of 4247the grid dispatch. 4248 4249SGPR register initial state is defined in 4250:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4251 4252 .. table:: SGPR Register Set Up Order 4253 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4254 4255 ========== ========================== ====== ============================== 4256 SGPR Order Name Number Description 4257 (kernel descriptor enable of 4258 field) SGPRs 4259 ========== ========================== ====== ============================== 4260 First Private Segment Buffer 4 See 4261 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4262 _segment_buffer) 4263 then Dispatch Ptr 2 64-bit address of AQL dispatch 4264 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4265 actually executing. 4266 then Queue Ptr 2 64-bit address of amd_queue_t 4267 (enable_sgpr_queue_ptr) object for AQL queue on which 4268 the dispatch packet was 4269 queued. 4270 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4271 (enable_sgpr_kernarg segment. This is directly 4272 _segment_ptr) copied from the 4273 kernarg_address in the kernel 4274 dispatch packet. 4275 4276 Having CP load it once avoids 4277 loading it at the beginning of 4278 every wavefront. 4279 then Dispatch Id 2 64-bit Dispatch ID of the 4280 (enable_sgpr_dispatch_id) dispatch packet being 4281 executed. 4282 then Flat Scratch Init 2 See 4283 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4284 then Private Segment Size 1 The 32-bit byte size of a 4285 (enable_sgpr_private single 4286 work-item's 4287 scratch_segment_size) memory 4288 allocation. This is the 4289 value from the kernel 4290 dispatch packet Private 4291 Segment Byte Size rounded up 4292 by CP to a multiple of 4293 DWORD. 4294 4295 Having CP load it once avoids 4296 loading it at the beginning of 4297 every wavefront. 4298 4299 This is not used for 4300 GFX7-GFX8 since it is the same 4301 value as the second SGPR of 4302 Flat Scratch Init. However, it 4303 may be needed for GFX9-GFX10 which 4304 changes the meaning of the 4305 Flat Scratch Init value. 4306 then Grid Work-Group Count X 1 32-bit count of the number of 4307 (enable_sgpr_grid work-groups in the X dimension 4308 _workgroup_count_X) for the grid being 4309 executed. Computed from the 4310 fields in the kernel dispatch 4311 packet as ((grid_size.x + 4312 workgroup_size.x - 1) / 4313 workgroup_size.x). 4314 then Grid Work-Group Count Y 1 32-bit count of the number of 4315 (enable_sgpr_grid work-groups in the Y dimension 4316 _workgroup_count_Y && for the grid being 4317 less than 16 previous executed. Computed from the 4318 SGPRs) fields in the kernel dispatch 4319 packet as ((grid_size.y + 4320 workgroup_size.y - 1) / 4321 workgroupSize.y). 4322 4323 Only initialized if <16 4324 previous SGPRs initialized. 4325 then Grid Work-Group Count Z 1 32-bit count of the number of 4326 (enable_sgpr_grid work-groups in the Z dimension 4327 _workgroup_count_Z && for the grid being 4328 less than 16 previous executed. Computed from the 4329 SGPRs) fields in the kernel dispatch 4330 packet as ((grid_size.z + 4331 workgroup_size.z - 1) / 4332 workgroupSize.z). 4333 4334 Only initialized if <16 4335 previous SGPRs initialized. 4336 then Work-Group Id X 1 32-bit work-group id in X 4337 (enable_sgpr_workgroup_id dimension of grid for 4338 _X) wavefront. 4339 then Work-Group Id Y 1 32-bit work-group id in Y 4340 (enable_sgpr_workgroup_id dimension of grid for 4341 _Y) wavefront. 4342 then Work-Group Id Z 1 32-bit work-group id in Z 4343 (enable_sgpr_workgroup_id dimension of grid for 4344 _Z) wavefront. 4345 then Work-Group Info 1 {first_wavefront, 14'b0000, 4346 (enable_sgpr_workgroup ordered_append_term[10:0], 4347 _info) threadgroup_size_in_wavefronts[5:0]} 4348 then Scratch Wavefront Offset 1 See 4349 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4350 _segment_wavefront_offset) and 4351 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4352 ========== ========================== ====== ============================== 4353 4354The order of the VGPR registers is defined, but the compiler can specify which 4355ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4356fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4357for enabled registers are dense starting at VGPR0: the first enabled register is 4358VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4359VGPR number. 4360 4361There are different methods used for the VGPR initial state: 4362 4363* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4364 specifies otherwise, a separate VGPR register is used per work-item ID. The 4365 VGPR register initial state for this method is defined in 4366 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4367* If *Target Properties* column of :ref:`amdgpu-processor-table` 4368 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4369 for all work-item IDs. The register layout for this method is defined in 4370 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4371 4372 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4373 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4374 4375 ========== ========================== ====== ============================== 4376 VGPR Order Name Number Description 4377 (kernel descriptor enable of 4378 field) VGPRs 4379 ========== ========================== ====== ============================== 4380 First Work-Item Id X 1 32-bit work-item id in X 4381 (Always initialized) dimension of work-group for 4382 wavefront lane. 4383 then Work-Item Id Y 1 32-bit work-item id in Y 4384 (enable_vgpr_workitem_id dimension of work-group for 4385 > 0) wavefront lane. 4386 then Work-Item Id Z 1 32-bit work-item id in Z 4387 (enable_vgpr_workitem_id dimension of work-group for 4388 > 1) wavefront lane. 4389 ========== ========================== ====== ============================== 4390 4391.. 4392 4393 .. table:: Register Layout for Packed Work-Item ID Method 4394 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4395 4396 ======= ======= ================ ========================================= 4397 Bits Size Field Name Description 4398 ======= ======= ================ ========================================= 4399 0:9 10 bits Work-Item Id X Work-item id in X 4400 dimension of work-group for 4401 wavefront lane. 4402 4403 Always initialized. 4404 4405 10:19 10 bits Work-Item Id Y Work-item id in Y 4406 dimension of work-group for 4407 wavefront lane. 4408 4409 Initialized if enable_vgpr_workitem_id > 4410 0, otherwise set to 0. 4411 20:29 10 bits Work-Item Id Z Work-item id in Z 4412 dimension of work-group for 4413 wavefront lane. 4414 4415 Initialized if enable_vgpr_workitem_id > 4416 1, otherwise set to 0. 4417 30:31 2 bits Reserved, set to 0. 4418 ======= ======= ================ ========================================= 4419 4420The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4421 44221. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4423 registers. 44242. Work-group Id registers X, Y, Z are set by ADC which supports any 4425 combination including none. 44263. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4427 its value cannot be included with the flat scratch init value which is per 4428 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 44294. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4430 or (X, Y, Z). 44315. Flat Scratch register pair initialization is described in 4432 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4433 4434The global segment can be accessed either using buffer instructions (GFX6 which 4435has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4436instructions (GFX9-GFX10). 4437 4438If buffer operations are used, then the compiler can generate a V# with the 4439following properties: 4440 4441* base address of 0 4442* no swizzle 4443* ATC: 1 if IOMMU present (such as APU) 4444* ptr64: 1 4445* MTYPE set to support memory coherence that matches the runtime (such as CC for 4446 APU and NC for dGPU). 4447 4448.. _amdgpu-amdhsa-kernel-prolog: 4449 4450Kernel Prolog 4451~~~~~~~~~~~~~ 4452 4453The compiler performs initialization in the kernel prologue depending on the 4454target and information about things like stack usage in the kernel and called 4455functions. Some of this initialization requires the compiler to request certain 4456User and System SGPRs be present in the 4457:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4458:ref:`amdgpu-amdhsa-kernel-descriptor`. 4459 4460.. _amdgpu-amdhsa-kernel-prolog-cfi: 4461 4462CFI 4463+++ 4464 44651. The CFI return address is undefined. 4466 44672. The CFI CFA is defined using an expression which evaluates to a location 4468 description that comprises one memory location description for the 4469 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4470 4471.. _amdgpu-amdhsa-kernel-prolog-m0: 4472 4473M0 4474++ 4475 4476GFX6-GFX8 4477 The M0 register must be initialized with a value at least the total LDS size 4478 if the kernel may access LDS via DS or flat operations. Total LDS size is 4479 available in dispatch packet. For M0, it is also possible to use maximum 4480 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4481 GFX7-GFX8). 4482GFX9-GFX10 4483 The M0 register is not used for range checking LDS accesses and so does not 4484 need to be initialized in the prolog. 4485 4486.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4487 4488Stack Pointer 4489+++++++++++++ 4490 4491If the kernel has function calls it must set up the ABI stack pointer described 4492in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4493SGPR32 to the unswizzled scratch offset of the address past the last local 4494allocation. 4495 4496.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4497 4498Frame Pointer 4499+++++++++++++ 4500 4501If the kernel needs a frame pointer for the reasons defined in 4502``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4503kernel prolog. If a frame pointer is not required then all uses of the frame 4504pointer are replaced with immediate ``0`` offsets. 4505 4506.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4507 4508Flat Scratch 4509++++++++++++ 4510 4511There are different methods used for initializing flat scratch: 4512 4513* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4514 specifies *Does not support generic address space*: 4515 4516 Flat scratch is not supported and there is no flat scratch register pair. 4517 4518* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4519 specifies *Offset flat scratch*: 4520 4521 If the kernel or any function it calls may use flat operations to access 4522 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4523 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4524 Scratch Wavefront Offset SGPR registers (see 4525 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4526 4527 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4528 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4529 being managed by SPI for the queue executing the kernel dispatch. This is 4530 the same value used in the Scratch Segment Buffer V# base address. 4531 4532 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4533 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4534 4535 The prolog must add the value of Scratch Wavefront Offset to get the 4536 wavefront's byte scratch backing memory offset from 4537 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4538 4539 The Scratch Wavefront Offset must also be used as an offset with Private 4540 segment address when using the Scratch Segment Buffer. 4541 4542 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4543 shifted by 8 before moving into FLAT_SCRATCH_HI. 4544 4545 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4546 SGPRn is the highest numbered SGPR allocated to the wavefront). 4547 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4548 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4549 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4550 aperture. 4551 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4552 work-items scratch memory usage. 4553 4554 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4555 checks that the value in the kernel dispatch packet Private Segment Byte 4556 Size is not larger and requests the runtime to increase the queue's scratch 4557 size if necessary. 4558 4559 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4560 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4561 loading it at the beginning of every wavefront. 4562 4563 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4564 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4565 in flat memory instructions. 4566 4567* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4568 specifies *Absolute flat scratch*: 4569 4570 If the kernel or any function it calls may use flat operations to access 4571 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4572 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4573 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4574 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4575 4576 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4577 memory being managed by SPI for the queue executing the kernel dispatch. 4578 4579 CP obtains this from the runtime. 4580 4581 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4582 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4583 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4584 memory instructions. 4585 4586 The Scratch Wavefront Offset must also be used as an offset with Private 4587 segment address when using the Scratch Segment Buffer (see 4588 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4589 4590.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4591 4592Private Segment Buffer 4593++++++++++++++++++++++ 4594 4595Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4596that are used as a V# to access scratch. CP uses the value provided by the 4597runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4598access the private memory space using a segment address. See 4599:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4600 4601The scratch V# is a four-aligned SGPR and always selected for the kernel as 4602follows: 4603 4604 - If it is known during instruction selection that there is stack usage, 4605 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4606 optimizations are disabled (``-O0``), if stack objects already exist (for 4607 locals, etc.), or if there are any function calls. 4608 4609 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4610 are reserved for the tentative scratch V#. These will be used if it is 4611 determined that spilling is needed. 4612 4613 - If no use is made of the tentative scratch V#, then it is unreserved, 4614 and the register count is determined ignoring it. 4615 - If use is made of the tentative scratch V#, then its register numbers 4616 are shifted to the first four-aligned SGPR index after the highest one 4617 allocated by the register allocator, and all uses are updated. The 4618 register count includes them in the shifted location. 4619 - In either case, if the processor has the SGPR allocation bug, the 4620 tentative allocation is not shifted or unreserved in order to ensure 4621 the register count is higher to workaround the bug. 4622 4623 .. note:: 4624 4625 This approach of using a tentative scratch V# and shifting the register 4626 numbers if used avoids having to perform register allocation a second 4627 time if the tentative V# is eliminated. This is more efficient and 4628 avoids the problem that the second register allocation may perform 4629 spilling which will fail as there is no longer a scratch V#. 4630 4631When the kernel prolog code is being emitted it is known whether the scratch V# 4632described above is actually used. If it is, the prolog code must set it up by 4633copying the Private Segment Buffer to the scratch V# registers and then adding 4634the Private Segment Wavefront Offset to the queue base address in the V#. The 4635result is a V# with a base address pointing to the beginning of the wavefront 4636scratch backing memory. 4637 4638The Private Segment Buffer is always requested, but the Private Segment 4639Wavefront Offset is only requested if it is used (see 4640:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4641 4642.. _amdgpu-amdhsa-memory-model: 4643 4644Memory Model 4645~~~~~~~~~~~~ 4646 4647This section describes the mapping of the LLVM memory model onto AMDGPU machine 4648code (see :ref:`memmodel`). 4649 4650The AMDGPU backend supports the memory synchronization scopes specified in 4651:ref:`amdgpu-memory-scopes`. 4652 4653The code sequences used to implement the memory model specify the order of 4654instructions that a single thread must execute. The ``s_waitcnt`` and cache 4655management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4656to other memory instructions executed by the same thread. This allows them to be 4657moved earlier or later which can allow them to be combined with other instances 4658of the same instruction, or hoisted/sunk out of loops to improve performance. 4659Only the instructions related to the memory model are given; additional 4660``s_waitcnt`` instructions are required to ensure registers are defined before 4661being used. These may be able to be combined with the memory model ``s_waitcnt`` 4662instructions as described above. 4663 4664The AMDGPU backend supports the following memory models: 4665 4666 HSA Memory Model [HSA]_ 4667 The HSA memory model uses a single happens-before relation for all address 4668 spaces (see :ref:`amdgpu-address-spaces`). 4669 OpenCL Memory Model [OpenCL]_ 4670 The OpenCL memory model which has separate happens-before relations for the 4671 global and local address spaces. Only a fence specifying both global and 4672 local address space, and seq_cst instructions join the relationships. Since 4673 the LLVM ``memfence`` instruction does not allow an address space to be 4674 specified the OpenCL fence has to conservatively assume both local and 4675 global address space was specified. However, optimizations can often be 4676 done to eliminate the additional ``s_waitcnt`` instructions when there are 4677 no intervening memory instructions which access the corresponding address 4678 space. The code sequences in the table indicate what can be omitted for the 4679 OpenCL memory. The target triple environment is used to determine if the 4680 source language is OpenCL (see :ref:`amdgpu-opencl`). 4681 4682``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4683operations. 4684 4685``buffer/global/flat_load/store/atomic`` instructions to global memory are 4686termed vector memory operations. 4687 4688Private address space uses ``buffer_load/store`` using the scratch V# 4689(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4690is accessing the memory, atomic memory orderings are not meaningful, and all 4691accesses are treated as non-atomic. 4692 4693Constant address space uses ``buffer/global_load`` instructions (or equivalent 4694scalar memory instructions). Since the constant address space contents do not 4695change during the execution of a kernel dispatch it is not legal to perform 4696stores, and atomic memory orderings are not meaningful, and all accesses are 4697treated as non-atomic. 4698 4699A memory synchronization scope wider than work-group is not meaningful for the 4700group (LDS) address space and is treated as work-group. 4701 4702The memory model does not support the region address space which is treated as 4703non-atomic. 4704 4705Acquire memory ordering is not meaningful on store atomic instructions and is 4706treated as non-atomic. 4707 4708Release memory ordering is not meaningful on load atomic instructions and is 4709treated a non-atomic. 4710 4711Acquire-release memory ordering is not meaningful on load or store atomic 4712instructions and is treated as acquire and release respectively. 4713 4714The memory order also adds the single thread optimization constraints defined in 4715table 4716:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4717 4718 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4719 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4720 4721 ============ ============================================================== 4722 LLVM Memory Optimization Constraints 4723 Ordering 4724 ============ ============================================================== 4725 unordered *none* 4726 monotonic *none* 4727 acquire - If a load atomic/atomicrmw then no following load/load 4728 atomic/store/store atomic/atomicrmw/fence instruction can be 4729 moved before the acquire. 4730 - If a fence then same as load atomic, plus no preceding 4731 associated fence-paired-atomic can be moved after the fence. 4732 release - If a store atomic/atomicrmw then no preceding load/load 4733 atomic/store/store atomic/atomicrmw/fence instruction can be 4734 moved after the release. 4735 - If a fence then same as store atomic, plus no following 4736 associated fence-paired-atomic can be moved before the 4737 fence. 4738 acq_rel Same constraints as both acquire and release. 4739 seq_cst - If a load atomic then same constraints as acquire, plus no 4740 preceding sequentially consistent load atomic/store 4741 atomic/atomicrmw/fence instruction can be moved after the 4742 seq_cst. 4743 - If a store atomic then the same constraints as release, plus 4744 no following sequentially consistent load atomic/store 4745 atomic/atomicrmw/fence instruction can be moved before the 4746 seq_cst. 4747 - If an atomicrmw/fence then same constraints as acq_rel. 4748 ============ ============================================================== 4749 4750The code sequences used to implement the memory model are defined in the 4751following sections: 4752 4753* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4754* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4755* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4756 4757.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4758 4759Memory Model GFX6-GFX9 4760++++++++++++++++++++++ 4761 4762For GFX6-GFX9: 4763 4764* Each agent has multiple shader arrays (SA). 4765* Each SA has multiple compute units (CU). 4766* Each CU has multiple SIMDs that execute wavefronts. 4767* The wavefronts for a single work-group are executed in the same CU but may be 4768 executed by different SIMDs. 4769* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4770 executing on it. 4771* All LDS operations of a CU are performed as wavefront wide operations in a 4772 global order and involve no caching. Completion is reported to a wavefront in 4773 execution order. 4774* The LDS memory has multiple request queues shared by the SIMDs of a 4775 CU. Therefore, the LDS operations performed by different wavefronts of a 4776 work-group can be reordered relative to each other, which can result in 4777 reordering the visibility of vector memory operations with respect to LDS 4778 operations of other wavefronts in the same work-group. A ``s_waitcnt 4779 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4780 vector memory operations between wavefronts of a work-group, but not between 4781 operations performed by the same wavefront. 4782* The vector memory operations are performed as wavefront wide operations and 4783 completion is reported to a wavefront in execution order. The exception is 4784 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4785 vector memory order if they access LDS memory, and out of LDS operation order 4786 if they access global memory. 4787* The vector memory operations access a single vector L1 cache shared by all 4788 SIMDs a CU. Therefore, no special action is required for coherence between the 4789 lanes of a single wavefront, or for coherence between wavefronts in the same 4790 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4791 wavefronts executing in different work-groups as they may be executing on 4792 different CUs. 4793* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4794 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4795 scalar operations are used in a restricted way so do not impact the memory 4796 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4797* The vector and scalar memory operations use an L2 cache shared by all CUs on 4798 the same agent. 4799* The L2 cache has independent channels to service disjoint ranges of virtual 4800 addresses. 4801* Each CU has a separate request queue per channel. Therefore, the vector and 4802 scalar memory operations performed by wavefronts executing in different 4803 work-groups (which may be executing on different CUs) of an agent can be 4804 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4805 ensure synchronization between vector memory operations of different CUs. It 4806 ensures a previous vector memory operation has completed before executing a 4807 subsequent vector memory or LDS operation and so can be used to meet the 4808 requirements of acquire and release. 4809* The L2 cache can be kept coherent with other agents on some targets, or ranges 4810 of virtual addresses can be set up to bypass it to ensure system coherence. 4811 4812Scalar memory operations are only used to access memory that is proven to not 4813change during the execution of the kernel dispatch. This includes constant 4814address space and global address space for program scope ``const`` variables. 4815Therefore, the kernel machine code does not have to maintain the scalar cache to 4816ensure it is coherent with the vector caches. The scalar and vector caches are 4817invalidated between kernel dispatches by CP since constant address space data 4818may change between kernel dispatch executions. See 4819:ref:`amdgpu-amdhsa-memory-spaces`. 4820 4821The one exception is if scalar writes are used to spill SGPR registers. In this 4822case the AMDGPU backend ensures the memory location used to spill is never 4823accessed by vector memory operations at the same time. If scalar writes are used 4824then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4825return since the locations may be used for vector memory instructions by a 4826future wavefront that uses the same scratch area, or a function call that 4827creates a frame at the same address, respectively. There is no need for a 4828``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4829 4830For kernarg backing memory: 4831 4832* CP invalidates the L1 cache at the start of each kernel dispatch. 4833* On dGPU the kernarg backing memory is allocated in host memory accessed as 4834 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4835 causes it to be treated as non-volatile and so is not invalidated by 4836 ``*_vol``. 4837* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4838 and so the L2 cache will be coherent with the CPU and other agents. 4839 4840Scratch backing memory (which is used for the private address space) is accessed 4841with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4842only accessed by a single thread, and is always write-before-read, there is 4843never a need to invalidate these entries from the L1 cache. Hence all cache 4844invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4845 4846The code sequences used to implement the memory model for GFX6-GFX9 are defined 4847in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4848 4849 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4850 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4851 4852 ============ ============ ============== ========== ================================ 4853 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4854 Ordering Sync Scope Address GFX6-GFX9 4855 Space 4856 ============ ============ ============== ========== ================================ 4857 **Non-Atomic** 4858 ------------------------------------------------------------------------------------ 4859 load *none* *none* - global - !volatile & !nontemporal 4860 - generic 4861 - private 1. buffer/global/flat_load 4862 - constant 4863 - !volatile & nontemporal 4864 4865 1. buffer/global/flat_load 4866 glc=1 slc=1 4867 4868 - volatile 4869 4870 1. buffer/global/flat_load 4871 glc=1 4872 2. s_waitcnt vmcnt(0) 4873 4874 - Must happen before 4875 any following volatile 4876 global/generic 4877 load/store. 4878 - Ensures that 4879 volatile 4880 operations to 4881 different 4882 addresses will not 4883 be reordered by 4884 hardware. 4885 4886 load *none* *none* - local 1. ds_load 4887 store *none* *none* - global - !volatile & !nontemporal 4888 - generic 4889 - private 1. buffer/global/flat_store 4890 - constant 4891 - !volatile & nontemporal 4892 4893 1. buffer/global/flat_store 4894 glc=1 slc=1 4895 4896 - volatile 4897 4898 1. buffer/global/flat_store 4899 2. s_waitcnt vmcnt(0) 4900 4901 - Must happen before 4902 any following volatile 4903 global/generic 4904 load/store. 4905 - Ensures that 4906 volatile 4907 operations to 4908 different 4909 addresses will not 4910 be reordered by 4911 hardware. 4912 4913 store *none* *none* - local 1. ds_store 4914 **Unordered Atomic** 4915 ------------------------------------------------------------------------------------ 4916 load atomic unordered *any* *any* *Same as non-atomic*. 4917 store atomic unordered *any* *any* *Same as non-atomic*. 4918 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4919 **Monotonic Atomic** 4920 ------------------------------------------------------------------------------------ 4921 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4922 - wavefront - local 4923 - workgroup - generic 4924 load atomic monotonic - agent - global 1. buffer/global/flat_load 4925 - system - generic glc=1 4926 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4927 - wavefront - generic 4928 - workgroup 4929 - agent 4930 - system 4931 store atomic monotonic - singlethread - local 1. ds_store 4932 - wavefront 4933 - workgroup 4934 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4935 - wavefront - generic 4936 - workgroup 4937 - agent 4938 - system 4939 atomicrmw monotonic - singlethread - local 1. ds_atomic 4940 - wavefront 4941 - workgroup 4942 **Acquire Atomic** 4943 ------------------------------------------------------------------------------------ 4944 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4945 - wavefront - local 4946 - generic 4947 load atomic acquire - workgroup - global 1. buffer/global_load 4948 load atomic acquire - workgroup - local 1. ds/flat_load 4949 - generic 2. s_waitcnt lgkmcnt(0) 4950 4951 - If OpenCL, omit. 4952 - Must happen before 4953 any following 4954 global/generic 4955 load/load 4956 atomic/store/store 4957 atomic/atomicrmw. 4958 - Ensures any 4959 following global 4960 data read is no 4961 older than a local load 4962 atomic value being 4963 acquired. 4964 4965 load atomic acquire - agent - global 1. buffer/global_load 4966 - system glc=1 4967 2. s_waitcnt vmcnt(0) 4968 4969 - Must happen before 4970 following 4971 buffer_wbinvl1_vol. 4972 - Ensures the load 4973 has completed 4974 before invalidating 4975 the cache. 4976 4977 3. buffer_wbinvl1_vol 4978 4979 - Must happen before 4980 any following 4981 global/generic 4982 load/load 4983 atomic/atomicrmw. 4984 - Ensures that 4985 following 4986 loads will not see 4987 stale global data. 4988 4989 load atomic acquire - agent - generic 1. flat_load glc=1 4990 - system 2. s_waitcnt vmcnt(0) & 4991 lgkmcnt(0) 4992 4993 - If OpenCL omit 4994 lgkmcnt(0). 4995 - Must happen before 4996 following 4997 buffer_wbinvl1_vol. 4998 - Ensures the flat_load 4999 has completed 5000 before invalidating 5001 the cache. 5002 5003 3. buffer_wbinvl1_vol 5004 5005 - Must happen before 5006 any following 5007 global/generic 5008 load/load 5009 atomic/atomicrmw. 5010 - Ensures that 5011 following loads 5012 will not see stale 5013 global data. 5014 5015 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5016 - wavefront - local 5017 - generic 5018 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5019 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5020 - generic 2. s_waitcnt lgkmcnt(0) 5021 5022 - If OpenCL, omit. 5023 - Must happen before 5024 any following 5025 global/generic 5026 load/load 5027 atomic/store/store 5028 atomic/atomicrmw. 5029 - Ensures any 5030 following global 5031 data read is no 5032 older than a local 5033 atomicrmw value 5034 being acquired. 5035 5036 atomicrmw acquire - agent - global 1. buffer/global_atomic 5037 - system 2. s_waitcnt vmcnt(0) 5038 5039 - Must happen before 5040 following 5041 buffer_wbinvl1_vol. 5042 - Ensures the 5043 atomicrmw has 5044 completed before 5045 invalidating the 5046 cache. 5047 5048 3. buffer_wbinvl1_vol 5049 5050 - Must happen before 5051 any following 5052 global/generic 5053 load/load 5054 atomic/atomicrmw. 5055 - Ensures that 5056 following loads 5057 will not see stale 5058 global data. 5059 5060 atomicrmw acquire - agent - generic 1. flat_atomic 5061 - system 2. s_waitcnt vmcnt(0) & 5062 lgkmcnt(0) 5063 5064 - If OpenCL, omit 5065 lgkmcnt(0). 5066 - Must happen before 5067 following 5068 buffer_wbinvl1_vol. 5069 - Ensures the 5070 atomicrmw has 5071 completed before 5072 invalidating the 5073 cache. 5074 5075 3. buffer_wbinvl1_vol 5076 5077 - Must happen before 5078 any following 5079 global/generic 5080 load/load 5081 atomic/atomicrmw. 5082 - Ensures that 5083 following loads 5084 will not see stale 5085 global data. 5086 5087 fence acquire - singlethread *none* *none* 5088 - wavefront 5089 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5090 5091 - If OpenCL and 5092 address space is 5093 not generic, omit. 5094 - However, since LLVM 5095 currently has no 5096 address space on 5097 the fence need to 5098 conservatively 5099 always generate. If 5100 fence had an 5101 address space then 5102 set to address 5103 space of OpenCL 5104 fence flag, or to 5105 generic if both 5106 local and global 5107 flags are 5108 specified. 5109 - Must happen after 5110 any preceding 5111 local/generic load 5112 atomic/atomicrmw 5113 with an equal or 5114 wider sync scope 5115 and memory ordering 5116 stronger than 5117 unordered (this is 5118 termed the 5119 fence-paired-atomic). 5120 - Must happen before 5121 any following 5122 global/generic 5123 load/load 5124 atomic/store/store 5125 atomic/atomicrmw. 5126 - Ensures any 5127 following global 5128 data read is no 5129 older than the 5130 value read by the 5131 fence-paired-atomic. 5132 5133 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5134 - system vmcnt(0) 5135 5136 - If OpenCL and 5137 address space is 5138 not generic, omit 5139 lgkmcnt(0). 5140 - However, since LLVM 5141 currently has no 5142 address space on 5143 the fence need to 5144 conservatively 5145 always generate 5146 (see comment for 5147 previous fence). 5148 - Could be split into 5149 separate s_waitcnt 5150 vmcnt(0) and 5151 s_waitcnt 5152 lgkmcnt(0) to allow 5153 them to be 5154 independently moved 5155 according to the 5156 following rules. 5157 - s_waitcnt vmcnt(0) 5158 must happen after 5159 any preceding 5160 global/generic load 5161 atomic/atomicrmw 5162 with an equal or 5163 wider sync scope 5164 and memory ordering 5165 stronger than 5166 unordered (this is 5167 termed the 5168 fence-paired-atomic). 5169 - s_waitcnt lgkmcnt(0) 5170 must happen after 5171 any preceding 5172 local/generic load 5173 atomic/atomicrmw 5174 with an equal or 5175 wider sync scope 5176 and memory ordering 5177 stronger than 5178 unordered (this is 5179 termed the 5180 fence-paired-atomic). 5181 - Must happen before 5182 the following 5183 buffer_wbinvl1_vol. 5184 - Ensures that the 5185 fence-paired atomic 5186 has completed 5187 before invalidating 5188 the 5189 cache. Therefore 5190 any following 5191 locations read must 5192 be no older than 5193 the value read by 5194 the 5195 fence-paired-atomic. 5196 5197 2. buffer_wbinvl1_vol 5198 5199 - Must happen before any 5200 following global/generic 5201 load/load 5202 atomic/store/store 5203 atomic/atomicrmw. 5204 - Ensures that 5205 following loads 5206 will not see stale 5207 global data. 5208 5209 **Release Atomic** 5210 ------------------------------------------------------------------------------------ 5211 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5212 - wavefront - local 5213 - generic 5214 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5215 - generic 5216 - If OpenCL, omit. 5217 - Must happen after 5218 any preceding 5219 local/generic 5220 load/store/load 5221 atomic/store 5222 atomic/atomicrmw. 5223 - Must happen before 5224 the following 5225 store. 5226 - Ensures that all 5227 memory operations 5228 to local have 5229 completed before 5230 performing the 5231 store that is being 5232 released. 5233 5234 2. buffer/global/flat_store 5235 store atomic release - workgroup - local 1. ds_store 5236 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5237 - system - generic vmcnt(0) 5238 5239 - If OpenCL and 5240 address space is 5241 not generic, omit 5242 lgkmcnt(0). 5243 - Could be split into 5244 separate s_waitcnt 5245 vmcnt(0) and 5246 s_waitcnt 5247 lgkmcnt(0) to allow 5248 them to be 5249 independently moved 5250 according to the 5251 following rules. 5252 - s_waitcnt vmcnt(0) 5253 must happen after 5254 any preceding 5255 global/generic 5256 load/store/load 5257 atomic/store 5258 atomic/atomicrmw. 5259 - s_waitcnt lgkmcnt(0) 5260 must happen after 5261 any preceding 5262 local/generic 5263 load/store/load 5264 atomic/store 5265 atomic/atomicrmw. 5266 - Must happen before 5267 the following 5268 store. 5269 - Ensures that all 5270 memory operations 5271 to memory have 5272 completed before 5273 performing the 5274 store that is being 5275 released. 5276 5277 2. buffer/global/flat_store 5278 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5279 - wavefront - local 5280 - generic 5281 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5282 - generic 5283 - If OpenCL, omit. 5284 - Must happen after 5285 any preceding 5286 local/generic 5287 load/store/load 5288 atomic/store 5289 atomic/atomicrmw. 5290 - Must happen before 5291 the following 5292 atomicrmw. 5293 - Ensures that all 5294 memory operations 5295 to local have 5296 completed before 5297 performing the 5298 atomicrmw that is 5299 being released. 5300 5301 2. buffer/global/flat_atomic 5302 atomicrmw release - workgroup - local 1. ds_atomic 5303 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5304 - system - generic vmcnt(0) 5305 5306 - If OpenCL, omit 5307 lgkmcnt(0). 5308 - Could be split into 5309 separate s_waitcnt 5310 vmcnt(0) and 5311 s_waitcnt 5312 lgkmcnt(0) to allow 5313 them to be 5314 independently moved 5315 according to the 5316 following rules. 5317 - s_waitcnt vmcnt(0) 5318 must happen after 5319 any preceding 5320 global/generic 5321 load/store/load 5322 atomic/store 5323 atomic/atomicrmw. 5324 - s_waitcnt lgkmcnt(0) 5325 must happen after 5326 any preceding 5327 local/generic 5328 load/store/load 5329 atomic/store 5330 atomic/atomicrmw. 5331 - Must happen before 5332 the following 5333 atomicrmw. 5334 - Ensures that all 5335 memory operations 5336 to global and local 5337 have completed 5338 before performing 5339 the atomicrmw that 5340 is being released. 5341 5342 2. buffer/global/flat_atomic 5343 fence release - singlethread *none* *none* 5344 - wavefront 5345 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5346 5347 - If OpenCL and 5348 address space is 5349 not generic, omit. 5350 - However, since LLVM 5351 currently has no 5352 address space on 5353 the fence need to 5354 conservatively 5355 always generate. If 5356 fence had an 5357 address space then 5358 set to address 5359 space of OpenCL 5360 fence flag, or to 5361 generic if both 5362 local and global 5363 flags are 5364 specified. 5365 - Must happen after 5366 any preceding 5367 local/generic 5368 load/load 5369 atomic/store/store 5370 atomic/atomicrmw. 5371 - Must happen before 5372 any following store 5373 atomic/atomicrmw 5374 with an equal or 5375 wider sync scope 5376 and memory ordering 5377 stronger than 5378 unordered (this is 5379 termed the 5380 fence-paired-atomic). 5381 - Ensures that all 5382 memory operations 5383 to local have 5384 completed before 5385 performing the 5386 following 5387 fence-paired-atomic. 5388 5389 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5390 - system vmcnt(0) 5391 5392 - If OpenCL and 5393 address space is 5394 not generic, omit 5395 lgkmcnt(0). 5396 - If OpenCL and 5397 address space is 5398 local, omit 5399 vmcnt(0). 5400 - However, since LLVM 5401 currently has no 5402 address space on 5403 the fence need to 5404 conservatively 5405 always generate. If 5406 fence had an 5407 address space then 5408 set to address 5409 space of OpenCL 5410 fence flag, or to 5411 generic if both 5412 local and global 5413 flags are 5414 specified. 5415 - Could be split into 5416 separate s_waitcnt 5417 vmcnt(0) and 5418 s_waitcnt 5419 lgkmcnt(0) to allow 5420 them to be 5421 independently moved 5422 according to the 5423 following rules. 5424 - s_waitcnt vmcnt(0) 5425 must happen after 5426 any preceding 5427 global/generic 5428 load/store/load 5429 atomic/store 5430 atomic/atomicrmw. 5431 - s_waitcnt lgkmcnt(0) 5432 must happen after 5433 any preceding 5434 local/generic 5435 load/store/load 5436 atomic/store 5437 atomic/atomicrmw. 5438 - Must happen before 5439 any following store 5440 atomic/atomicrmw 5441 with an equal or 5442 wider sync scope 5443 and memory ordering 5444 stronger than 5445 unordered (this is 5446 termed the 5447 fence-paired-atomic). 5448 - Ensures that all 5449 memory operations 5450 have 5451 completed before 5452 performing the 5453 following 5454 fence-paired-atomic. 5455 5456 **Acquire-Release Atomic** 5457 ------------------------------------------------------------------------------------ 5458 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5459 - wavefront - local 5460 - generic 5461 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5462 5463 - If OpenCL, omit. 5464 - Must happen after 5465 any preceding 5466 local/generic 5467 load/store/load 5468 atomic/store 5469 atomic/atomicrmw. 5470 - Must happen before 5471 the following 5472 atomicrmw. 5473 - Ensures that all 5474 memory operations 5475 to local have 5476 completed before 5477 performing the 5478 atomicrmw that is 5479 being released. 5480 5481 2. buffer/global_atomic 5482 5483 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5484 2. s_waitcnt lgkmcnt(0) 5485 5486 - If OpenCL, omit. 5487 - Must happen before 5488 any following 5489 global/generic 5490 load/load 5491 atomic/store/store 5492 atomic/atomicrmw. 5493 - Ensures any 5494 following global 5495 data read is no 5496 older than the local load 5497 atomic value being 5498 acquired. 5499 5500 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5501 5502 - If OpenCL, omit. 5503 - Must happen after 5504 any preceding 5505 local/generic 5506 load/store/load 5507 atomic/store 5508 atomic/atomicrmw. 5509 - Must happen before 5510 the following 5511 atomicrmw. 5512 - Ensures that all 5513 memory operations 5514 to local have 5515 completed before 5516 performing the 5517 atomicrmw that is 5518 being released. 5519 5520 2. flat_atomic 5521 3. s_waitcnt lgkmcnt(0) 5522 5523 - If OpenCL, omit. 5524 - Must happen before 5525 any following 5526 global/generic 5527 load/load 5528 atomic/store/store 5529 atomic/atomicrmw. 5530 - Ensures any 5531 following global 5532 data read is no 5533 older than a local load 5534 atomic value being 5535 acquired. 5536 5537 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5538 - system vmcnt(0) 5539 5540 - If OpenCL, omit 5541 lgkmcnt(0). 5542 - Could be split into 5543 separate s_waitcnt 5544 vmcnt(0) and 5545 s_waitcnt 5546 lgkmcnt(0) to allow 5547 them to be 5548 independently moved 5549 according to the 5550 following rules. 5551 - s_waitcnt vmcnt(0) 5552 must happen after 5553 any preceding 5554 global/generic 5555 load/store/load 5556 atomic/store 5557 atomic/atomicrmw. 5558 - s_waitcnt lgkmcnt(0) 5559 must happen after 5560 any preceding 5561 local/generic 5562 load/store/load 5563 atomic/store 5564 atomic/atomicrmw. 5565 - Must happen before 5566 the following 5567 atomicrmw. 5568 - Ensures that all 5569 memory operations 5570 to global have 5571 completed before 5572 performing the 5573 atomicrmw that is 5574 being released. 5575 5576 2. buffer/global_atomic 5577 3. s_waitcnt vmcnt(0) 5578 5579 - Must happen before 5580 following 5581 buffer_wbinvl1_vol. 5582 - Ensures the 5583 atomicrmw has 5584 completed before 5585 invalidating the 5586 cache. 5587 5588 4. buffer_wbinvl1_vol 5589 5590 - Must happen before 5591 any following 5592 global/generic 5593 load/load 5594 atomic/atomicrmw. 5595 - Ensures that 5596 following loads 5597 will not see stale 5598 global data. 5599 5600 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5601 - system vmcnt(0) 5602 5603 - If OpenCL, omit 5604 lgkmcnt(0). 5605 - Could be split into 5606 separate s_waitcnt 5607 vmcnt(0) and 5608 s_waitcnt 5609 lgkmcnt(0) to allow 5610 them to be 5611 independently moved 5612 according to the 5613 following rules. 5614 - s_waitcnt vmcnt(0) 5615 must happen after 5616 any preceding 5617 global/generic 5618 load/store/load 5619 atomic/store 5620 atomic/atomicrmw. 5621 - s_waitcnt lgkmcnt(0) 5622 must happen after 5623 any preceding 5624 local/generic 5625 load/store/load 5626 atomic/store 5627 atomic/atomicrmw. 5628 - Must happen before 5629 the following 5630 atomicrmw. 5631 - Ensures that all 5632 memory operations 5633 to global have 5634 completed before 5635 performing the 5636 atomicrmw that is 5637 being released. 5638 5639 2. flat_atomic 5640 3. s_waitcnt vmcnt(0) & 5641 lgkmcnt(0) 5642 5643 - If OpenCL, omit 5644 lgkmcnt(0). 5645 - Must happen before 5646 following 5647 buffer_wbinvl1_vol. 5648 - Ensures the 5649 atomicrmw has 5650 completed before 5651 invalidating the 5652 cache. 5653 5654 4. buffer_wbinvl1_vol 5655 5656 - Must happen before 5657 any following 5658 global/generic 5659 load/load 5660 atomic/atomicrmw. 5661 - Ensures that 5662 following loads 5663 will not see stale 5664 global data. 5665 5666 fence acq_rel - singlethread *none* *none* 5667 - wavefront 5668 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5669 5670 - If OpenCL and 5671 address space is 5672 not generic, omit. 5673 - However, 5674 since LLVM 5675 currently has no 5676 address space on 5677 the fence need to 5678 conservatively 5679 always generate 5680 (see comment for 5681 previous fence). 5682 - Must happen after 5683 any preceding 5684 local/generic 5685 load/load 5686 atomic/store/store 5687 atomic/atomicrmw. 5688 - Must happen before 5689 any following 5690 global/generic 5691 load/load 5692 atomic/store/store 5693 atomic/atomicrmw. 5694 - Ensures that all 5695 memory operations 5696 to local have 5697 completed before 5698 performing any 5699 following global 5700 memory operations. 5701 - Ensures that the 5702 preceding 5703 local/generic load 5704 atomic/atomicrmw 5705 with an equal or 5706 wider sync scope 5707 and memory ordering 5708 stronger than 5709 unordered (this is 5710 termed the 5711 acquire-fence-paired-atomic) 5712 has completed 5713 before following 5714 global memory 5715 operations. This 5716 satisfies the 5717 requirements of 5718 acquire. 5719 - Ensures that all 5720 previous memory 5721 operations have 5722 completed before a 5723 following 5724 local/generic store 5725 atomic/atomicrmw 5726 with an equal or 5727 wider sync scope 5728 and memory ordering 5729 stronger than 5730 unordered (this is 5731 termed the 5732 release-fence-paired-atomic). 5733 This satisfies the 5734 requirements of 5735 release. 5736 5737 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5738 - system vmcnt(0) 5739 5740 - If OpenCL and 5741 address space is 5742 not generic, omit 5743 lgkmcnt(0). 5744 - However, since LLVM 5745 currently has no 5746 address space on 5747 the fence need to 5748 conservatively 5749 always generate 5750 (see comment for 5751 previous fence). 5752 - Could be split into 5753 separate s_waitcnt 5754 vmcnt(0) and 5755 s_waitcnt 5756 lgkmcnt(0) to allow 5757 them to be 5758 independently moved 5759 according to the 5760 following rules. 5761 - s_waitcnt vmcnt(0) 5762 must happen after 5763 any preceding 5764 global/generic 5765 load/store/load 5766 atomic/store 5767 atomic/atomicrmw. 5768 - s_waitcnt lgkmcnt(0) 5769 must happen after 5770 any preceding 5771 local/generic 5772 load/store/load 5773 atomic/store 5774 atomic/atomicrmw. 5775 - Must happen before 5776 the following 5777 buffer_wbinvl1_vol. 5778 - Ensures that the 5779 preceding 5780 global/local/generic 5781 load 5782 atomic/atomicrmw 5783 with an equal or 5784 wider sync scope 5785 and memory ordering 5786 stronger than 5787 unordered (this is 5788 termed the 5789 acquire-fence-paired-atomic) 5790 has completed 5791 before invalidating 5792 the cache. This 5793 satisfies the 5794 requirements of 5795 acquire. 5796 - Ensures that all 5797 previous memory 5798 operations have 5799 completed before a 5800 following 5801 global/local/generic 5802 store 5803 atomic/atomicrmw 5804 with an equal or 5805 wider sync scope 5806 and memory ordering 5807 stronger than 5808 unordered (this is 5809 termed the 5810 release-fence-paired-atomic). 5811 This satisfies the 5812 requirements of 5813 release. 5814 5815 2. buffer_wbinvl1_vol 5816 5817 - Must happen before 5818 any following 5819 global/generic 5820 load/load 5821 atomic/store/store 5822 atomic/atomicrmw. 5823 - Ensures that 5824 following loads 5825 will not see stale 5826 global data. This 5827 satisfies the 5828 requirements of 5829 acquire. 5830 5831 **Sequential Consistent Atomic** 5832 ------------------------------------------------------------------------------------ 5833 load atomic seq_cst - singlethread - global *Same as corresponding 5834 - wavefront - local load atomic acquire, 5835 - generic except must generated 5836 all instructions even 5837 for OpenCL.* 5838 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5839 - generic 5840 5841 - Must 5842 happen after 5843 preceding 5844 local/generic load 5845 atomic/store 5846 atomic/atomicrmw 5847 with memory 5848 ordering of seq_cst 5849 and with equal or 5850 wider sync scope. 5851 (Note that seq_cst 5852 fences have their 5853 own s_waitcnt 5854 lgkmcnt(0) and so do 5855 not need to be 5856 considered.) 5857 - Ensures any 5858 preceding 5859 sequential 5860 consistent local 5861 memory instructions 5862 have completed 5863 before executing 5864 this sequentially 5865 consistent 5866 instruction. This 5867 prevents reordering 5868 a seq_cst store 5869 followed by a 5870 seq_cst load. (Note 5871 that seq_cst is 5872 stronger than 5873 acquire/release as 5874 the reordering of 5875 load acquire 5876 followed by a store 5877 release is 5878 prevented by the 5879 s_waitcnt of 5880 the release, but 5881 there is nothing 5882 preventing a store 5883 release followed by 5884 load acquire from 5885 completing out of 5886 order. The s_waitcnt 5887 could be placed after 5888 seq_store or before 5889 the seq_load. We 5890 choose the load to 5891 make the s_waitcnt be 5892 as late as possible 5893 so that the store 5894 may have already 5895 completed.) 5896 5897 2. *Following 5898 instructions same as 5899 corresponding load 5900 atomic acquire, 5901 except must generated 5902 all instructions even 5903 for OpenCL.* 5904 load atomic seq_cst - workgroup - local *Same as corresponding 5905 load atomic acquire, 5906 except must generated 5907 all instructions even 5908 for OpenCL.* 5909 5910 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5911 - system - generic vmcnt(0) 5912 5913 - Could be split into 5914 separate s_waitcnt 5915 vmcnt(0) 5916 and s_waitcnt 5917 lgkmcnt(0) to allow 5918 them to be 5919 independently moved 5920 according to the 5921 following rules. 5922 - s_waitcnt lgkmcnt(0) 5923 must happen after 5924 preceding 5925 global/generic load 5926 atomic/store 5927 atomic/atomicrmw 5928 with memory 5929 ordering of seq_cst 5930 and with equal or 5931 wider sync scope. 5932 (Note that seq_cst 5933 fences have their 5934 own s_waitcnt 5935 lgkmcnt(0) and so do 5936 not need to be 5937 considered.) 5938 - s_waitcnt vmcnt(0) 5939 must happen after 5940 preceding 5941 global/generic load 5942 atomic/store 5943 atomic/atomicrmw 5944 with memory 5945 ordering of seq_cst 5946 and with equal or 5947 wider sync scope. 5948 (Note that seq_cst 5949 fences have their 5950 own s_waitcnt 5951 vmcnt(0) and so do 5952 not need to be 5953 considered.) 5954 - Ensures any 5955 preceding 5956 sequential 5957 consistent global 5958 memory instructions 5959 have completed 5960 before executing 5961 this sequentially 5962 consistent 5963 instruction. This 5964 prevents reordering 5965 a seq_cst store 5966 followed by a 5967 seq_cst load. (Note 5968 that seq_cst is 5969 stronger than 5970 acquire/release as 5971 the reordering of 5972 load acquire 5973 followed by a store 5974 release is 5975 prevented by the 5976 s_waitcnt of 5977 the release, but 5978 there is nothing 5979 preventing a store 5980 release followed by 5981 load acquire from 5982 completing out of 5983 order. The s_waitcnt 5984 could be placed after 5985 seq_store or before 5986 the seq_load. We 5987 choose the load to 5988 make the s_waitcnt be 5989 as late as possible 5990 so that the store 5991 may have already 5992 completed.) 5993 5994 2. *Following 5995 instructions same as 5996 corresponding load 5997 atomic acquire, 5998 except must generated 5999 all instructions even 6000 for OpenCL.* 6001 store atomic seq_cst - singlethread - global *Same as corresponding 6002 - wavefront - local store atomic release, 6003 - workgroup - generic except must generated 6004 - agent all instructions even 6005 - system for OpenCL.* 6006 atomicrmw seq_cst - singlethread - global *Same as corresponding 6007 - wavefront - local atomicrmw acq_rel, 6008 - workgroup - generic except must generated 6009 - agent all instructions even 6010 - system for OpenCL.* 6011 fence seq_cst - singlethread *none* *Same as corresponding 6012 - wavefront fence acq_rel, 6013 - workgroup except must generated 6014 - agent all instructions even 6015 - system for OpenCL.* 6016 ============ ============ ============== ========== ================================ 6017 6018.. _amdgpu-amdhsa-memory-model-gfx90a: 6019 6020Memory Model GFX90A 6021+++++++++++++++++++ 6022 6023For GFX90A: 6024 6025* Each agent has multiple shader arrays (SA). 6026* Each SA has multiple compute units (CU). 6027* Each CU has multiple SIMDs that execute wavefronts. 6028* The wavefronts for a single work-group are executed in the same CU but may be 6029 executed by different SIMDs. The exception is when in tgsplit execution mode 6030 when the wavefronts may be executed by different SIMDs in different CUs. 6031* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6032 executing on it. The exception is when in tgsplit execution mode when no LDS 6033 is allocated as wavefronts of the same work-group can be in different CUs. 6034* All LDS operations of a CU are performed as wavefront wide operations in a 6035 global order and involve no caching. Completion is reported to a wavefront in 6036 execution order. 6037* The LDS memory has multiple request queues shared by the SIMDs of a 6038 CU. Therefore, the LDS operations performed by different wavefronts of a 6039 work-group can be reordered relative to each other, which can result in 6040 reordering the visibility of vector memory operations with respect to LDS 6041 operations of other wavefronts in the same work-group. A ``s_waitcnt 6042 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6043 vector memory operations between wavefronts of a work-group, but not between 6044 operations performed by the same wavefront. 6045* The vector memory operations are performed as wavefront wide operations and 6046 completion is reported to a wavefront in execution order. The exception is 6047 that ``flat_load/store/atomic`` instructions can report out of vector memory 6048 order if they access LDS memory, and out of LDS operation order if they access 6049 global memory. 6050* The vector memory operations access a single vector L1 cache shared by all 6051 SIMDs a CU. Therefore: 6052 6053 * No special action is required for coherence between the lanes of a single 6054 wavefront. 6055 6056 * No special action is required for coherence between wavefronts in the same 6057 work-group since they execute on the same CU. The exception is when in 6058 tgsplit execution mode as wavefronts of the same work-group can be in 6059 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6060 the following item. 6061 6062 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6063 executing in different work-groups as they may be executing on different 6064 CUs. 6065 6066* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6067 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6068 scalar operations are used in a restricted way so do not impact the memory 6069 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6070* The vector and scalar memory operations use an L2 cache shared by all CUs on 6071 the same agent. 6072 6073 * The L2 cache has independent channels to service disjoint ranges of virtual 6074 addresses. 6075 * Each CU has a separate request queue per channel. Therefore, the vector and 6076 scalar memory operations performed by wavefronts executing in different 6077 work-groups (which may be executing on different CUs), or the same 6078 work-group if executing in tgsplit mode, of an agent can be reordered 6079 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6080 synchronization between vector memory operations of different CUs. It 6081 ensures a previous vector memory operation has completed before executing a 6082 subsequent vector memory or LDS operation and so can be used to meet the 6083 requirements of acquire and release. 6084 * The L2 cache of one agent can be kept coherent with other agents by using 6085 the MTYPE RW (read-write) for memory local to the L2, and MTYPE NC 6086 (non-coherent) with the PTE C-bit set for memory not local to the L2. 6087 6088 * Any local memory cache lines will be automatically invalidated by writes 6089 from CUs associated with other L2 caches, or writes from the CPU, due to 6090 the cache probe caused by the PTE C-bit. 6091 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6092 Subsequent access from the GPU will automatically invalidate or writeback 6093 the CPU cache due to the L2 probe filter. 6094 * Since all work-groups on the same agent share the same L2, no L2 6095 invalidation or writeback is required for coherence. 6096 * To ensure coherence of local memory writes of work-groups in different 6097 agents a ``buffer_wbl2`` is required. It will writeback dirty L2 cache 6098 lines. 6099 * To ensure coherence of local memory reads of work-groups in different 6100 agents a ``buffer_invl2`` is required. It will invalidate non-local L2 6101 cache lines. 6102 6103 * PCIe access from the GPU to the CPU memory can be kept coherent by using the 6104 MTYPE UC (uncached) which bypasses the L2. 6105 6106Scalar memory operations are only used to access memory that is proven to not 6107change during the execution of the kernel dispatch. This includes constant 6108address space and global address space for program scope ``const`` variables. 6109Therefore, the kernel machine code does not have to maintain the scalar cache to 6110ensure it is coherent with the vector caches. The scalar and vector caches are 6111invalidated between kernel dispatches by CP since constant address space data 6112may change between kernel dispatch executions. See 6113:ref:`amdgpu-amdhsa-memory-spaces`. 6114 6115The one exception is if scalar writes are used to spill SGPR registers. In this 6116case the AMDGPU backend ensures the memory location used to spill is never 6117accessed by vector memory operations at the same time. If scalar writes are used 6118then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6119return since the locations may be used for vector memory instructions by a 6120future wavefront that uses the same scratch area, or a function call that 6121creates a frame at the same address, respectively. There is no need for a 6122``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6123 6124For kernarg backing memory: 6125 6126* CP invalidates the L1 cache at the start of each kernel dispatch. 6127* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6128 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6129 cache. This also causes it to be treated as non-volatile and so is not 6130 invalidated by ``*_vol``. 6131* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6132 so the L2 cache will be coherent with the CPU and other agents. 6133 6134Scratch backing memory (which is used for the private address space) is accessed 6135with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6136only accessed by a single thread, and is always write-before-read, there is 6137never a need to invalidate these entries from the L1 cache. Hence all cache 6138invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6139 6140The code sequences used to implement the memory model for GFX90A are defined 6141in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6142 6143 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6144 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6145 6146 ============ ============ ============== ========== ================================ 6147 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6148 Ordering Sync Scope Address GFX90A 6149 Space 6150 ============ ============ ============== ========== ================================ 6151 **Non-Atomic** 6152 ------------------------------------------------------------------------------------ 6153 load *none* *none* - global - !volatile & !nontemporal 6154 - generic 6155 - private 1. buffer/global/flat_load 6156 - constant 6157 - !volatile & nontemporal 6158 6159 1. buffer/global/flat_load 6160 glc=1 slc=1 6161 6162 - volatile 6163 6164 1. buffer/global/flat_load 6165 glc=1 scc=1 6166 2. s_waitcnt vmcnt(0) 6167 6168 - Must happen before 6169 any following volatile 6170 global/generic 6171 load/store. 6172 - Ensures that 6173 volatile 6174 operations to 6175 different 6176 addresses will not 6177 be reordered by 6178 hardware. 6179 6180 load *none* *none* - local 1. ds_load 6181 store *none* *none* - global - !volatile & !nontemporal 6182 - generic 6183 - private 1. buffer/global/flat_store 6184 - constant 6185 - !volatile & nontemporal 6186 6187 1. buffer/global/flat_store 6188 glc=1 slc=1 6189 6190 - volatile 6191 6192 1. buffer/global/flat_store 6193 scc=1 6194 2. s_waitcnt vmcnt(0) 6195 6196 - Must happen before 6197 any following volatile 6198 global/generic 6199 load/store. 6200 - Ensures that 6201 volatile 6202 operations to 6203 different 6204 addresses will not 6205 be reordered by 6206 hardware. 6207 6208 store *none* *none* - local 1. ds_store 6209 **Unordered Atomic** 6210 ------------------------------------------------------------------------------------ 6211 load atomic unordered *any* *any* *Same as non-atomic*. 6212 store atomic unordered *any* *any* *Same as non-atomic*. 6213 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6214 **Monotonic Atomic** 6215 ------------------------------------------------------------------------------------ 6216 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6217 - wavefront - generic 6218 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6219 - generic glc=1 6220 6221 - If not TgSplit execution 6222 mode, omit glc=1. 6223 6224 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6225 - wavefront local address space cannot 6226 - workgroup be used.* 6227 6228 1. ds_load 6229 load atomic monotonic - agent - global 1. buffer/global/flat_load 6230 - generic glc=1 6231 load atomic monotonic - system - global 1. buffer/global/flat_load 6232 - generic glc=1 scc=1 6233 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6234 - wavefront - generic 6235 - workgroup 6236 - agent 6237 store atomic monotonic - system - global 1. buffer/global/flat_store 6238 - generic scc=1 6239 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6240 - wavefront local address space cannot 6241 - workgroup be used.* 6242 6243 1. ds_store 6244 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6245 - wavefront - generic 6246 - workgroup 6247 - agent 6248 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6249 - generic scc=1 6250 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6251 - wavefront local address space cannot 6252 - workgroup be used.* 6253 6254 1. ds_atomic 6255 **Acquire Atomic** 6256 ------------------------------------------------------------------------------------ 6257 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6258 - wavefront - local 6259 - generic 6260 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6261 6262 - If not TgSplit execution 6263 mode, omit glc=1. 6264 6265 2. s_waitcnt vmcnt(0) 6266 6267 - If not TgSplit execution 6268 mode, omit. 6269 - Must happen before the 6270 following buffer_wbinvl1_vol. 6271 6272 3. buffer_wbinvl1_vol 6273 6274 - If not TgSplit execution 6275 mode, omit. 6276 - Must happen before 6277 any following 6278 global/generic 6279 load/load 6280 atomic/store/store 6281 atomic/atomicrmw. 6282 - Ensures that 6283 following 6284 loads will not see 6285 stale data. 6286 6287 load atomic acquire - workgroup - local *If TgSplit execution mode, 6288 local address space cannot 6289 be used.* 6290 6291 1. ds_load 6292 2. s_waitcnt lgkmcnt(0) 6293 6294 - If OpenCL, omit. 6295 - Must happen before 6296 any following 6297 global/generic 6298 load/load 6299 atomic/store/store 6300 atomic/atomicrmw. 6301 - Ensures any 6302 following global 6303 data read is no 6304 older than the local load 6305 atomic value being 6306 acquired. 6307 6308 load atomic acquire - workgroup - generic 1. flat_load glc=1 6309 6310 - If not TgSplit execution 6311 mode, omit glc=1. 6312 6313 2. s_waitcnt lgkm/vmcnt(0) 6314 6315 - Use lgkmcnt(0) if not 6316 TgSplit execution mode 6317 and vmcnt(0) if TgSplit 6318 execution mode. 6319 - If OpenCL, omit lgkmcnt(0). 6320 - Must happen before 6321 the following 6322 buffer_wbinvl1_vol and any 6323 following global/generic 6324 load/load 6325 atomic/store/store 6326 atomic/atomicrmw. 6327 - Ensures any 6328 following global 6329 data read is no 6330 older than a local load 6331 atomic value being 6332 acquired. 6333 6334 3. buffer_wbinvl1_vol 6335 6336 - If not TgSplit execution 6337 mode, omit. 6338 - Ensures that 6339 following 6340 loads will not see 6341 stale data. 6342 6343 load atomic acquire - agent - global 1. buffer/global_load 6344 glc=1 6345 2. s_waitcnt vmcnt(0) 6346 6347 - Must happen before 6348 following 6349 buffer_wbinvl1_vol. 6350 - Ensures the load 6351 has completed 6352 before invalidating 6353 the cache. 6354 6355 3. buffer_wbinvl1_vol 6356 6357 - Must happen before 6358 any following 6359 global/generic 6360 load/load 6361 atomic/atomicrmw. 6362 - Ensures that 6363 following 6364 loads will not see 6365 stale global data. 6366 6367 load atomic acquire - system - global 1. buffer/global/flat_load 6368 glc=1 scc=1 6369 2. s_waitcnt vmcnt(0) 6370 6371 - Must happen before 6372 following buffer_invl2 and 6373 buffer_wbinvl1_vol. 6374 - Ensures the load 6375 has completed 6376 before invalidating 6377 the cache. 6378 6379 3. buffer_invl2; 6380 buffer_wbinvl1_vol 6381 6382 - Must happen before 6383 any following 6384 global/generic 6385 load/load 6386 atomic/atomicrmw. 6387 - Ensures that 6388 following 6389 loads will not see 6390 stale MTYPE NC global data. 6391 MTYPE RW and CC memory will 6392 never be stale due to the 6393 memory probes. 6394 6395 load atomic acquire - agent - generic 1. flat_load glc=1 6396 2. s_waitcnt vmcnt(0) & 6397 lgkmcnt(0) 6398 6399 - If TgSplit execution mode, 6400 omit lgkmcnt(0). 6401 - If OpenCL omit 6402 lgkmcnt(0). 6403 - Must happen before 6404 following 6405 buffer_wbinvl1_vol. 6406 - Ensures the flat_load 6407 has completed 6408 before invalidating 6409 the cache. 6410 6411 3. buffer_wbinvl1_vol 6412 6413 - Must happen before 6414 any following 6415 global/generic 6416 load/load 6417 atomic/atomicrmw. 6418 - Ensures that 6419 following loads 6420 will not see stale 6421 global data. 6422 6423 load atomic acquire - system - generic 1. flat_load glc=1 scc=1 6424 2. s_waitcnt vmcnt(0) & 6425 lgkmcnt(0) 6426 6427 - If TgSplit execution mode, 6428 omit lgkmcnt(0). 6429 - If OpenCL omit 6430 lgkmcnt(0). 6431 - Must happen before 6432 following 6433 buffer_invl2 and 6434 buffer_wbinvl1_vol. 6435 - Ensures the flat_load 6436 has completed 6437 before invalidating 6438 the caches. 6439 6440 3. buffer_invl2; 6441 buffer_wbinvl1_vol 6442 6443 - Must happen before 6444 any following 6445 global/generic 6446 load/load 6447 atomic/atomicrmw. 6448 - Ensures that 6449 following 6450 loads will not see 6451 stale MTYPE NC global data. 6452 MTYPE RW and CC memory will 6453 never be stale due to the 6454 memory probes. 6455 6456 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6457 - wavefront - generic 6458 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6459 - wavefront local address space cannot 6460 be used.* 6461 6462 1. ds_atomic 6463 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6464 2. s_waitcnt vmcnt(0) 6465 6466 - If not TgSplit execution 6467 mode, omit. 6468 - Must happen before the 6469 following buffer_wbinvl1_vol. 6470 - Ensures the atomicrmw 6471 has completed 6472 before invalidating 6473 the cache. 6474 6475 3. buffer_wbinvl1_vol 6476 6477 - If not TgSplit execution 6478 mode, omit. 6479 - Must happen before 6480 any following 6481 global/generic 6482 load/load 6483 atomic/atomicrmw. 6484 - Ensures that 6485 following loads 6486 will not see stale 6487 global data. 6488 6489 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6490 local address space cannot 6491 be used.* 6492 6493 1. ds_atomic 6494 2. s_waitcnt lgkmcnt(0) 6495 6496 - If OpenCL, omit. 6497 - Must happen before 6498 any following 6499 global/generic 6500 load/load 6501 atomic/store/store 6502 atomic/atomicrmw. 6503 - Ensures any 6504 following global 6505 data read is no 6506 older than the local 6507 atomicrmw value 6508 being acquired. 6509 6510 atomicrmw acquire - workgroup - generic 1. flat_atomic 6511 2. s_waitcnt lgkm/vmcnt(0) 6512 6513 - Use lgkmcnt(0) if not 6514 TgSplit execution mode 6515 and vmcnt(0) if TgSplit 6516 execution mode. 6517 - If OpenCL, omit lgkmcnt(0). 6518 - Must happen before 6519 the following 6520 buffer_wbinvl1_vol and 6521 any following 6522 global/generic 6523 load/load 6524 atomic/store/store 6525 atomic/atomicrmw. 6526 - Ensures any 6527 following global 6528 data read is no 6529 older than a local 6530 atomicrmw value 6531 being acquired. 6532 6533 3. buffer_wbinvl1_vol 6534 6535 - If not TgSplit execution 6536 mode, omit. 6537 - Ensures that 6538 following 6539 loads will not see 6540 stale data. 6541 6542 atomicrmw acquire - agent - global 1. buffer/global_atomic 6543 2. s_waitcnt vmcnt(0) 6544 6545 - Must happen before 6546 following 6547 buffer_wbinvl1_vol. 6548 - Ensures the 6549 atomicrmw has 6550 completed before 6551 invalidating the 6552 cache. 6553 6554 3. buffer_wbinvl1_vol 6555 6556 - Must happen before 6557 any following 6558 global/generic 6559 load/load 6560 atomic/atomicrmw. 6561 - Ensures that 6562 following loads 6563 will not see stale 6564 global data. 6565 6566 atomicrmw acquire - system - global 1. buffer/global_atomic 6567 scc=1 6568 2. s_waitcnt vmcnt(0) 6569 6570 - Must happen before 6571 following buffer_invl2 and 6572 buffer_wbinvl1_vol. 6573 - Ensures the 6574 atomicrmw has 6575 completed before 6576 invalidating the 6577 caches. 6578 6579 3. buffer_invl2; 6580 buffer_wbinvl1_vol 6581 6582 - Must happen before 6583 any following 6584 global/generic 6585 load/load 6586 atomic/atomicrmw. 6587 - Ensures that 6588 following 6589 loads will not see 6590 stale MTYPE NC global data. 6591 MTYPE RW and CC memory will 6592 never be stale due to the 6593 memory probes. 6594 6595 atomicrmw acquire - agent - generic 1. flat_atomic 6596 2. s_waitcnt vmcnt(0) & 6597 lgkmcnt(0) 6598 6599 - If TgSplit execution mode, 6600 omit lgkmcnt(0). 6601 - If OpenCL, omit 6602 lgkmcnt(0). 6603 - Must happen before 6604 following 6605 buffer_wbinvl1_vol. 6606 - Ensures the 6607 atomicrmw has 6608 completed before 6609 invalidating the 6610 cache. 6611 6612 3. buffer_wbinvl1_vol 6613 6614 - Must happen before 6615 any following 6616 global/generic 6617 load/load 6618 atomic/atomicrmw. 6619 - Ensures that 6620 following loads 6621 will not see stale 6622 global data. 6623 6624 atomicrmw acquire - system - generic 1. flat_atomic scc=1 6625 2. s_waitcnt vmcnt(0) & 6626 lgkmcnt(0) 6627 6628 - If TgSplit execution mode, 6629 omit lgkmcnt(0). 6630 - If OpenCL, omit 6631 lgkmcnt(0). 6632 - Must happen before 6633 following 6634 buffer_invl2 and 6635 buffer_wbinvl1_vol. 6636 - Ensures the 6637 atomicrmw has 6638 completed before 6639 invalidating the 6640 caches. 6641 6642 3. buffer_invl2; 6643 buffer_wbinvl1_vol 6644 6645 - Must happen before 6646 any following 6647 global/generic 6648 load/load 6649 atomic/atomicrmw. 6650 - Ensures that 6651 following 6652 loads will not see 6653 stale MTYPE NC global data. 6654 MTYPE RW and CC memory will 6655 never be stale due to the 6656 memory probes. 6657 6658 fence acquire - singlethread *none* *none* 6659 - wavefront 6660 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6661 6662 - Use lgkmcnt(0) if not 6663 TgSplit execution mode 6664 and vmcnt(0) if TgSplit 6665 execution mode. 6666 - If OpenCL and 6667 address space is 6668 not generic, omit 6669 lgkmcnt(0). 6670 - If OpenCL and 6671 address space is 6672 local, omit 6673 vmcnt(0). 6674 - However, since LLVM 6675 currently has no 6676 address space on 6677 the fence need to 6678 conservatively 6679 always generate. If 6680 fence had an 6681 address space then 6682 set to address 6683 space of OpenCL 6684 fence flag, or to 6685 generic if both 6686 local and global 6687 flags are 6688 specified. 6689 - s_waitcnt vmcnt(0) 6690 must happen after 6691 any preceding 6692 global/generic load 6693 atomic/ 6694 atomicrmw 6695 with an equal or 6696 wider sync scope 6697 and memory ordering 6698 stronger than 6699 unordered (this is 6700 termed the 6701 fence-paired-atomic). 6702 - s_waitcnt lgkmcnt(0) 6703 must happen after 6704 any preceding 6705 local/generic load 6706 atomic/atomicrmw 6707 with an equal or 6708 wider sync scope 6709 and memory ordering 6710 stronger than 6711 unordered (this is 6712 termed the 6713 fence-paired-atomic). 6714 - Must happen before 6715 the following 6716 buffer_wbinvl1_vol and 6717 any following 6718 global/generic 6719 load/load 6720 atomic/store/store 6721 atomic/atomicrmw. 6722 - Ensures any 6723 following global 6724 data read is no 6725 older than the 6726 value read by the 6727 fence-paired-atomic. 6728 6729 3. buffer_wbinvl1_vol 6730 6731 - If not TgSplit execution 6732 mode, omit. 6733 - Ensures that 6734 following 6735 loads will not see 6736 stale data. 6737 6738 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6739 vmcnt(0) 6740 6741 - If TgSplit execution mode, 6742 omit lgkmcnt(0). 6743 - If OpenCL and 6744 address space is 6745 not generic, omit 6746 lgkmcnt(0). 6747 - However, since LLVM 6748 currently has no 6749 address space on 6750 the fence need to 6751 conservatively 6752 always generate 6753 (see comment for 6754 previous fence). 6755 - Could be split into 6756 separate s_waitcnt 6757 vmcnt(0) and 6758 s_waitcnt 6759 lgkmcnt(0) to allow 6760 them to be 6761 independently moved 6762 according to the 6763 following rules. 6764 - s_waitcnt vmcnt(0) 6765 must happen after 6766 any preceding 6767 global/generic load 6768 atomic/atomicrmw 6769 with an equal or 6770 wider sync scope 6771 and memory ordering 6772 stronger than 6773 unordered (this is 6774 termed the 6775 fence-paired-atomic). 6776 - s_waitcnt lgkmcnt(0) 6777 must happen after 6778 any preceding 6779 local/generic load 6780 atomic/atomicrmw 6781 with an equal or 6782 wider sync scope 6783 and memory ordering 6784 stronger than 6785 unordered (this is 6786 termed the 6787 fence-paired-atomic). 6788 - Must happen before 6789 the following 6790 buffer_wbinvl1_vol. 6791 - Ensures that the 6792 fence-paired atomic 6793 has completed 6794 before invalidating 6795 the 6796 cache. Therefore 6797 any following 6798 locations read must 6799 be no older than 6800 the value read by 6801 the 6802 fence-paired-atomic. 6803 6804 2. buffer_wbinvl1_vol 6805 6806 - Must happen before any 6807 following global/generic 6808 load/load 6809 atomic/store/store 6810 atomic/atomicrmw. 6811 - Ensures that 6812 following loads 6813 will not see stale 6814 global data. 6815 6816 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6817 vmcnt(0) 6818 6819 - If TgSplit execution mode, 6820 omit lgkmcnt(0). 6821 - If OpenCL and 6822 address space is 6823 not generic, omit 6824 lgkmcnt(0). 6825 - However, since LLVM 6826 currently has no 6827 address space on 6828 the fence need to 6829 conservatively 6830 always generate 6831 (see comment for 6832 previous fence). 6833 - Could be split into 6834 separate s_waitcnt 6835 vmcnt(0) and 6836 s_waitcnt 6837 lgkmcnt(0) to allow 6838 them to be 6839 independently moved 6840 according to the 6841 following rules. 6842 - s_waitcnt vmcnt(0) 6843 must happen after 6844 any preceding 6845 global/generic load 6846 atomic/atomicrmw 6847 with an equal or 6848 wider sync scope 6849 and memory ordering 6850 stronger than 6851 unordered (this is 6852 termed the 6853 fence-paired-atomic). 6854 - s_waitcnt lgkmcnt(0) 6855 must happen after 6856 any preceding 6857 local/generic load 6858 atomic/atomicrmw 6859 with an equal or 6860 wider sync scope 6861 and memory ordering 6862 stronger than 6863 unordered (this is 6864 termed the 6865 fence-paired-atomic). 6866 - Must happen before 6867 the following buffer_invl2 and 6868 buffer_wbinvl1_vol. 6869 - Ensures that the 6870 fence-paired atomic 6871 has completed 6872 before invalidating 6873 the 6874 cache. Therefore 6875 any following 6876 locations read must 6877 be no older than 6878 the value read by 6879 the 6880 fence-paired-atomic. 6881 6882 2. buffer_invl2; 6883 buffer_wbinvl1_vol 6884 6885 - Must happen before any 6886 following global/generic 6887 load/load 6888 atomic/store/store 6889 atomic/atomicrmw. 6890 - Ensures that 6891 following loads 6892 will not see stale 6893 global data. 6894 6895 **Release Atomic** 6896 ------------------------------------------------------------------------------------ 6897 store atomic release - singlethread - global 1. buffer/global/flat_store 6898 - wavefront - generic 6899 store atomic release - singlethread - local *If TgSplit execution mode, 6900 - wavefront local address space cannot 6901 be used.* 6902 6903 1. ds_store 6904 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6905 - generic 6906 - Use lgkmcnt(0) if not 6907 TgSplit execution mode 6908 and vmcnt(0) if TgSplit 6909 execution mode. 6910 - If OpenCL, omit lgkmcnt(0). 6911 - s_waitcnt vmcnt(0) 6912 must happen after 6913 any preceding 6914 global/generic load/store/ 6915 load atomic/store atomic/ 6916 atomicrmw. 6917 - s_waitcnt lgkmcnt(0) 6918 must happen after 6919 any preceding 6920 local/generic 6921 load/store/load 6922 atomic/store 6923 atomic/atomicrmw. 6924 - Must happen before 6925 the following 6926 store. 6927 - Ensures that all 6928 memory operations 6929 have 6930 completed before 6931 performing the 6932 store that is being 6933 released. 6934 6935 2. buffer/global/flat_store 6936 store atomic release - workgroup - local *If TgSplit execution mode, 6937 local address space cannot 6938 be used.* 6939 6940 1. ds_store 6941 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6942 - generic vmcnt(0) 6943 6944 - If TgSplit execution mode, 6945 omit lgkmcnt(0). 6946 - If OpenCL and 6947 address space is 6948 not generic, omit 6949 lgkmcnt(0). 6950 - Could be split into 6951 separate s_waitcnt 6952 vmcnt(0) and 6953 s_waitcnt 6954 lgkmcnt(0) to allow 6955 them to be 6956 independently moved 6957 according to the 6958 following rules. 6959 - s_waitcnt vmcnt(0) 6960 must happen after 6961 any preceding 6962 global/generic 6963 load/store/load 6964 atomic/store 6965 atomic/atomicrmw. 6966 - s_waitcnt lgkmcnt(0) 6967 must happen after 6968 any preceding 6969 local/generic 6970 load/store/load 6971 atomic/store 6972 atomic/atomicrmw. 6973 - Must happen before 6974 the following 6975 store. 6976 - Ensures that all 6977 memory operations 6978 to memory have 6979 completed before 6980 performing the 6981 store that is being 6982 released. 6983 6984 2. buffer/global/flat_store 6985 store atomic release - system - global 1. buffer_wbl2 6986 - generic 6987 - Must happen before 6988 following s_waitcnt. 6989 - Performs L2 writeback to 6990 ensure previous 6991 global/generic 6992 store/atomicrmw are 6993 visible at system scope. 6994 6995 2. s_waitcnt lgkmcnt(0) & 6996 vmcnt(0) 6997 6998 - If TgSplit execution mode, 6999 omit lgkmcnt(0). 7000 - If OpenCL and 7001 address space is 7002 not generic, omit 7003 lgkmcnt(0). 7004 - Could be split into 7005 separate s_waitcnt 7006 vmcnt(0) and 7007 s_waitcnt 7008 lgkmcnt(0) to allow 7009 them to be 7010 independently moved 7011 according to the 7012 following rules. 7013 - s_waitcnt vmcnt(0) 7014 must happen after any 7015 preceding 7016 global/generic 7017 load/store/load 7018 atomic/store 7019 atomic/atomicrmw. 7020 - s_waitcnt lgkmcnt(0) 7021 must happen after any 7022 preceding 7023 local/generic 7024 load/store/load 7025 atomic/store 7026 atomic/atomicrmw. 7027 - Must happen before 7028 the following 7029 store. 7030 - Ensures that all 7031 memory operations 7032 to memory and the L2 7033 writeback have 7034 completed before 7035 performing the 7036 store that is being 7037 released. 7038 7039 2. buffer/global/flat_store 7040 scc=1 7041 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7042 - wavefront - generic 7043 atomicrmw release - singlethread - local *If TgSplit execution mode, 7044 - wavefront local address space cannot 7045 be used.* 7046 7047 1. ds_atomic 7048 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7049 - generic 7050 - Use lgkmcnt(0) if not 7051 TgSplit execution mode 7052 and vmcnt(0) if TgSplit 7053 execution mode. 7054 - If OpenCL, omit 7055 lgkmcnt(0). 7056 - s_waitcnt vmcnt(0) 7057 must happen after 7058 any preceding 7059 global/generic load/store/ 7060 load atomic/store atomic/ 7061 atomicrmw. 7062 - s_waitcnt lgkmcnt(0) 7063 must happen after 7064 any preceding 7065 local/generic 7066 load/store/load 7067 atomic/store 7068 atomic/atomicrmw. 7069 - Must happen before 7070 the following 7071 atomicrmw. 7072 - Ensures that all 7073 memory operations 7074 have 7075 completed before 7076 performing the 7077 atomicrmw that is 7078 being released. 7079 7080 2. buffer/global/flat_atomic 7081 atomicrmw release - workgroup - local *If TgSplit execution mode, 7082 local address space cannot 7083 be used.* 7084 7085 1. ds_atomic 7086 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7087 - generic vmcnt(0) 7088 7089 - If TgSplit execution mode, 7090 omit lgkmcnt(0). 7091 - If OpenCL, omit 7092 lgkmcnt(0). 7093 - Could be split into 7094 separate s_waitcnt 7095 vmcnt(0) and 7096 s_waitcnt 7097 lgkmcnt(0) to allow 7098 them to be 7099 independently moved 7100 according to the 7101 following rules. 7102 - s_waitcnt vmcnt(0) 7103 must happen after 7104 any preceding 7105 global/generic 7106 load/store/load 7107 atomic/store 7108 atomic/atomicrmw. 7109 - s_waitcnt lgkmcnt(0) 7110 must happen after 7111 any preceding 7112 local/generic 7113 load/store/load 7114 atomic/store 7115 atomic/atomicrmw. 7116 - Must happen before 7117 the following 7118 atomicrmw. 7119 - Ensures that all 7120 memory operations 7121 to global and local 7122 have completed 7123 before performing 7124 the atomicrmw that 7125 is being released. 7126 7127 2. buffer/global/flat_atomic 7128 atomicrmw release - system - global 1. buffer_wbl2 7129 - generic 7130 - Must happen before 7131 following s_waitcnt. 7132 - Performs L2 writeback to 7133 ensure previous 7134 global/generic 7135 store/atomicrmw are 7136 visible at system scope. 7137 7138 2. s_waitcnt lgkmcnt(0) & 7139 vmcnt(0) 7140 7141 - If TgSplit execution mode, 7142 omit lgkmcnt(0). 7143 - If OpenCL, omit 7144 lgkmcnt(0). 7145 - Could be split into 7146 separate s_waitcnt 7147 vmcnt(0) and 7148 s_waitcnt 7149 lgkmcnt(0) to allow 7150 them to be 7151 independently moved 7152 according to the 7153 following rules. 7154 - s_waitcnt vmcnt(0) 7155 must happen after 7156 any preceding 7157 global/generic 7158 load/store/load 7159 atomic/store 7160 atomic/atomicrmw. 7161 - s_waitcnt lgkmcnt(0) 7162 must happen after 7163 any preceding 7164 local/generic 7165 load/store/load 7166 atomic/store 7167 atomic/atomicrmw. 7168 - Must happen before 7169 the following 7170 atomicrmw. 7171 - Ensures that all 7172 memory operations 7173 to memory and the L2 7174 writeback have 7175 completed before 7176 performing the 7177 store that is being 7178 released. 7179 7180 3. buffer/global/flat_atomic 7181 scc=1 7182 fence release - singlethread *none* *none* 7183 - wavefront 7184 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7185 7186 - Use lgkmcnt(0) if not 7187 TgSplit execution mode 7188 and vmcnt(0) if TgSplit 7189 execution mode. 7190 - If OpenCL and 7191 address space is 7192 not generic, omit 7193 lgkmcnt(0). 7194 - If OpenCL and 7195 address space is 7196 local, omit 7197 vmcnt(0). 7198 - However, since LLVM 7199 currently has no 7200 address space on 7201 the fence need to 7202 conservatively 7203 always generate. If 7204 fence had an 7205 address space then 7206 set to address 7207 space of OpenCL 7208 fence flag, or to 7209 generic if both 7210 local and global 7211 flags are 7212 specified. 7213 - s_waitcnt vmcnt(0) 7214 must happen after 7215 any preceding 7216 global/generic 7217 load/store/ 7218 load atomic/store atomic/ 7219 atomicrmw. 7220 - s_waitcnt lgkmcnt(0) 7221 must happen after 7222 any preceding 7223 local/generic 7224 load/load 7225 atomic/store/store 7226 atomic/atomicrmw. 7227 - Must happen before 7228 any following store 7229 atomic/atomicrmw 7230 with an equal or 7231 wider sync scope 7232 and memory ordering 7233 stronger than 7234 unordered (this is 7235 termed the 7236 fence-paired-atomic). 7237 - Ensures that all 7238 memory operations 7239 have 7240 completed before 7241 performing the 7242 following 7243 fence-paired-atomic. 7244 7245 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7246 vmcnt(0) 7247 7248 - If TgSplit execution mode, 7249 omit lgkmcnt(0). 7250 - If OpenCL and 7251 address space is 7252 not generic, omit 7253 lgkmcnt(0). 7254 - If OpenCL and 7255 address space is 7256 local, omit 7257 vmcnt(0). 7258 - However, since LLVM 7259 currently has no 7260 address space on 7261 the fence need to 7262 conservatively 7263 always generate. If 7264 fence had an 7265 address space then 7266 set to address 7267 space of OpenCL 7268 fence flag, or to 7269 generic if both 7270 local and global 7271 flags are 7272 specified. 7273 - Could be split into 7274 separate s_waitcnt 7275 vmcnt(0) and 7276 s_waitcnt 7277 lgkmcnt(0) to allow 7278 them to be 7279 independently moved 7280 according to the 7281 following rules. 7282 - s_waitcnt vmcnt(0) 7283 must happen after 7284 any preceding 7285 global/generic 7286 load/store/load 7287 atomic/store 7288 atomic/atomicrmw. 7289 - s_waitcnt lgkmcnt(0) 7290 must happen after 7291 any preceding 7292 local/generic 7293 load/store/load 7294 atomic/store 7295 atomic/atomicrmw. 7296 - Must happen before 7297 any following store 7298 atomic/atomicrmw 7299 with an equal or 7300 wider sync scope 7301 and memory ordering 7302 stronger than 7303 unordered (this is 7304 termed the 7305 fence-paired-atomic). 7306 - Ensures that all 7307 memory operations 7308 have 7309 completed before 7310 performing the 7311 following 7312 fence-paired-atomic. 7313 7314 fence release - system *none* 1. buffer_wbl2 7315 7316 - If OpenCL and 7317 address space is 7318 local, omit. 7319 - Must happen before 7320 following s_waitcnt. 7321 - Performs L2 writeback to 7322 ensure previous 7323 global/generic 7324 store/atomicrmw are 7325 visible at system scope. 7326 7327 2. s_waitcnt lgkmcnt(0) & 7328 vmcnt(0) 7329 7330 - If TgSplit execution mode, 7331 omit lgkmcnt(0). 7332 - If OpenCL and 7333 address space is 7334 not generic, omit 7335 lgkmcnt(0). 7336 - If OpenCL and 7337 address space is 7338 local, omit 7339 vmcnt(0). 7340 - However, since LLVM 7341 currently has no 7342 address space on 7343 the fence need to 7344 conservatively 7345 always generate. If 7346 fence had an 7347 address space then 7348 set to address 7349 space of OpenCL 7350 fence flag, or to 7351 generic if both 7352 local and global 7353 flags are 7354 specified. 7355 - Could be split into 7356 separate s_waitcnt 7357 vmcnt(0) and 7358 s_waitcnt 7359 lgkmcnt(0) to allow 7360 them to be 7361 independently moved 7362 according to the 7363 following rules. 7364 - s_waitcnt vmcnt(0) 7365 must happen after 7366 any preceding 7367 global/generic 7368 load/store/load 7369 atomic/store 7370 atomic/atomicrmw. 7371 - s_waitcnt lgkmcnt(0) 7372 must happen after 7373 any preceding 7374 local/generic 7375 load/store/load 7376 atomic/store 7377 atomic/atomicrmw. 7378 - Must happen before 7379 any following store 7380 atomic/atomicrmw 7381 with an equal or 7382 wider sync scope 7383 and memory ordering 7384 stronger than 7385 unordered (this is 7386 termed the 7387 fence-paired-atomic). 7388 - Ensures that all 7389 memory operations 7390 have 7391 completed before 7392 performing the 7393 following 7394 fence-paired-atomic. 7395 7396 **Acquire-Release Atomic** 7397 ------------------------------------------------------------------------------------ 7398 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7399 - wavefront - generic 7400 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7401 - wavefront local address space cannot 7402 be used.* 7403 7404 1. ds_atomic 7405 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7406 7407 - Use lgkmcnt(0) if not 7408 TgSplit execution mode 7409 and vmcnt(0) if TgSplit 7410 execution mode. 7411 - If OpenCL, omit 7412 lgkmcnt(0). 7413 - Must happen after 7414 any preceding 7415 local/generic 7416 load/store/load 7417 atomic/store 7418 atomic/atomicrmw. 7419 - s_waitcnt vmcnt(0) 7420 must happen after 7421 any preceding 7422 global/generic load/store/ 7423 load atomic/store atomic/ 7424 atomicrmw. 7425 - s_waitcnt lgkmcnt(0) 7426 must happen after 7427 any preceding 7428 local/generic 7429 load/store/load 7430 atomic/store 7431 atomic/atomicrmw. 7432 - Must happen before 7433 the following 7434 atomicrmw. 7435 - Ensures that all 7436 memory operations 7437 have 7438 completed before 7439 performing the 7440 atomicrmw that is 7441 being released. 7442 7443 2. buffer/global_atomic 7444 3. s_waitcnt vmcnt(0) 7445 7446 - If not TgSplit execution 7447 mode, omit. 7448 - Must happen before 7449 the following 7450 buffer_wbinvl1_vol. 7451 - Ensures any 7452 following global 7453 data read is no 7454 older than the 7455 atomicrmw value 7456 being acquired. 7457 7458 4. buffer_wbinvl1_vol 7459 7460 - If not TgSplit execution 7461 mode, omit. 7462 - Ensures that 7463 following 7464 loads will not see 7465 stale data. 7466 7467 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7468 local address space cannot 7469 be used.* 7470 7471 1. ds_atomic 7472 2. s_waitcnt lgkmcnt(0) 7473 7474 - If OpenCL, omit. 7475 - Must happen before 7476 any following 7477 global/generic 7478 load/load 7479 atomic/store/store 7480 atomic/atomicrmw. 7481 - Ensures any 7482 following global 7483 data read is no 7484 older than the local load 7485 atomic value being 7486 acquired. 7487 7488 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7489 7490 - Use lgkmcnt(0) if not 7491 TgSplit execution mode 7492 and vmcnt(0) if TgSplit 7493 execution mode. 7494 - If OpenCL, omit 7495 lgkmcnt(0). 7496 - s_waitcnt vmcnt(0) 7497 must happen after 7498 any preceding 7499 global/generic load/store/ 7500 load atomic/store atomic/ 7501 atomicrmw. 7502 - s_waitcnt lgkmcnt(0) 7503 must happen after 7504 any preceding 7505 local/generic 7506 load/store/load 7507 atomic/store 7508 atomic/atomicrmw. 7509 - Must happen before 7510 the following 7511 atomicrmw. 7512 - Ensures that all 7513 memory operations 7514 have 7515 completed before 7516 performing the 7517 atomicrmw that is 7518 being released. 7519 7520 2. flat_atomic 7521 3. s_waitcnt lgkmcnt(0) & 7522 vmcnt(0) 7523 7524 - If not TgSplit execution 7525 mode, omit vmcnt(0). 7526 - If OpenCL, omit 7527 lgkmcnt(0). 7528 - Must happen before 7529 the following 7530 buffer_wbinvl1_vol and 7531 any following 7532 global/generic 7533 load/load 7534 atomic/store/store 7535 atomic/atomicrmw. 7536 - Ensures any 7537 following global 7538 data read is no 7539 older than a local load 7540 atomic value being 7541 acquired. 7542 7543 3. buffer_wbinvl1_vol 7544 7545 - If not TgSplit execution 7546 mode, omit. 7547 - Ensures that 7548 following 7549 loads will not see 7550 stale data. 7551 7552 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7553 vmcnt(0) 7554 7555 - If TgSplit execution mode, 7556 omit lgkmcnt(0). 7557 - If OpenCL, omit 7558 lgkmcnt(0). 7559 - Could be split into 7560 separate s_waitcnt 7561 vmcnt(0) and 7562 s_waitcnt 7563 lgkmcnt(0) to allow 7564 them to be 7565 independently moved 7566 according to the 7567 following rules. 7568 - s_waitcnt vmcnt(0) 7569 must happen after 7570 any preceding 7571 global/generic 7572 load/store/load 7573 atomic/store 7574 atomic/atomicrmw. 7575 - s_waitcnt lgkmcnt(0) 7576 must happen after 7577 any preceding 7578 local/generic 7579 load/store/load 7580 atomic/store 7581 atomic/atomicrmw. 7582 - Must happen before 7583 the following 7584 atomicrmw. 7585 - Ensures that all 7586 memory operations 7587 to global have 7588 completed before 7589 performing the 7590 atomicrmw that is 7591 being released. 7592 7593 2. buffer/global_atomic 7594 3. s_waitcnt vmcnt(0) 7595 7596 - Must happen before 7597 following 7598 buffer_wbinvl1_vol. 7599 - Ensures the 7600 atomicrmw has 7601 completed before 7602 invalidating the 7603 cache. 7604 7605 4. buffer_wbinvl1_vol 7606 7607 - Must happen before 7608 any following 7609 global/generic 7610 load/load 7611 atomic/atomicrmw. 7612 - Ensures that 7613 following loads 7614 will not see stale 7615 global data. 7616 7617 atomicrmw acq_rel - system - global 1. buffer_wbl2 7618 7619 - Must happen before 7620 following s_waitcnt. 7621 - Performs L2 writeback to 7622 ensure previous 7623 global/generic 7624 store/atomicrmw are 7625 visible at system scope. 7626 7627 2. s_waitcnt lgkmcnt(0) & 7628 vmcnt(0) 7629 7630 - If TgSplit execution mode, 7631 omit lgkmcnt(0). 7632 - If OpenCL, omit 7633 lgkmcnt(0). 7634 - Could be split into 7635 separate s_waitcnt 7636 vmcnt(0) and 7637 s_waitcnt 7638 lgkmcnt(0) to allow 7639 them to be 7640 independently moved 7641 according to the 7642 following rules. 7643 - s_waitcnt vmcnt(0) 7644 must happen after 7645 any preceding 7646 global/generic 7647 load/store/load 7648 atomic/store 7649 atomic/atomicrmw. 7650 - s_waitcnt lgkmcnt(0) 7651 must happen after 7652 any preceding 7653 local/generic 7654 load/store/load 7655 atomic/store 7656 atomic/atomicrmw. 7657 - Must happen before 7658 the following 7659 atomicrmw. 7660 - Ensures that all 7661 memory operations 7662 to global and L2 writeback 7663 have completed before 7664 performing the 7665 atomicrmw that is 7666 being released. 7667 7668 3. buffer/global_atomic 7669 scc=1 7670 4. s_waitcnt vmcnt(0) 7671 7672 - Must happen before 7673 following buffer_invl2 and 7674 buffer_wbinvl1_vol. 7675 - Ensures the 7676 atomicrmw has 7677 completed before 7678 invalidating the 7679 caches. 7680 7681 5. buffer_invl2; 7682 buffer_wbinvl1_vol 7683 7684 - Must happen before 7685 any following 7686 global/generic 7687 load/load 7688 atomic/atomicrmw. 7689 - Ensures that 7690 following loads 7691 will not see stale 7692 MTYPE NC global data. 7693 MTYPE RW and CC memory will 7694 never be stale due to the 7695 memory probes. 7696 7697 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7698 vmcnt(0) 7699 7700 - If TgSplit execution mode, 7701 omit lgkmcnt(0). 7702 - If OpenCL, omit 7703 lgkmcnt(0). 7704 - Could be split into 7705 separate s_waitcnt 7706 vmcnt(0) and 7707 s_waitcnt 7708 lgkmcnt(0) to allow 7709 them to be 7710 independently moved 7711 according to the 7712 following rules. 7713 - s_waitcnt vmcnt(0) 7714 must happen after 7715 any preceding 7716 global/generic 7717 load/store/load 7718 atomic/store 7719 atomic/atomicrmw. 7720 - s_waitcnt lgkmcnt(0) 7721 must happen after 7722 any preceding 7723 local/generic 7724 load/store/load 7725 atomic/store 7726 atomic/atomicrmw. 7727 - Must happen before 7728 the following 7729 atomicrmw. 7730 - Ensures that all 7731 memory operations 7732 to global have 7733 completed before 7734 performing the 7735 atomicrmw that is 7736 being released. 7737 7738 2. flat_atomic 7739 3. s_waitcnt vmcnt(0) & 7740 lgkmcnt(0) 7741 7742 - If TgSplit execution mode, 7743 omit lgkmcnt(0). 7744 - If OpenCL, omit 7745 lgkmcnt(0). 7746 - Must happen before 7747 following 7748 buffer_wbinvl1_vol. 7749 - Ensures the 7750 atomicrmw has 7751 completed before 7752 invalidating the 7753 cache. 7754 7755 4. buffer_wbinvl1_vol 7756 7757 - Must happen before 7758 any following 7759 global/generic 7760 load/load 7761 atomic/atomicrmw. 7762 - Ensures that 7763 following loads 7764 will not see stale 7765 global data. 7766 7767 atomicrmw acq_rel - system - generic 1. buffer_wbl2 7768 7769 - Must happen before 7770 following s_waitcnt. 7771 - Performs L2 writeback to 7772 ensure previous 7773 global/generic 7774 store/atomicrmw are 7775 visible at system scope. 7776 7777 2. s_waitcnt lgkmcnt(0) & 7778 vmcnt(0) 7779 7780 - If TgSplit execution mode, 7781 omit lgkmcnt(0). 7782 - If OpenCL, omit 7783 lgkmcnt(0). 7784 - Could be split into 7785 separate s_waitcnt 7786 vmcnt(0) and 7787 s_waitcnt 7788 lgkmcnt(0) to allow 7789 them to be 7790 independently moved 7791 according to the 7792 following rules. 7793 - s_waitcnt vmcnt(0) 7794 must happen after 7795 any preceding 7796 global/generic 7797 load/store/load 7798 atomic/store 7799 atomic/atomicrmw. 7800 - s_waitcnt lgkmcnt(0) 7801 must happen after 7802 any preceding 7803 local/generic 7804 load/store/load 7805 atomic/store 7806 atomic/atomicrmw. 7807 - Must happen before 7808 the following 7809 atomicrmw. 7810 - Ensures that all 7811 memory operations 7812 to global and L2 writeback 7813 have completed before 7814 performing the 7815 atomicrmw that is 7816 being released. 7817 7818 3. flat_atomic scc=1 7819 4. s_waitcnt vmcnt(0) & 7820 lgkmcnt(0) 7821 7822 - If TgSplit execution mode, 7823 omit lgkmcnt(0). 7824 - If OpenCL, omit 7825 lgkmcnt(0). 7826 - Must happen before 7827 following buffer_invl2 and 7828 buffer_wbinvl1_vol. 7829 - Ensures the 7830 atomicrmw has 7831 completed before 7832 invalidating the 7833 caches. 7834 7835 5. buffer_invl2; 7836 buffer_wbinvl1_vol 7837 7838 - Must happen before 7839 any following 7840 global/generic 7841 load/load 7842 atomic/atomicrmw. 7843 - Ensures that 7844 following loads 7845 will not see stale 7846 MTYPE NC global data. 7847 MTYPE RW and CC memory will 7848 never be stale due to the 7849 memory probes. 7850 7851 fence acq_rel - singlethread *none* *none* 7852 - wavefront 7853 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7854 7855 - Use lgkmcnt(0) if not 7856 TgSplit execution mode 7857 and vmcnt(0) if TgSplit 7858 execution mode. 7859 - If OpenCL and 7860 address space is 7861 not generic, omit 7862 lgkmcnt(0). 7863 - If OpenCL and 7864 address space is 7865 local, omit 7866 vmcnt(0). 7867 - However, 7868 since LLVM 7869 currently has no 7870 address space on 7871 the fence need to 7872 conservatively 7873 always generate 7874 (see comment for 7875 previous fence). 7876 - s_waitcnt vmcnt(0) 7877 must happen after 7878 any preceding 7879 global/generic 7880 load/store/ 7881 load atomic/store atomic/ 7882 atomicrmw. 7883 - s_waitcnt lgkmcnt(0) 7884 must happen after 7885 any preceding 7886 local/generic 7887 load/load 7888 atomic/store/store 7889 atomic/atomicrmw. 7890 - Must happen before 7891 any following 7892 global/generic 7893 load/load 7894 atomic/store/store 7895 atomic/atomicrmw. 7896 - Ensures that all 7897 memory operations 7898 have 7899 completed before 7900 performing any 7901 following global 7902 memory operations. 7903 - Ensures that the 7904 preceding 7905 local/generic load 7906 atomic/atomicrmw 7907 with an equal or 7908 wider sync scope 7909 and memory ordering 7910 stronger than 7911 unordered (this is 7912 termed the 7913 acquire-fence-paired-atomic) 7914 has completed 7915 before following 7916 global memory 7917 operations. This 7918 satisfies the 7919 requirements of 7920 acquire. 7921 - Ensures that all 7922 previous memory 7923 operations have 7924 completed before a 7925 following 7926 local/generic store 7927 atomic/atomicrmw 7928 with an equal or 7929 wider sync scope 7930 and memory ordering 7931 stronger than 7932 unordered (this is 7933 termed the 7934 release-fence-paired-atomic). 7935 This satisfies the 7936 requirements of 7937 release. 7938 - Must happen before 7939 the following 7940 buffer_wbinvl1_vol. 7941 - Ensures that the 7942 acquire-fence-paired 7943 atomic has completed 7944 before invalidating 7945 the 7946 cache. Therefore 7947 any following 7948 locations read must 7949 be no older than 7950 the value read by 7951 the 7952 acquire-fence-paired-atomic. 7953 7954 3. buffer_wbinvl1_vol 7955 7956 - If not TgSplit execution 7957 mode, omit. 7958 - Ensures that 7959 following 7960 loads will not see 7961 stale data. 7962 7963 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7964 vmcnt(0) 7965 7966 - If TgSplit execution mode, 7967 omit lgkmcnt(0). 7968 - If OpenCL and 7969 address space is 7970 not generic, omit 7971 lgkmcnt(0). 7972 - However, since LLVM 7973 currently has no 7974 address space on 7975 the fence need to 7976 conservatively 7977 always generate 7978 (see comment for 7979 previous fence). 7980 - Could be split into 7981 separate s_waitcnt 7982 vmcnt(0) and 7983 s_waitcnt 7984 lgkmcnt(0) to allow 7985 them to be 7986 independently moved 7987 according to the 7988 following rules. 7989 - s_waitcnt vmcnt(0) 7990 must happen after 7991 any preceding 7992 global/generic 7993 load/store/load 7994 atomic/store 7995 atomic/atomicrmw. 7996 - s_waitcnt lgkmcnt(0) 7997 must happen after 7998 any preceding 7999 local/generic 8000 load/store/load 8001 atomic/store 8002 atomic/atomicrmw. 8003 - Must happen before 8004 the following 8005 buffer_wbinvl1_vol. 8006 - Ensures that the 8007 preceding 8008 global/local/generic 8009 load 8010 atomic/atomicrmw 8011 with an equal or 8012 wider sync scope 8013 and memory ordering 8014 stronger than 8015 unordered (this is 8016 termed the 8017 acquire-fence-paired-atomic) 8018 has completed 8019 before invalidating 8020 the cache. This 8021 satisfies the 8022 requirements of 8023 acquire. 8024 - Ensures that all 8025 previous memory 8026 operations have 8027 completed before a 8028 following 8029 global/local/generic 8030 store 8031 atomic/atomicrmw 8032 with an equal or 8033 wider sync scope 8034 and memory ordering 8035 stronger than 8036 unordered (this is 8037 termed the 8038 release-fence-paired-atomic). 8039 This satisfies the 8040 requirements of 8041 release. 8042 8043 2. buffer_wbinvl1_vol 8044 8045 - Must happen before 8046 any following 8047 global/generic 8048 load/load 8049 atomic/store/store 8050 atomic/atomicrmw. 8051 - Ensures that 8052 following loads 8053 will not see stale 8054 global data. This 8055 satisfies the 8056 requirements of 8057 acquire. 8058 8059 fence acq_rel - system *none* 1. buffer_wbl2 8060 8061 - If OpenCL and 8062 address space is 8063 local, omit. 8064 - Must happen before 8065 following s_waitcnt. 8066 - Performs L2 writeback to 8067 ensure previous 8068 global/generic 8069 store/atomicrmw are 8070 visible at system scope. 8071 8072 2. s_waitcnt lgkmcnt(0) & 8073 vmcnt(0) 8074 8075 - If TgSplit execution mode, 8076 omit lgkmcnt(0). 8077 - If OpenCL and 8078 address space is 8079 not generic, omit 8080 lgkmcnt(0). 8081 - However, since LLVM 8082 currently has no 8083 address space on 8084 the fence need to 8085 conservatively 8086 always generate 8087 (see comment for 8088 previous fence). 8089 - Could be split into 8090 separate s_waitcnt 8091 vmcnt(0) and 8092 s_waitcnt 8093 lgkmcnt(0) to allow 8094 them to be 8095 independently moved 8096 according to the 8097 following rules. 8098 - s_waitcnt vmcnt(0) 8099 must happen after 8100 any preceding 8101 global/generic 8102 load/store/load 8103 atomic/store 8104 atomic/atomicrmw. 8105 - s_waitcnt lgkmcnt(0) 8106 must happen after 8107 any preceding 8108 local/generic 8109 load/store/load 8110 atomic/store 8111 atomic/atomicrmw. 8112 - Must happen before 8113 the following buffer_invl2 and 8114 buffer_wbinvl1_vol. 8115 - Ensures that the 8116 preceding 8117 global/local/generic 8118 load 8119 atomic/atomicrmw 8120 with an equal or 8121 wider sync scope 8122 and memory ordering 8123 stronger than 8124 unordered (this is 8125 termed the 8126 acquire-fence-paired-atomic) 8127 has completed 8128 before invalidating 8129 the cache. This 8130 satisfies the 8131 requirements of 8132 acquire. 8133 - Ensures that all 8134 previous memory 8135 operations have 8136 completed before a 8137 following 8138 global/local/generic 8139 store 8140 atomic/atomicrmw 8141 with an equal or 8142 wider sync scope 8143 and memory ordering 8144 stronger than 8145 unordered (this is 8146 termed the 8147 release-fence-paired-atomic). 8148 This satisfies the 8149 requirements of 8150 release. 8151 8152 3. buffer_invl2; 8153 buffer_wbinvl1_vol 8154 8155 - Must happen before 8156 any following 8157 global/generic 8158 load/load 8159 atomic/store/store 8160 atomic/atomicrmw. 8161 - Ensures that 8162 following loads 8163 will not see stale 8164 MTYPE NC global data. 8165 MTYPE RW and CC memory will 8166 never be stale due to the 8167 memory probes. 8168 8169 **Sequential Consistent Atomic** 8170 ------------------------------------------------------------------------------------ 8171 load atomic seq_cst - singlethread - global *Same as corresponding 8172 - wavefront - local load atomic acquire, 8173 - generic except must generated 8174 all instructions even 8175 for OpenCL.* 8176 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8177 - generic 8178 - Use lgkmcnt(0) if not 8179 TgSplit execution mode 8180 and vmcnt(0) if TgSplit 8181 execution mode. 8182 - s_waitcnt lgkmcnt(0) must 8183 happen after 8184 preceding 8185 local/generic load 8186 atomic/store 8187 atomic/atomicrmw 8188 with memory 8189 ordering of seq_cst 8190 and with equal or 8191 wider sync scope. 8192 (Note that seq_cst 8193 fences have their 8194 own s_waitcnt 8195 lgkmcnt(0) and so do 8196 not need to be 8197 considered.) 8198 - s_waitcnt vmcnt(0) 8199 must happen after 8200 preceding 8201 global/generic load 8202 atomic/store 8203 atomic/atomicrmw 8204 with memory 8205 ordering of seq_cst 8206 and with equal or 8207 wider sync scope. 8208 (Note that seq_cst 8209 fences have their 8210 own s_waitcnt 8211 vmcnt(0) and so do 8212 not need to be 8213 considered.) 8214 - Ensures any 8215 preceding 8216 sequential 8217 consistent global/local 8218 memory instructions 8219 have completed 8220 before executing 8221 this sequentially 8222 consistent 8223 instruction. This 8224 prevents reordering 8225 a seq_cst store 8226 followed by a 8227 seq_cst load. (Note 8228 that seq_cst is 8229 stronger than 8230 acquire/release as 8231 the reordering of 8232 load acquire 8233 followed by a store 8234 release is 8235 prevented by the 8236 s_waitcnt of 8237 the release, but 8238 there is nothing 8239 preventing a store 8240 release followed by 8241 load acquire from 8242 completing out of 8243 order. The s_waitcnt 8244 could be placed after 8245 seq_store or before 8246 the seq_load. We 8247 choose the load to 8248 make the s_waitcnt be 8249 as late as possible 8250 so that the store 8251 may have already 8252 completed.) 8253 8254 2. *Following 8255 instructions same as 8256 corresponding load 8257 atomic acquire, 8258 except must generated 8259 all instructions even 8260 for OpenCL.* 8261 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8262 local address space cannot 8263 be used.* 8264 8265 *Same as corresponding 8266 load atomic acquire, 8267 except must generated 8268 all instructions even 8269 for OpenCL.* 8270 8271 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8272 - system - generic vmcnt(0) 8273 8274 - If TgSplit execution mode, 8275 omit lgkmcnt(0). 8276 - Could be split into 8277 separate s_waitcnt 8278 vmcnt(0) 8279 and s_waitcnt 8280 lgkmcnt(0) to allow 8281 them to be 8282 independently moved 8283 according to the 8284 following rules. 8285 - s_waitcnt lgkmcnt(0) 8286 must happen after 8287 preceding 8288 global/generic load 8289 atomic/store 8290 atomic/atomicrmw 8291 with memory 8292 ordering of seq_cst 8293 and with equal or 8294 wider sync scope. 8295 (Note that seq_cst 8296 fences have their 8297 own s_waitcnt 8298 lgkmcnt(0) and so do 8299 not need to be 8300 considered.) 8301 - s_waitcnt vmcnt(0) 8302 must happen after 8303 preceding 8304 global/generic load 8305 atomic/store 8306 atomic/atomicrmw 8307 with memory 8308 ordering of seq_cst 8309 and with equal or 8310 wider sync scope. 8311 (Note that seq_cst 8312 fences have their 8313 own s_waitcnt 8314 vmcnt(0) and so do 8315 not need to be 8316 considered.) 8317 - Ensures any 8318 preceding 8319 sequential 8320 consistent global 8321 memory instructions 8322 have completed 8323 before executing 8324 this sequentially 8325 consistent 8326 instruction. This 8327 prevents reordering 8328 a seq_cst store 8329 followed by a 8330 seq_cst load. (Note 8331 that seq_cst is 8332 stronger than 8333 acquire/release as 8334 the reordering of 8335 load acquire 8336 followed by a store 8337 release is 8338 prevented by the 8339 s_waitcnt of 8340 the release, but 8341 there is nothing 8342 preventing a store 8343 release followed by 8344 load acquire from 8345 completing out of 8346 order. The s_waitcnt 8347 could be placed after 8348 seq_store or before 8349 the seq_load. We 8350 choose the load to 8351 make the s_waitcnt be 8352 as late as possible 8353 so that the store 8354 may have already 8355 completed.) 8356 8357 2. *Following 8358 instructions same as 8359 corresponding load 8360 atomic acquire, 8361 except must generated 8362 all instructions even 8363 for OpenCL.* 8364 store atomic seq_cst - singlethread - global *Same as corresponding 8365 - wavefront - local store atomic release, 8366 - workgroup - generic except must generated 8367 - agent all instructions even 8368 - system for OpenCL.* 8369 atomicrmw seq_cst - singlethread - global *Same as corresponding 8370 - wavefront - local atomicrmw acq_rel, 8371 - workgroup - generic except must generated 8372 - agent all instructions even 8373 - system for OpenCL.* 8374 fence seq_cst - singlethread *none* *Same as corresponding 8375 - wavefront fence acq_rel, 8376 - workgroup except must generated 8377 - agent all instructions even 8378 - system for OpenCL.* 8379 ============ ============ ============== ========== ================================ 8380 8381.. _amdgpu-amdhsa-memory-model-gfx10: 8382 8383Memory Model GFX10 8384++++++++++++++++++ 8385 8386For GFX10: 8387 8388* Each agent has multiple shader arrays (SA). 8389* Each SA has multiple work-group processors (WGP). 8390* Each WGP has multiple compute units (CU). 8391* Each CU has multiple SIMDs that execute wavefronts. 8392* The wavefronts for a single work-group are executed in the same 8393 WGP. In CU wavefront execution mode the wavefronts may be executed by 8394 different SIMDs in the same CU. In WGP wavefront execution mode the 8395 wavefronts may be executed by different SIMDs in different CUs in the same 8396 WGP. 8397* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8398 executing on it. 8399* All LDS operations of a WGP are performed as wavefront wide operations in a 8400 global order and involve no caching. Completion is reported to a wavefront in 8401 execution order. 8402* The LDS memory has multiple request queues shared by the SIMDs of a 8403 WGP. Therefore, the LDS operations performed by different wavefronts of a 8404 work-group can be reordered relative to each other, which can result in 8405 reordering the visibility of vector memory operations with respect to LDS 8406 operations of other wavefronts in the same work-group. A ``s_waitcnt 8407 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8408 vector memory operations between wavefronts of a work-group, but not between 8409 operations performed by the same wavefront. 8410* The vector memory operations are performed as wavefront wide operations. 8411 Completion of load/store/sample operations are reported to a wavefront in 8412 execution order of other load/store/sample operations performed by that 8413 wavefront. 8414* The vector memory operations access a vector L0 cache. There is a single L0 8415 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8416 special action is required for coherence between the lanes of a single 8417 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8418 wavefronts executing in the same work-group as they may be executing on SIMDs 8419 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8420 required for coherence between wavefronts executing in different work-groups 8421 as they may be executing on different WGPs. 8422* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8423 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8424 operations are used in a restricted way so do not impact the memory model. See 8425 :ref:`amdgpu-amdhsa-memory-spaces`. 8426* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8427 the same SA. Therefore, no special action is required for coherence between 8428 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8429 required for coherence between wavefronts executing in different work-groups 8430 as they may be executing on different SAs that access different L1s. 8431* The L1 caches have independent quadrants to service disjoint ranges of virtual 8432 addresses. 8433* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8434 vector and scalar memory operations performed by different wavefronts, whether 8435 executing in the same or different work-groups (which may be executing on 8436 different CUs accessing different L0s), can be reordered relative to each 8437 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8438 synchronization between vector memory operations of different wavefronts. It 8439 ensures a previous vector memory operation has completed before executing a 8440 subsequent vector memory or LDS operation and so can be used to meet the 8441 requirements of acquire, release and sequential consistency. 8442* The L1 caches use an L2 cache shared by all SAs on the same agent. 8443* The L2 cache has independent channels to service disjoint ranges of virtual 8444 addresses. 8445* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8446 quadrant has a separate request queue per L2 channel. Therefore, the vector 8447 and scalar memory operations performed by wavefronts executing in different 8448 work-groups (which may be executing on different SAs) of an agent can be 8449 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8450 required to ensure synchronization between vector memory operations of 8451 different SAs. It ensures a previous vector memory operation has completed 8452 before executing a subsequent vector memory and so can be used to meet the 8453 requirements of acquire, release and sequential consistency. 8454* The L2 cache can be kept coherent with other agents on some targets, or ranges 8455 of virtual addresses can be set up to bypass it to ensure system coherence. 8456 8457Scalar memory operations are only used to access memory that is proven to not 8458change during the execution of the kernel dispatch. This includes constant 8459address space and global address space for program scope ``const`` variables. 8460Therefore, the kernel machine code does not have to maintain the scalar cache to 8461ensure it is coherent with the vector caches. The scalar and vector caches are 8462invalidated between kernel dispatches by CP since constant address space data 8463may change between kernel dispatch executions. See 8464:ref:`amdgpu-amdhsa-memory-spaces`. 8465 8466The one exception is if scalar writes are used to spill SGPR registers. In this 8467case the AMDGPU backend ensures the memory location used to spill is never 8468accessed by vector memory operations at the same time. If scalar writes are used 8469then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8470return since the locations may be used for vector memory instructions by a 8471future wavefront that uses the same scratch area, or a function call that 8472creates a frame at the same address, respectively. There is no need for a 8473``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8474 8475For kernarg backing memory: 8476 8477* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8478* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8479 needing to invalidate the L2 cache. 8480* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8481 so the L2 cache will be coherent with the CPU and other agents. 8482 8483Scratch backing memory (which is used for the private address space) is accessed 8484with MTYPE NC (non-coherent). Since the private address space is only accessed 8485by a single thread, and is always write-before-read, there is never a need to 8486invalidate these entries from the L0 or L1 caches. 8487 8488Wavefronts are executed in native mode with in-order reporting of loads and 8489sample instructions. In this mode vmcnt reports completion of load, atomic with 8490return and sample instructions in order, and the vscnt reports the completion of 8491store and atomic without return in order. See ``MEM_ORDERED`` field in 8492:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8493 8494Wavefronts can be executed in WGP or CU wavefront execution mode: 8495 8496* In WGP wavefront execution mode the wavefronts of a work-group are executed 8497 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8498 CU L0 caches is required for work-group synchronization. Also accesses to L1 8499 at work-group scope need to be explicitly ordered as the accesses from 8500 different CUs are not ordered. 8501* In CU wavefront execution mode the wavefronts of a work-group are executed on 8502 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8503 the work-group access the same L0 which in turn ensures L1 accesses are 8504 ordered and so do not require explicit management of the caches for 8505 work-group synchronization. 8506 8507See ``WGP_MODE`` field in 8508:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8509:ref:`amdgpu-target-features`. 8510 8511The code sequences used to implement the memory model for GFX10 are defined in 8512table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8513 8514 .. table:: AMDHSA Memory Model Code Sequences GFX10 8515 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8516 8517 ============ ============ ============== ========== ================================ 8518 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8519 Ordering Sync Scope Address GFX10 8520 Space 8521 ============ ============ ============== ========== ================================ 8522 **Non-Atomic** 8523 ------------------------------------------------------------------------------------ 8524 load *none* *none* - global - !volatile & !nontemporal 8525 - generic 8526 - private 1. buffer/global/flat_load 8527 - constant 8528 - !volatile & nontemporal 8529 8530 1. buffer/global/flat_load 8531 slc=1 8532 8533 - volatile 8534 8535 1. buffer/global/flat_load 8536 glc=1 dlc=1 8537 2. s_waitcnt vmcnt(0) 8538 8539 - Must happen before 8540 any following volatile 8541 global/generic 8542 load/store. 8543 - Ensures that 8544 volatile 8545 operations to 8546 different 8547 addresses will not 8548 be reordered by 8549 hardware. 8550 8551 load *none* *none* - local 1. ds_load 8552 store *none* *none* - global - !volatile & !nontemporal 8553 - generic 8554 - private 1. buffer/global/flat_store 8555 - constant 8556 - !volatile & nontemporal 8557 8558 1. buffer/global/flat_store 8559 slc=1 8560 8561 - volatile 8562 8563 1. buffer/global/flat_store 8564 2. s_waitcnt vscnt(0) 8565 8566 - Must happen before 8567 any following volatile 8568 global/generic 8569 load/store. 8570 - Ensures that 8571 volatile 8572 operations to 8573 different 8574 addresses will not 8575 be reordered by 8576 hardware. 8577 8578 store *none* *none* - local 1. ds_store 8579 **Unordered Atomic** 8580 ------------------------------------------------------------------------------------ 8581 load atomic unordered *any* *any* *Same as non-atomic*. 8582 store atomic unordered *any* *any* *Same as non-atomic*. 8583 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8584 **Monotonic Atomic** 8585 ------------------------------------------------------------------------------------ 8586 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8587 - wavefront - generic 8588 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8589 - generic glc=1 8590 8591 - If CU wavefront execution 8592 mode, omit glc=1. 8593 8594 load atomic monotonic - singlethread - local 1. ds_load 8595 - wavefront 8596 - workgroup 8597 load atomic monotonic - agent - global 1. buffer/global/flat_load 8598 - system - generic glc=1 dlc=1 8599 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8600 - wavefront - generic 8601 - workgroup 8602 - agent 8603 - system 8604 store atomic monotonic - singlethread - local 1. ds_store 8605 - wavefront 8606 - workgroup 8607 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8608 - wavefront - generic 8609 - workgroup 8610 - agent 8611 - system 8612 atomicrmw monotonic - singlethread - local 1. ds_atomic 8613 - wavefront 8614 - workgroup 8615 **Acquire Atomic** 8616 ------------------------------------------------------------------------------------ 8617 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8618 - wavefront - local 8619 - generic 8620 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8621 8622 - If CU wavefront execution 8623 mode, omit glc=1. 8624 8625 2. s_waitcnt vmcnt(0) 8626 8627 - If CU wavefront execution 8628 mode, omit. 8629 - Must happen before 8630 the following buffer_gl0_inv 8631 and before any following 8632 global/generic 8633 load/load 8634 atomic/store/store 8635 atomic/atomicrmw. 8636 8637 3. buffer_gl0_inv 8638 8639 - If CU wavefront execution 8640 mode, omit. 8641 - Ensures that 8642 following 8643 loads will not see 8644 stale data. 8645 8646 load atomic acquire - workgroup - local 1. ds_load 8647 2. s_waitcnt lgkmcnt(0) 8648 8649 - If OpenCL, omit. 8650 - Must happen before 8651 the following buffer_gl0_inv 8652 and before any following 8653 global/generic load/load 8654 atomic/store/store 8655 atomic/atomicrmw. 8656 - Ensures any 8657 following global 8658 data read is no 8659 older than the local load 8660 atomic value being 8661 acquired. 8662 8663 3. buffer_gl0_inv 8664 8665 - If CU wavefront execution 8666 mode, omit. 8667 - If OpenCL, omit. 8668 - Ensures that 8669 following 8670 loads will not see 8671 stale data. 8672 8673 load atomic acquire - workgroup - generic 1. flat_load glc=1 8674 8675 - If CU wavefront execution 8676 mode, omit glc=1. 8677 8678 2. s_waitcnt lgkmcnt(0) & 8679 vmcnt(0) 8680 8681 - If CU wavefront execution 8682 mode, omit vmcnt(0). 8683 - If OpenCL, omit 8684 lgkmcnt(0). 8685 - Must happen before 8686 the following 8687 buffer_gl0_inv and any 8688 following global/generic 8689 load/load 8690 atomic/store/store 8691 atomic/atomicrmw. 8692 - Ensures any 8693 following global 8694 data read is no 8695 older than a local load 8696 atomic value being 8697 acquired. 8698 8699 3. buffer_gl0_inv 8700 8701 - If CU wavefront execution 8702 mode, omit. 8703 - Ensures that 8704 following 8705 loads will not see 8706 stale data. 8707 8708 load atomic acquire - agent - global 1. buffer/global_load 8709 - system glc=1 dlc=1 8710 2. s_waitcnt vmcnt(0) 8711 8712 - Must happen before 8713 following 8714 buffer_gl*_inv. 8715 - Ensures the load 8716 has completed 8717 before invalidating 8718 the caches. 8719 8720 3. buffer_gl0_inv; 8721 buffer_gl1_inv 8722 8723 - Must happen before 8724 any following 8725 global/generic 8726 load/load 8727 atomic/atomicrmw. 8728 - Ensures that 8729 following 8730 loads will not see 8731 stale global data. 8732 8733 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8734 - system 2. s_waitcnt vmcnt(0) & 8735 lgkmcnt(0) 8736 8737 - If OpenCL omit 8738 lgkmcnt(0). 8739 - Must happen before 8740 following 8741 buffer_gl*_invl. 8742 - Ensures the flat_load 8743 has completed 8744 before invalidating 8745 the caches. 8746 8747 3. buffer_gl0_inv; 8748 buffer_gl1_inv 8749 8750 - Must happen before 8751 any following 8752 global/generic 8753 load/load 8754 atomic/atomicrmw. 8755 - Ensures that 8756 following loads 8757 will not see stale 8758 global data. 8759 8760 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8761 - wavefront - local 8762 - generic 8763 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8764 2. s_waitcnt vm/vscnt(0) 8765 8766 - If CU wavefront execution 8767 mode, omit. 8768 - Use vmcnt(0) if atomic with 8769 return and vscnt(0) if 8770 atomic with no-return. 8771 - Must happen before 8772 the following buffer_gl0_inv 8773 and before any following 8774 global/generic 8775 load/load 8776 atomic/store/store 8777 atomic/atomicrmw. 8778 8779 3. buffer_gl0_inv 8780 8781 - If CU wavefront execution 8782 mode, omit. 8783 - Ensures that 8784 following 8785 loads will not see 8786 stale data. 8787 8788 atomicrmw acquire - workgroup - local 1. ds_atomic 8789 2. s_waitcnt lgkmcnt(0) 8790 8791 - If OpenCL, omit. 8792 - Must happen before 8793 the following 8794 buffer_gl0_inv. 8795 - Ensures any 8796 following global 8797 data read is no 8798 older than the local 8799 atomicrmw value 8800 being acquired. 8801 8802 3. buffer_gl0_inv 8803 8804 - If OpenCL omit. 8805 - Ensures that 8806 following 8807 loads will not see 8808 stale data. 8809 8810 atomicrmw acquire - workgroup - generic 1. flat_atomic 8811 2. s_waitcnt lgkmcnt(0) & 8812 vm/vscnt(0) 8813 8814 - If CU wavefront execution 8815 mode, omit vm/vscnt(0). 8816 - If OpenCL, omit lgkmcnt(0). 8817 - Use vmcnt(0) if atomic with 8818 return and vscnt(0) if 8819 atomic with no-return. 8820 - Must happen before 8821 the following 8822 buffer_gl0_inv. 8823 - Ensures any 8824 following global 8825 data read is no 8826 older than a local 8827 atomicrmw value 8828 being acquired. 8829 8830 3. buffer_gl0_inv 8831 8832 - If CU wavefront execution 8833 mode, omit. 8834 - Ensures that 8835 following 8836 loads will not see 8837 stale data. 8838 8839 atomicrmw acquire - agent - global 1. buffer/global_atomic 8840 - system 2. s_waitcnt vm/vscnt(0) 8841 8842 - Use vmcnt(0) if atomic with 8843 return and vscnt(0) if 8844 atomic with no-return. 8845 - Must happen before 8846 following 8847 buffer_gl*_inv. 8848 - Ensures the 8849 atomicrmw has 8850 completed before 8851 invalidating the 8852 caches. 8853 8854 3. buffer_gl0_inv; 8855 buffer_gl1_inv 8856 8857 - Must happen before 8858 any following 8859 global/generic 8860 load/load 8861 atomic/atomicrmw. 8862 - Ensures that 8863 following loads 8864 will not see stale 8865 global data. 8866 8867 atomicrmw acquire - agent - generic 1. flat_atomic 8868 - system 2. s_waitcnt vm/vscnt(0) & 8869 lgkmcnt(0) 8870 8871 - If OpenCL, omit 8872 lgkmcnt(0). 8873 - Use vmcnt(0) if atomic with 8874 return and vscnt(0) if 8875 atomic with no-return. 8876 - Must happen before 8877 following 8878 buffer_gl*_inv. 8879 - Ensures the 8880 atomicrmw has 8881 completed before 8882 invalidating the 8883 caches. 8884 8885 3. buffer_gl0_inv; 8886 buffer_gl1_inv 8887 8888 - Must happen before 8889 any following 8890 global/generic 8891 load/load 8892 atomic/atomicrmw. 8893 - Ensures that 8894 following loads 8895 will not see stale 8896 global data. 8897 8898 fence acquire - singlethread *none* *none* 8899 - wavefront 8900 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 8901 vmcnt(0) & vscnt(0) 8902 8903 - If CU wavefront execution 8904 mode, omit vmcnt(0) and 8905 vscnt(0). 8906 - If OpenCL and 8907 address space is 8908 not generic, omit 8909 lgkmcnt(0). 8910 - If OpenCL and 8911 address space is 8912 local, omit 8913 vmcnt(0) and vscnt(0). 8914 - However, since LLVM 8915 currently has no 8916 address space on 8917 the fence need to 8918 conservatively 8919 always generate. If 8920 fence had an 8921 address space then 8922 set to address 8923 space of OpenCL 8924 fence flag, or to 8925 generic if both 8926 local and global 8927 flags are 8928 specified. 8929 - Could be split into 8930 separate s_waitcnt 8931 vmcnt(0), s_waitcnt 8932 vscnt(0) and s_waitcnt 8933 lgkmcnt(0) to allow 8934 them to be 8935 independently moved 8936 according to the 8937 following rules. 8938 - s_waitcnt vmcnt(0) 8939 must happen after 8940 any preceding 8941 global/generic load 8942 atomic/ 8943 atomicrmw-with-return-value 8944 with an equal or 8945 wider sync scope 8946 and memory ordering 8947 stronger than 8948 unordered (this is 8949 termed the 8950 fence-paired-atomic). 8951 - s_waitcnt vscnt(0) 8952 must happen after 8953 any preceding 8954 global/generic 8955 atomicrmw-no-return-value 8956 with an equal or 8957 wider sync scope 8958 and memory ordering 8959 stronger than 8960 unordered (this is 8961 termed the 8962 fence-paired-atomic). 8963 - s_waitcnt lgkmcnt(0) 8964 must happen after 8965 any preceding 8966 local/generic load 8967 atomic/atomicrmw 8968 with an equal or 8969 wider sync scope 8970 and memory ordering 8971 stronger than 8972 unordered (this is 8973 termed the 8974 fence-paired-atomic). 8975 - Must happen before 8976 the following 8977 buffer_gl0_inv. 8978 - Ensures that the 8979 fence-paired atomic 8980 has completed 8981 before invalidating 8982 the 8983 cache. Therefore 8984 any following 8985 locations read must 8986 be no older than 8987 the value read by 8988 the 8989 fence-paired-atomic. 8990 8991 3. buffer_gl0_inv 8992 8993 - If CU wavefront execution 8994 mode, omit. 8995 - Ensures that 8996 following 8997 loads will not see 8998 stale data. 8999 9000 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9001 - system vmcnt(0) & vscnt(0) 9002 9003 - If OpenCL and 9004 address space is 9005 not generic, omit 9006 lgkmcnt(0). 9007 - If OpenCL and 9008 address space is 9009 local, omit 9010 vmcnt(0) and vscnt(0). 9011 - However, since LLVM 9012 currently has no 9013 address space on 9014 the fence need to 9015 conservatively 9016 always generate 9017 (see comment for 9018 previous fence). 9019 - Could be split into 9020 separate s_waitcnt 9021 vmcnt(0), s_waitcnt 9022 vscnt(0) and s_waitcnt 9023 lgkmcnt(0) to allow 9024 them to be 9025 independently moved 9026 according to the 9027 following rules. 9028 - s_waitcnt vmcnt(0) 9029 must happen after 9030 any preceding 9031 global/generic load 9032 atomic/ 9033 atomicrmw-with-return-value 9034 with an equal or 9035 wider sync scope 9036 and memory ordering 9037 stronger than 9038 unordered (this is 9039 termed the 9040 fence-paired-atomic). 9041 - s_waitcnt vscnt(0) 9042 must happen after 9043 any preceding 9044 global/generic 9045 atomicrmw-no-return-value 9046 with an equal or 9047 wider sync scope 9048 and memory ordering 9049 stronger than 9050 unordered (this is 9051 termed the 9052 fence-paired-atomic). 9053 - s_waitcnt lgkmcnt(0) 9054 must happen after 9055 any preceding 9056 local/generic load 9057 atomic/atomicrmw 9058 with an equal or 9059 wider sync scope 9060 and memory ordering 9061 stronger than 9062 unordered (this is 9063 termed the 9064 fence-paired-atomic). 9065 - Must happen before 9066 the following 9067 buffer_gl*_inv. 9068 - Ensures that the 9069 fence-paired atomic 9070 has completed 9071 before invalidating 9072 the 9073 caches. Therefore 9074 any following 9075 locations read must 9076 be no older than 9077 the value read by 9078 the 9079 fence-paired-atomic. 9080 9081 2. buffer_gl0_inv; 9082 buffer_gl1_inv 9083 9084 - Must happen before any 9085 following global/generic 9086 load/load 9087 atomic/store/store 9088 atomic/atomicrmw. 9089 - Ensures that 9090 following loads 9091 will not see stale 9092 global data. 9093 9094 **Release Atomic** 9095 ------------------------------------------------------------------------------------ 9096 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9097 - wavefront - local 9098 - generic 9099 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9100 - generic vmcnt(0) & vscnt(0) 9101 9102 - If CU wavefront execution 9103 mode, omit vmcnt(0) and 9104 vscnt(0). 9105 - If OpenCL, omit 9106 lgkmcnt(0). 9107 - Could be split into 9108 separate s_waitcnt 9109 vmcnt(0), s_waitcnt 9110 vscnt(0) and s_waitcnt 9111 lgkmcnt(0) to allow 9112 them to be 9113 independently moved 9114 according to the 9115 following rules. 9116 - s_waitcnt vmcnt(0) 9117 must happen after 9118 any preceding 9119 global/generic load/load 9120 atomic/ 9121 atomicrmw-with-return-value. 9122 - s_waitcnt vscnt(0) 9123 must happen after 9124 any preceding 9125 global/generic 9126 store/store 9127 atomic/ 9128 atomicrmw-no-return-value. 9129 - s_waitcnt lgkmcnt(0) 9130 must happen after 9131 any preceding 9132 local/generic 9133 load/store/load 9134 atomic/store 9135 atomic/atomicrmw. 9136 - Must happen before 9137 the following 9138 store. 9139 - Ensures that all 9140 memory operations 9141 have 9142 completed before 9143 performing the 9144 store that is being 9145 released. 9146 9147 2. buffer/global/flat_store 9148 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9149 9150 - If CU wavefront execution 9151 mode, omit. 9152 - If OpenCL, omit. 9153 - Could be split into 9154 separate s_waitcnt 9155 vmcnt(0) and s_waitcnt 9156 vscnt(0) to allow 9157 them to be 9158 independently moved 9159 according to the 9160 following rules. 9161 - s_waitcnt vmcnt(0) 9162 must happen after 9163 any preceding 9164 global/generic load/load 9165 atomic/ 9166 atomicrmw-with-return-value. 9167 - s_waitcnt vscnt(0) 9168 must happen after 9169 any preceding 9170 global/generic 9171 store/store atomic/ 9172 atomicrmw-no-return-value. 9173 - Must happen before 9174 the following 9175 store. 9176 - Ensures that all 9177 global memory 9178 operations have 9179 completed before 9180 performing the 9181 store that is being 9182 released. 9183 9184 2. ds_store 9185 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9186 - system - generic vmcnt(0) & vscnt(0) 9187 9188 - If OpenCL and 9189 address space is 9190 not generic, omit 9191 lgkmcnt(0). 9192 - Could be split into 9193 separate s_waitcnt 9194 vmcnt(0), s_waitcnt vscnt(0) 9195 and s_waitcnt 9196 lgkmcnt(0) to allow 9197 them to be 9198 independently moved 9199 according to the 9200 following rules. 9201 - s_waitcnt vmcnt(0) 9202 must happen after 9203 any preceding 9204 global/generic 9205 load/load 9206 atomic/ 9207 atomicrmw-with-return-value. 9208 - s_waitcnt vscnt(0) 9209 must happen after 9210 any preceding 9211 global/generic 9212 store/store atomic/ 9213 atomicrmw-no-return-value. 9214 - s_waitcnt lgkmcnt(0) 9215 must happen after 9216 any preceding 9217 local/generic 9218 load/store/load 9219 atomic/store 9220 atomic/atomicrmw. 9221 - Must happen before 9222 the following 9223 store. 9224 - Ensures that all 9225 memory operations 9226 have 9227 completed before 9228 performing the 9229 store that is being 9230 released. 9231 9232 2. buffer/global/flat_store 9233 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9234 - wavefront - local 9235 - generic 9236 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9237 - generic vmcnt(0) & vscnt(0) 9238 9239 - If CU wavefront execution 9240 mode, omit vmcnt(0) and 9241 vscnt(0). 9242 - If OpenCL, omit lgkmcnt(0). 9243 - Could be split into 9244 separate s_waitcnt 9245 vmcnt(0), s_waitcnt 9246 vscnt(0) and s_waitcnt 9247 lgkmcnt(0) to allow 9248 them to be 9249 independently moved 9250 according to the 9251 following rules. 9252 - s_waitcnt vmcnt(0) 9253 must happen after 9254 any preceding 9255 global/generic load/load 9256 atomic/ 9257 atomicrmw-with-return-value. 9258 - s_waitcnt vscnt(0) 9259 must happen after 9260 any preceding 9261 global/generic 9262 store/store 9263 atomic/ 9264 atomicrmw-no-return-value. 9265 - s_waitcnt lgkmcnt(0) 9266 must happen after 9267 any preceding 9268 local/generic 9269 load/store/load 9270 atomic/store 9271 atomic/atomicrmw. 9272 - Must happen before 9273 the following 9274 atomicrmw. 9275 - Ensures that all 9276 memory operations 9277 have 9278 completed before 9279 performing the 9280 atomicrmw that is 9281 being released. 9282 9283 2. buffer/global/flat_atomic 9284 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9285 9286 - If CU wavefront execution 9287 mode, omit. 9288 - If OpenCL, omit. 9289 - Could be split into 9290 separate s_waitcnt 9291 vmcnt(0) and s_waitcnt 9292 vscnt(0) to allow 9293 them to be 9294 independently moved 9295 according to the 9296 following rules. 9297 - s_waitcnt vmcnt(0) 9298 must happen after 9299 any preceding 9300 global/generic load/load 9301 atomic/ 9302 atomicrmw-with-return-value. 9303 - s_waitcnt vscnt(0) 9304 must happen after 9305 any preceding 9306 global/generic 9307 store/store atomic/ 9308 atomicrmw-no-return-value. 9309 - Must happen before 9310 the following 9311 store. 9312 - Ensures that all 9313 global memory 9314 operations have 9315 completed before 9316 performing the 9317 store that is being 9318 released. 9319 9320 2. ds_atomic 9321 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9322 - system - generic vmcnt(0) & vscnt(0) 9323 9324 - If OpenCL, omit 9325 lgkmcnt(0). 9326 - Could be split into 9327 separate s_waitcnt 9328 vmcnt(0), s_waitcnt 9329 vscnt(0) and s_waitcnt 9330 lgkmcnt(0) to allow 9331 them to be 9332 independently moved 9333 according to the 9334 following rules. 9335 - s_waitcnt vmcnt(0) 9336 must happen after 9337 any preceding 9338 global/generic 9339 load/load atomic/ 9340 atomicrmw-with-return-value. 9341 - s_waitcnt vscnt(0) 9342 must happen after 9343 any preceding 9344 global/generic 9345 store/store atomic/ 9346 atomicrmw-no-return-value. 9347 - s_waitcnt lgkmcnt(0) 9348 must happen after 9349 any preceding 9350 local/generic 9351 load/store/load 9352 atomic/store 9353 atomic/atomicrmw. 9354 - Must happen before 9355 the following 9356 atomicrmw. 9357 - Ensures that all 9358 memory operations 9359 to global and local 9360 have completed 9361 before performing 9362 the atomicrmw that 9363 is being released. 9364 9365 2. buffer/global/flat_atomic 9366 fence release - singlethread *none* *none* 9367 - wavefront 9368 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9369 vmcnt(0) & vscnt(0) 9370 9371 - If CU wavefront execution 9372 mode, omit vmcnt(0) and 9373 vscnt(0). 9374 - If OpenCL and 9375 address space is 9376 not generic, omit 9377 lgkmcnt(0). 9378 - If OpenCL and 9379 address space is 9380 local, omit 9381 vmcnt(0) and vscnt(0). 9382 - However, since LLVM 9383 currently has no 9384 address space on 9385 the fence need to 9386 conservatively 9387 always generate. If 9388 fence had an 9389 address space then 9390 set to address 9391 space of OpenCL 9392 fence flag, or to 9393 generic if both 9394 local and global 9395 flags are 9396 specified. 9397 - Could be split into 9398 separate s_waitcnt 9399 vmcnt(0), s_waitcnt 9400 vscnt(0) and s_waitcnt 9401 lgkmcnt(0) to allow 9402 them to be 9403 independently moved 9404 according to the 9405 following rules. 9406 - s_waitcnt vmcnt(0) 9407 must happen after 9408 any preceding 9409 global/generic 9410 load/load 9411 atomic/ 9412 atomicrmw-with-return-value. 9413 - s_waitcnt vscnt(0) 9414 must happen after 9415 any preceding 9416 global/generic 9417 store/store atomic/ 9418 atomicrmw-no-return-value. 9419 - s_waitcnt lgkmcnt(0) 9420 must happen after 9421 any preceding 9422 local/generic 9423 load/store/load 9424 atomic/store atomic/ 9425 atomicrmw. 9426 - Must happen before 9427 any following store 9428 atomic/atomicrmw 9429 with an equal or 9430 wider sync scope 9431 and memory ordering 9432 stronger than 9433 unordered (this is 9434 termed the 9435 fence-paired-atomic). 9436 - Ensures that all 9437 memory operations 9438 have 9439 completed before 9440 performing the 9441 following 9442 fence-paired-atomic. 9443 9444 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9445 - system vmcnt(0) & vscnt(0) 9446 9447 - If OpenCL and 9448 address space is 9449 not generic, omit 9450 lgkmcnt(0). 9451 - If OpenCL and 9452 address space is 9453 local, omit 9454 vmcnt(0) and vscnt(0). 9455 - However, since LLVM 9456 currently has no 9457 address space on 9458 the fence need to 9459 conservatively 9460 always generate. If 9461 fence had an 9462 address space then 9463 set to address 9464 space of OpenCL 9465 fence flag, or to 9466 generic if both 9467 local and global 9468 flags are 9469 specified. 9470 - Could be split into 9471 separate s_waitcnt 9472 vmcnt(0), s_waitcnt 9473 vscnt(0) and s_waitcnt 9474 lgkmcnt(0) to allow 9475 them to be 9476 independently moved 9477 according to the 9478 following rules. 9479 - s_waitcnt vmcnt(0) 9480 must happen after 9481 any preceding 9482 global/generic 9483 load/load atomic/ 9484 atomicrmw-with-return-value. 9485 - s_waitcnt vscnt(0) 9486 must happen after 9487 any preceding 9488 global/generic 9489 store/store atomic/ 9490 atomicrmw-no-return-value. 9491 - s_waitcnt lgkmcnt(0) 9492 must happen after 9493 any preceding 9494 local/generic 9495 load/store/load 9496 atomic/store 9497 atomic/atomicrmw. 9498 - Must happen before 9499 any following store 9500 atomic/atomicrmw 9501 with an equal or 9502 wider sync scope 9503 and memory ordering 9504 stronger than 9505 unordered (this is 9506 termed the 9507 fence-paired-atomic). 9508 - Ensures that all 9509 memory operations 9510 have 9511 completed before 9512 performing the 9513 following 9514 fence-paired-atomic. 9515 9516 **Acquire-Release Atomic** 9517 ------------------------------------------------------------------------------------ 9518 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9519 - wavefront - local 9520 - generic 9521 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9522 vmcnt(0) & vscnt(0) 9523 9524 - If CU wavefront execution 9525 mode, omit vmcnt(0) and 9526 vscnt(0). 9527 - If OpenCL, omit 9528 lgkmcnt(0). 9529 - Must happen after 9530 any preceding 9531 local/generic 9532 load/store/load 9533 atomic/store 9534 atomic/atomicrmw. 9535 - Could be split into 9536 separate s_waitcnt 9537 vmcnt(0), s_waitcnt 9538 vscnt(0), and s_waitcnt 9539 lgkmcnt(0) to allow 9540 them to be 9541 independently moved 9542 according to the 9543 following rules. 9544 - s_waitcnt vmcnt(0) 9545 must happen after 9546 any preceding 9547 global/generic load/load 9548 atomic/ 9549 atomicrmw-with-return-value. 9550 - s_waitcnt vscnt(0) 9551 must happen after 9552 any preceding 9553 global/generic 9554 store/store 9555 atomic/ 9556 atomicrmw-no-return-value. 9557 - s_waitcnt lgkmcnt(0) 9558 must happen after 9559 any preceding 9560 local/generic 9561 load/store/load 9562 atomic/store 9563 atomic/atomicrmw. 9564 - Must happen before 9565 the following 9566 atomicrmw. 9567 - Ensures that all 9568 memory operations 9569 have 9570 completed before 9571 performing the 9572 atomicrmw that is 9573 being released. 9574 9575 2. buffer/global_atomic 9576 3. s_waitcnt vm/vscnt(0) 9577 9578 - If CU wavefront execution 9579 mode, omit. 9580 - Use vmcnt(0) if atomic with 9581 return and vscnt(0) if 9582 atomic with no-return. 9583 - Must happen before 9584 the following 9585 buffer_gl0_inv. 9586 - Ensures any 9587 following global 9588 data read is no 9589 older than the 9590 atomicrmw value 9591 being acquired. 9592 9593 4. buffer_gl0_inv 9594 9595 - If CU wavefront execution 9596 mode, omit. 9597 - Ensures that 9598 following 9599 loads will not see 9600 stale data. 9601 9602 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9603 9604 - If CU wavefront execution 9605 mode, omit. 9606 - If OpenCL, omit. 9607 - Could be split into 9608 separate s_waitcnt 9609 vmcnt(0) and s_waitcnt 9610 vscnt(0) to allow 9611 them to be 9612 independently moved 9613 according to the 9614 following rules. 9615 - s_waitcnt vmcnt(0) 9616 must happen after 9617 any preceding 9618 global/generic load/load 9619 atomic/ 9620 atomicrmw-with-return-value. 9621 - s_waitcnt vscnt(0) 9622 must happen after 9623 any preceding 9624 global/generic 9625 store/store atomic/ 9626 atomicrmw-no-return-value. 9627 - Must happen before 9628 the following 9629 store. 9630 - Ensures that all 9631 global memory 9632 operations have 9633 completed before 9634 performing the 9635 store that is being 9636 released. 9637 9638 2. ds_atomic 9639 3. s_waitcnt lgkmcnt(0) 9640 9641 - If OpenCL, omit. 9642 - Must happen before 9643 the following 9644 buffer_gl0_inv. 9645 - Ensures any 9646 following global 9647 data read is no 9648 older than the local load 9649 atomic value being 9650 acquired. 9651 9652 4. buffer_gl0_inv 9653 9654 - If CU wavefront execution 9655 mode, omit. 9656 - If OpenCL omit. 9657 - Ensures that 9658 following 9659 loads will not see 9660 stale data. 9661 9662 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9663 vmcnt(0) & vscnt(0) 9664 9665 - If CU wavefront execution 9666 mode, omit vmcnt(0) and 9667 vscnt(0). 9668 - If OpenCL, omit lgkmcnt(0). 9669 - Could be split into 9670 separate s_waitcnt 9671 vmcnt(0), s_waitcnt 9672 vscnt(0) and s_waitcnt 9673 lgkmcnt(0) to allow 9674 them to be 9675 independently moved 9676 according to the 9677 following rules. 9678 - s_waitcnt vmcnt(0) 9679 must happen after 9680 any preceding 9681 global/generic load/load 9682 atomic/ 9683 atomicrmw-with-return-value. 9684 - s_waitcnt vscnt(0) 9685 must happen after 9686 any preceding 9687 global/generic 9688 store/store 9689 atomic/ 9690 atomicrmw-no-return-value. 9691 - s_waitcnt lgkmcnt(0) 9692 must happen after 9693 any preceding 9694 local/generic 9695 load/store/load 9696 atomic/store 9697 atomic/atomicrmw. 9698 - Must happen before 9699 the following 9700 atomicrmw. 9701 - Ensures that all 9702 memory operations 9703 have 9704 completed before 9705 performing the 9706 atomicrmw that is 9707 being released. 9708 9709 2. flat_atomic 9710 3. s_waitcnt lgkmcnt(0) & 9711 vmcnt(0) & vscnt(0) 9712 9713 - If CU wavefront execution 9714 mode, omit vmcnt(0) and 9715 vscnt(0). 9716 - If OpenCL, omit lgkmcnt(0). 9717 - Must happen before 9718 the following 9719 buffer_gl0_inv. 9720 - Ensures any 9721 following global 9722 data read is no 9723 older than the load 9724 atomic value being 9725 acquired. 9726 9727 3. buffer_gl0_inv 9728 9729 - If CU wavefront execution 9730 mode, omit. 9731 - Ensures that 9732 following 9733 loads will not see 9734 stale data. 9735 9736 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9737 - system vmcnt(0) & vscnt(0) 9738 9739 - If OpenCL, omit 9740 lgkmcnt(0). 9741 - Could be split into 9742 separate s_waitcnt 9743 vmcnt(0), s_waitcnt 9744 vscnt(0) and s_waitcnt 9745 lgkmcnt(0) to allow 9746 them to be 9747 independently moved 9748 according to the 9749 following rules. 9750 - s_waitcnt vmcnt(0) 9751 must happen after 9752 any preceding 9753 global/generic 9754 load/load atomic/ 9755 atomicrmw-with-return-value. 9756 - s_waitcnt vscnt(0) 9757 must happen after 9758 any preceding 9759 global/generic 9760 store/store atomic/ 9761 atomicrmw-no-return-value. 9762 - s_waitcnt lgkmcnt(0) 9763 must happen after 9764 any preceding 9765 local/generic 9766 load/store/load 9767 atomic/store 9768 atomic/atomicrmw. 9769 - Must happen before 9770 the following 9771 atomicrmw. 9772 - Ensures that all 9773 memory operations 9774 to global have 9775 completed before 9776 performing the 9777 atomicrmw that is 9778 being released. 9779 9780 2. buffer/global_atomic 9781 3. s_waitcnt vm/vscnt(0) 9782 9783 - Use vmcnt(0) if atomic with 9784 return and vscnt(0) if 9785 atomic with no-return. 9786 - Must happen before 9787 following 9788 buffer_gl*_inv. 9789 - Ensures the 9790 atomicrmw has 9791 completed before 9792 invalidating the 9793 caches. 9794 9795 4. buffer_gl0_inv; 9796 buffer_gl1_inv 9797 9798 - Must happen before 9799 any following 9800 global/generic 9801 load/load 9802 atomic/atomicrmw. 9803 - Ensures that 9804 following loads 9805 will not see stale 9806 global data. 9807 9808 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9809 - system vmcnt(0) & vscnt(0) 9810 9811 - If OpenCL, omit 9812 lgkmcnt(0). 9813 - Could be split into 9814 separate s_waitcnt 9815 vmcnt(0), s_waitcnt 9816 vscnt(0), and s_waitcnt 9817 lgkmcnt(0) to allow 9818 them to be 9819 independently moved 9820 according to the 9821 following rules. 9822 - s_waitcnt vmcnt(0) 9823 must happen after 9824 any preceding 9825 global/generic 9826 load/load atomic 9827 atomicrmw-with-return-value. 9828 - s_waitcnt vscnt(0) 9829 must happen after 9830 any preceding 9831 global/generic 9832 store/store atomic/ 9833 atomicrmw-no-return-value. 9834 - s_waitcnt lgkmcnt(0) 9835 must happen after 9836 any preceding 9837 local/generic 9838 load/store/load 9839 atomic/store 9840 atomic/atomicrmw. 9841 - Must happen before 9842 the following 9843 atomicrmw. 9844 - Ensures that all 9845 memory operations 9846 have 9847 completed before 9848 performing the 9849 atomicrmw that is 9850 being released. 9851 9852 2. flat_atomic 9853 3. s_waitcnt vm/vscnt(0) & 9854 lgkmcnt(0) 9855 9856 - If OpenCL, omit 9857 lgkmcnt(0). 9858 - Use vmcnt(0) if atomic with 9859 return and vscnt(0) if 9860 atomic with no-return. 9861 - Must happen before 9862 following 9863 buffer_gl*_inv. 9864 - Ensures the 9865 atomicrmw has 9866 completed before 9867 invalidating the 9868 caches. 9869 9870 4. buffer_gl0_inv; 9871 buffer_gl1_inv 9872 9873 - Must happen before 9874 any following 9875 global/generic 9876 load/load 9877 atomic/atomicrmw. 9878 - Ensures that 9879 following loads 9880 will not see stale 9881 global data. 9882 9883 fence acq_rel - singlethread *none* *none* 9884 - wavefront 9885 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9886 vmcnt(0) & vscnt(0) 9887 9888 - If CU wavefront execution 9889 mode, omit vmcnt(0) and 9890 vscnt(0). 9891 - If OpenCL and 9892 address space is 9893 not generic, omit 9894 lgkmcnt(0). 9895 - If OpenCL and 9896 address space is 9897 local, omit 9898 vmcnt(0) and vscnt(0). 9899 - However, 9900 since LLVM 9901 currently has no 9902 address space on 9903 the fence need to 9904 conservatively 9905 always generate 9906 (see comment for 9907 previous fence). 9908 - Could be split into 9909 separate s_waitcnt 9910 vmcnt(0), s_waitcnt 9911 vscnt(0) and s_waitcnt 9912 lgkmcnt(0) to allow 9913 them to be 9914 independently moved 9915 according to the 9916 following rules. 9917 - s_waitcnt vmcnt(0) 9918 must happen after 9919 any preceding 9920 global/generic 9921 load/load 9922 atomic/ 9923 atomicrmw-with-return-value. 9924 - s_waitcnt vscnt(0) 9925 must happen after 9926 any preceding 9927 global/generic 9928 store/store atomic/ 9929 atomicrmw-no-return-value. 9930 - s_waitcnt lgkmcnt(0) 9931 must happen after 9932 any preceding 9933 local/generic 9934 load/store/load 9935 atomic/store atomic/ 9936 atomicrmw. 9937 - Must happen before 9938 any following 9939 global/generic 9940 load/load 9941 atomic/store/store 9942 atomic/atomicrmw. 9943 - Ensures that all 9944 memory operations 9945 have 9946 completed before 9947 performing any 9948 following global 9949 memory operations. 9950 - Ensures that the 9951 preceding 9952 local/generic load 9953 atomic/atomicrmw 9954 with an equal or 9955 wider sync scope 9956 and memory ordering 9957 stronger than 9958 unordered (this is 9959 termed the 9960 acquire-fence-paired-atomic) 9961 has completed 9962 before following 9963 global memory 9964 operations. This 9965 satisfies the 9966 requirements of 9967 acquire. 9968 - Ensures that all 9969 previous memory 9970 operations have 9971 completed before a 9972 following 9973 local/generic store 9974 atomic/atomicrmw 9975 with an equal or 9976 wider sync scope 9977 and memory ordering 9978 stronger than 9979 unordered (this is 9980 termed the 9981 release-fence-paired-atomic). 9982 This satisfies the 9983 requirements of 9984 release. 9985 - Must happen before 9986 the following 9987 buffer_gl0_inv. 9988 - Ensures that the 9989 acquire-fence-paired 9990 atomic has completed 9991 before invalidating 9992 the 9993 cache. Therefore 9994 any following 9995 locations read must 9996 be no older than 9997 the value read by 9998 the 9999 acquire-fence-paired-atomic. 10000 10001 3. buffer_gl0_inv 10002 10003 - If CU wavefront execution 10004 mode, omit. 10005 - Ensures that 10006 following 10007 loads will not see 10008 stale data. 10009 10010 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 10011 - system vmcnt(0) & vscnt(0) 10012 10013 - If OpenCL and 10014 address space is 10015 not generic, omit 10016 lgkmcnt(0). 10017 - If OpenCL and 10018 address space is 10019 local, omit 10020 vmcnt(0) and vscnt(0). 10021 - However, since LLVM 10022 currently has no 10023 address space on 10024 the fence need to 10025 conservatively 10026 always generate 10027 (see comment for 10028 previous fence). 10029 - Could be split into 10030 separate s_waitcnt 10031 vmcnt(0), s_waitcnt 10032 vscnt(0) and s_waitcnt 10033 lgkmcnt(0) to allow 10034 them to be 10035 independently moved 10036 according to the 10037 following rules. 10038 - s_waitcnt vmcnt(0) 10039 must happen after 10040 any preceding 10041 global/generic 10042 load/load 10043 atomic/ 10044 atomicrmw-with-return-value. 10045 - s_waitcnt vscnt(0) 10046 must happen after 10047 any preceding 10048 global/generic 10049 store/store atomic/ 10050 atomicrmw-no-return-value. 10051 - s_waitcnt lgkmcnt(0) 10052 must happen after 10053 any preceding 10054 local/generic 10055 load/store/load 10056 atomic/store 10057 atomic/atomicrmw. 10058 - Must happen before 10059 the following 10060 buffer_gl*_inv. 10061 - Ensures that the 10062 preceding 10063 global/local/generic 10064 load 10065 atomic/atomicrmw 10066 with an equal or 10067 wider sync scope 10068 and memory ordering 10069 stronger than 10070 unordered (this is 10071 termed the 10072 acquire-fence-paired-atomic) 10073 has completed 10074 before invalidating 10075 the caches. This 10076 satisfies the 10077 requirements of 10078 acquire. 10079 - Ensures that all 10080 previous memory 10081 operations have 10082 completed before a 10083 following 10084 global/local/generic 10085 store 10086 atomic/atomicrmw 10087 with an equal or 10088 wider sync scope 10089 and memory ordering 10090 stronger than 10091 unordered (this is 10092 termed the 10093 release-fence-paired-atomic). 10094 This satisfies the 10095 requirements of 10096 release. 10097 10098 2. buffer_gl0_inv; 10099 buffer_gl1_inv 10100 10101 - Must happen before 10102 any following 10103 global/generic 10104 load/load 10105 atomic/store/store 10106 atomic/atomicrmw. 10107 - Ensures that 10108 following loads 10109 will not see stale 10110 global data. This 10111 satisfies the 10112 requirements of 10113 acquire. 10114 10115 **Sequential Consistent Atomic** 10116 ------------------------------------------------------------------------------------ 10117 load atomic seq_cst - singlethread - global *Same as corresponding 10118 - wavefront - local load atomic acquire, 10119 - generic except must generated 10120 all instructions even 10121 for OpenCL.* 10122 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10123 - generic vmcnt(0) & vscnt(0) 10124 10125 - If CU wavefront execution 10126 mode, omit vmcnt(0) and 10127 vscnt(0). 10128 - Could be split into 10129 separate s_waitcnt 10130 vmcnt(0), s_waitcnt 10131 vscnt(0), and s_waitcnt 10132 lgkmcnt(0) to allow 10133 them to be 10134 independently moved 10135 according to the 10136 following rules. 10137 - s_waitcnt lgkmcnt(0) must 10138 happen after 10139 preceding 10140 local/generic load 10141 atomic/store 10142 atomic/atomicrmw 10143 with memory 10144 ordering of seq_cst 10145 and with equal or 10146 wider sync scope. 10147 (Note that seq_cst 10148 fences have their 10149 own s_waitcnt 10150 lgkmcnt(0) and so do 10151 not need to be 10152 considered.) 10153 - s_waitcnt vmcnt(0) 10154 must happen after 10155 preceding 10156 global/generic load 10157 atomic/ 10158 atomicrmw-with-return-value 10159 with memory 10160 ordering of seq_cst 10161 and with equal or 10162 wider sync scope. 10163 (Note that seq_cst 10164 fences have their 10165 own s_waitcnt 10166 vmcnt(0) and so do 10167 not need to be 10168 considered.) 10169 - s_waitcnt vscnt(0) 10170 Must happen after 10171 preceding 10172 global/generic store 10173 atomic/ 10174 atomicrmw-no-return-value 10175 with memory 10176 ordering of seq_cst 10177 and with equal or 10178 wider sync scope. 10179 (Note that seq_cst 10180 fences have their 10181 own s_waitcnt 10182 vscnt(0) and so do 10183 not need to be 10184 considered.) 10185 - Ensures any 10186 preceding 10187 sequential 10188 consistent global/local 10189 memory instructions 10190 have completed 10191 before executing 10192 this sequentially 10193 consistent 10194 instruction. This 10195 prevents reordering 10196 a seq_cst store 10197 followed by a 10198 seq_cst load. (Note 10199 that seq_cst is 10200 stronger than 10201 acquire/release as 10202 the reordering of 10203 load acquire 10204 followed by a store 10205 release is 10206 prevented by the 10207 s_waitcnt of 10208 the release, but 10209 there is nothing 10210 preventing a store 10211 release followed by 10212 load acquire from 10213 completing out of 10214 order. The s_waitcnt 10215 could be placed after 10216 seq_store or before 10217 the seq_load. We 10218 choose the load to 10219 make the s_waitcnt be 10220 as late as possible 10221 so that the store 10222 may have already 10223 completed.) 10224 10225 2. *Following 10226 instructions same as 10227 corresponding load 10228 atomic acquire, 10229 except must generated 10230 all instructions even 10231 for OpenCL.* 10232 load atomic seq_cst - workgroup - local 10233 10234 1. s_waitcnt vmcnt(0) & vscnt(0) 10235 10236 - If CU wavefront execution 10237 mode, omit. 10238 - Could be split into 10239 separate s_waitcnt 10240 vmcnt(0) and s_waitcnt 10241 vscnt(0) to allow 10242 them to be 10243 independently moved 10244 according to the 10245 following rules. 10246 - s_waitcnt vmcnt(0) 10247 Must happen after 10248 preceding 10249 global/generic load 10250 atomic/ 10251 atomicrmw-with-return-value 10252 with memory 10253 ordering of seq_cst 10254 and with equal or 10255 wider sync scope. 10256 (Note that seq_cst 10257 fences have their 10258 own s_waitcnt 10259 vmcnt(0) and so do 10260 not need to be 10261 considered.) 10262 - s_waitcnt vscnt(0) 10263 Must happen after 10264 preceding 10265 global/generic store 10266 atomic/ 10267 atomicrmw-no-return-value 10268 with memory 10269 ordering of seq_cst 10270 and with equal or 10271 wider sync scope. 10272 (Note that seq_cst 10273 fences have their 10274 own s_waitcnt 10275 vscnt(0) and so do 10276 not need to be 10277 considered.) 10278 - Ensures any 10279 preceding 10280 sequential 10281 consistent global 10282 memory instructions 10283 have completed 10284 before executing 10285 this sequentially 10286 consistent 10287 instruction. This 10288 prevents reordering 10289 a seq_cst store 10290 followed by a 10291 seq_cst load. (Note 10292 that seq_cst is 10293 stronger than 10294 acquire/release as 10295 the reordering of 10296 load acquire 10297 followed by a store 10298 release is 10299 prevented by the 10300 s_waitcnt of 10301 the release, but 10302 there is nothing 10303 preventing a store 10304 release followed by 10305 load acquire from 10306 completing out of 10307 order. The s_waitcnt 10308 could be placed after 10309 seq_store or before 10310 the seq_load. We 10311 choose the load to 10312 make the s_waitcnt be 10313 as late as possible 10314 so that the store 10315 may have already 10316 completed.) 10317 10318 2. *Following 10319 instructions same as 10320 corresponding load 10321 atomic acquire, 10322 except must generated 10323 all instructions even 10324 for OpenCL.* 10325 10326 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10327 - system - generic vmcnt(0) & vscnt(0) 10328 10329 - Could be split into 10330 separate s_waitcnt 10331 vmcnt(0), s_waitcnt 10332 vscnt(0) and s_waitcnt 10333 lgkmcnt(0) to allow 10334 them to be 10335 independently moved 10336 according to the 10337 following rules. 10338 - s_waitcnt lgkmcnt(0) 10339 must happen after 10340 preceding 10341 local load 10342 atomic/store 10343 atomic/atomicrmw 10344 with memory 10345 ordering of seq_cst 10346 and with equal or 10347 wider sync scope. 10348 (Note that seq_cst 10349 fences have their 10350 own s_waitcnt 10351 lgkmcnt(0) and so do 10352 not need to be 10353 considered.) 10354 - s_waitcnt vmcnt(0) 10355 must happen after 10356 preceding 10357 global/generic load 10358 atomic/ 10359 atomicrmw-with-return-value 10360 with memory 10361 ordering of seq_cst 10362 and with equal or 10363 wider sync scope. 10364 (Note that seq_cst 10365 fences have their 10366 own s_waitcnt 10367 vmcnt(0) and so do 10368 not need to be 10369 considered.) 10370 - s_waitcnt vscnt(0) 10371 Must happen after 10372 preceding 10373 global/generic store 10374 atomic/ 10375 atomicrmw-no-return-value 10376 with memory 10377 ordering of seq_cst 10378 and with equal or 10379 wider sync scope. 10380 (Note that seq_cst 10381 fences have their 10382 own s_waitcnt 10383 vscnt(0) and so do 10384 not need to be 10385 considered.) 10386 - Ensures any 10387 preceding 10388 sequential 10389 consistent global 10390 memory instructions 10391 have completed 10392 before executing 10393 this sequentially 10394 consistent 10395 instruction. This 10396 prevents reordering 10397 a seq_cst store 10398 followed by a 10399 seq_cst load. (Note 10400 that seq_cst is 10401 stronger than 10402 acquire/release as 10403 the reordering of 10404 load acquire 10405 followed by a store 10406 release is 10407 prevented by the 10408 s_waitcnt of 10409 the release, but 10410 there is nothing 10411 preventing a store 10412 release followed by 10413 load acquire from 10414 completing out of 10415 order. The s_waitcnt 10416 could be placed after 10417 seq_store or before 10418 the seq_load. We 10419 choose the load to 10420 make the s_waitcnt be 10421 as late as possible 10422 so that the store 10423 may have already 10424 completed.) 10425 10426 2. *Following 10427 instructions same as 10428 corresponding load 10429 atomic acquire, 10430 except must generated 10431 all instructions even 10432 for OpenCL.* 10433 store atomic seq_cst - singlethread - global *Same as corresponding 10434 - wavefront - local store atomic release, 10435 - workgroup - generic except must generated 10436 - agent all instructions even 10437 - system for OpenCL.* 10438 atomicrmw seq_cst - singlethread - global *Same as corresponding 10439 - wavefront - local atomicrmw acq_rel, 10440 - workgroup - generic except must generated 10441 - agent all instructions even 10442 - system for OpenCL.* 10443 fence seq_cst - singlethread *none* *Same as corresponding 10444 - wavefront fence acq_rel, 10445 - workgroup except must generated 10446 - agent all instructions even 10447 - system for OpenCL.* 10448 ============ ============ ============== ========== ================================ 10449 10450Trap Handler ABI 10451~~~~~~~~~~~~~~~~ 10452 10453For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10454runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10455supports the ``s_trap`` instruction. For usage see: 10456 10457- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10458- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10459- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10460 10461 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10462 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10463 10464 =================== =============== =============== ======================================= 10465 Usage Code Sequence Trap Handler Description 10466 Inputs 10467 =================== =============== =============== ======================================= 10468 reserved ``s_trap 0x00`` Reserved by hardware. 10469 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10470 ``queue_ptr`` intrinsic (not implemented). 10471 ``VGPR0``: 10472 ``arg`` 10473 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10474 ``queue_ptr`` the trap instruction. The associated 10475 queue is signalled to put it into the 10476 error state. When the queue is put in 10477 the error state, the waves executing 10478 dispatches on the queue will be 10479 terminated. 10480 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10481 as a no-operation. The trap handler 10482 is entered and immediately returns to 10483 continue execution of the wavefront. 10484 - If the debugger is enabled, causes 10485 the debug trap to be reported by the 10486 debugger and the wavefront is put in 10487 the halt state with the PC at the 10488 instruction. The debugger must 10489 increment the PC and resume the wave. 10490 reserved ``s_trap 0x04`` Reserved. 10491 reserved ``s_trap 0x05`` Reserved. 10492 reserved ``s_trap 0x06`` Reserved. 10493 reserved ``s_trap 0x07`` Reserved. 10494 reserved ``s_trap 0x08`` Reserved. 10495 reserved ``s_trap 0xfe`` Reserved. 10496 reserved ``s_trap 0xff`` Reserved. 10497 =================== =============== =============== ======================================= 10498 10499.. 10500 10501 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10502 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10503 10504 =================== =============== =============== ======================================= 10505 Usage Code Sequence Trap Handler Description 10506 Inputs 10507 =================== =============== =============== ======================================= 10508 reserved ``s_trap 0x00`` Reserved by hardware. 10509 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10510 breakpoints. Causes wave to be halted 10511 with the PC at the trap instruction. 10512 The debugger is responsible to resume 10513 the wave, including the instruction 10514 that the breakpoint overwrote. 10515 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10516 ``queue_ptr`` the trap instruction. The associated 10517 queue is signalled to put it into the 10518 error state. When the queue is put in 10519 the error state, the waves executing 10520 dispatches on the queue will be 10521 terminated. 10522 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10523 as a no-operation. The trap handler 10524 is entered and immediately returns to 10525 continue execution of the wavefront. 10526 - If the debugger is enabled, causes 10527 the debug trap to be reported by the 10528 debugger and the wavefront is put in 10529 the halt state with the PC at the 10530 instruction. The debugger must 10531 increment the PC and resume the wave. 10532 reserved ``s_trap 0x04`` Reserved. 10533 reserved ``s_trap 0x05`` Reserved. 10534 reserved ``s_trap 0x06`` Reserved. 10535 reserved ``s_trap 0x07`` Reserved. 10536 reserved ``s_trap 0x08`` Reserved. 10537 reserved ``s_trap 0xfe`` Reserved. 10538 reserved ``s_trap 0xff`` Reserved. 10539 =================== =============== =============== ======================================= 10540 10541.. 10542 10543 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10544 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10545 10546 =================== =============== ================ ================= ======================================= 10547 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10548 =================== =============== ================ ================= ======================================= 10549 reserved ``s_trap 0x00`` Reserved by hardware. 10550 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10551 breakpoints. Causes wave to be halted 10552 with the PC at the trap instruction. 10553 The debugger is responsible to resume 10554 the wave, including the instruction 10555 that the breakpoint overwrote. 10556 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10557 ``queue_ptr`` the trap instruction. The associated 10558 queue is signalled to put it into the 10559 error state. When the queue is put in 10560 the error state, the waves executing 10561 dispatches on the queue will be 10562 terminated. 10563 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10564 as a no-operation. The trap handler 10565 is entered and immediately returns to 10566 continue execution of the wavefront. 10567 - If the debugger is enabled, causes 10568 the debug trap to be reported by the 10569 debugger and the wavefront is put in 10570 the halt state with the PC at the 10571 instruction. The debugger must 10572 increment the PC and resume the wave. 10573 reserved ``s_trap 0x04`` Reserved. 10574 reserved ``s_trap 0x05`` Reserved. 10575 reserved ``s_trap 0x06`` Reserved. 10576 reserved ``s_trap 0x07`` Reserved. 10577 reserved ``s_trap 0x08`` Reserved. 10578 reserved ``s_trap 0xfe`` Reserved. 10579 reserved ``s_trap 0xff`` Reserved. 10580 =================== =============== ================ ================= ======================================= 10581 10582.. _amdgpu-amdhsa-function-call-convention: 10583 10584Call Convention 10585~~~~~~~~~~~~~~~ 10586 10587.. note:: 10588 10589 This section is currently incomplete and has inaccuracies. It is WIP that will 10590 be updated as information is determined. 10591 10592See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10593addresses. Unswizzled addresses are normal linear addresses. 10594 10595.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10596 10597Kernel Functions 10598++++++++++++++++ 10599 10600This section describes the call convention ABI for the outer kernel function. 10601 10602See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10603convention. 10604 10605The following is not part of the AMDGPU kernel calling convention but describes 10606how the AMDGPU implements function calls: 10607 106081. Clang decides the kernarg layout to match the *HSA Programmer's Language 10609 Reference* [HSA]_. 10610 10611 - All structs are passed directly. 10612 - Lambda values are passed *TBA*. 10613 10614 .. TODO:: 10615 10616 - Does this really follow HSA rules? Or are structs >16 bytes passed 10617 by-value struct? 10618 - What is ABI for lambda values? 10619 106204. The kernel performs certain setup in its prolog, as described in 10621 :ref:`amdgpu-amdhsa-kernel-prolog`. 10622 10623.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10624 10625Non-Kernel Functions 10626++++++++++++++++++++ 10627 10628This section describes the call convention ABI for functions other than the 10629outer kernel function. 10630 10631If a kernel has function calls then scratch is always allocated and used for 10632the call stack which grows from low address to high address using the swizzled 10633scratch address space. 10634 10635On entry to a function: 10636 106371. SGPR0-3 contain a V# with the following properties (see 10638 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10639 10640 * Base address pointing to the beginning of the wavefront scratch backing 10641 memory. 10642 * Swizzled with dword element size and stride of wavefront size elements. 10643 106442. The FLAT_SCRATCH register pair is setup. See 10645 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 106463. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10647 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 106484. The EXEC register is set to the lanes active on entry to the function. 106495. MODE register: *TBD* 106506. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10651 below. 106527. SGPR30-31 return address (RA). The code address that the function must 10653 return to when it completes. The value is undefined if the function is *no 10654 return*. 106558. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10656 offset relative to the beginning of the wavefront scratch backing memory. 10657 10658 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10659 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10660 manner. 10661 10662 The unswizzled SP value can be converted into the swizzled SP value by: 10663 10664 | swizzled SP = unswizzled SP / wavefront size 10665 10666 This may be used to obtain the private address space address of stack 10667 objects and to convert this address to a flat address by adding the flat 10668 scratch aperture base address. 10669 10670 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10671 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10672 10673 .. note:: 10674 10675 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10676 OpenCL language which has the largest base type defined as 16 bytes. 10677 10678 On entry, the swizzled SP value is the address of the first function 10679 argument passed on the stack. Other stack passed arguments are positive 10680 offsets from the entry swizzled SP value. 10681 10682 The function may use positive offsets beyond the last stack passed argument 10683 for stack allocated local variables and register spill slots. If necessary, 10684 the function may align these to greater alignment than 16 bytes. After these 10685 the function may dynamically allocate space for such things as runtime sized 10686 ``alloca`` local allocations. 10687 10688 If the function calls another function, it will place any stack allocated 10689 arguments after the last local allocation and adjust SGPR32 to the address 10690 after the last local allocation. 10691 106929. All other registers are unspecified. 1069310. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10694 to the function. 10695 10696On exit from a function: 10697 106981. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10699 described below. Any registers used are considered clobbered registers. 107002. The following registers are preserved and have the same value as on entry: 10701 10702 * FLAT_SCRATCH 10703 * EXEC 10704 * GFX6-GFX8: M0 10705 * All SGPR registers except the clobbered registers of SGPR4-31. 10706 * VGPR40-47 10707 * VGPR56-63 10708 * VGPR72-79 10709 * VGPR88-95 10710 * VGPR104-111 10711 * VGPR120-127 10712 * VGPR136-143 10713 * VGPR152-159 10714 * VGPR168-175 10715 * VGPR184-191 10716 * VGPR200-207 10717 * VGPR216-223 10718 * VGPR232-239 10719 * VGPR248-255 10720 10721 .. note:: 10722 10723 Except the argument registers, the VGPRs clobbered and the preserved 10724 registers are intermixed at regular intervals in order to keep a 10725 similar ratio independent of the number of allocated VGPRs. 10726 10727 * Lanes of all VGPRs that are inactive at the call site. 10728 10729 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10730 optimization may mark some of clobbered SGPR and VGPR registers as 10731 preserved if it can be determined that the called function does not change 10732 their value. 10733 107342. The PC is set to the RA provided on entry. 107353. MODE register: *TBD*. 107364. All other registers are clobbered. 107375. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10738 function is available to the caller. 10739 10740.. TODO:: 10741 10742 - On gfx908 are all ACC registers clobbered? 10743 10744 - How are function results returned? The address of structured types is passed 10745 by reference, but what about other types? 10746 10747The function input arguments are made up of the formal arguments explicitly 10748declared by the source language function plus the implicit input arguments used 10749by the implementation. 10750 10751The source language input arguments are: 10752 107531. Any source language implicit ``this`` or ``self`` argument comes first as a 10754 pointer type. 107552. Followed by the function formal arguments in left to right source order. 10756 10757The source language result arguments are: 10758 107591. The function result argument. 10760 10761The source language input or result struct type arguments that are less than or 10762equal to 16 bytes, are decomposed recursively into their base type fields, and 10763each field is passed as if a separate argument. For input arguments, if the 10764called function requires the struct to be in memory, for example because its 10765address is taken, then the function body is responsible for allocating a stack 10766location and copying the field arguments into it. Clang terms this *direct 10767struct*. 10768 10769The source language input struct type arguments that are greater than 16 bytes, 10770are passed by reference. The caller is responsible for allocating a stack 10771location to make a copy of the struct value and pass the address as the input 10772argument. The called function is responsible to perform the dereference when 10773accessing the input argument. Clang terms this *by-value struct*. 10774 10775A source language result struct type argument that is greater than 16 bytes, is 10776returned by reference. The caller is responsible for allocating a stack location 10777to hold the result value and passes the address as the last input argument 10778(before the implicit input arguments). In this case there are no result 10779arguments. The called function is responsible to perform the dereference when 10780storing the result value. Clang terms this *structured return (sret)*. 10781 10782*TODO: correct the ``sret`` definition.* 10783 10784.. TODO:: 10785 10786 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10787 pass as non-decomposed struct as stack argument? Or something else? Is the 10788 memory location in the caller stack frame, or a stack memory argument and so 10789 no address is passed as the caller can directly write to the argument stack 10790 location? But then the stack location is still live after return. If an 10791 argument stack location is it the first stack argument or the last one? 10792 10793Lambda argument types are treated as struct types with an implementation defined 10794set of fields. 10795 10796.. TODO:: 10797 10798 Need to specify the ABI for lambda types for AMDGPU. 10799 10800For AMDGPU backend all source language arguments (including the decomposed 10801struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10802they are passed in SGPRs. 10803 10804The AMDGPU backend walks the function call graph from the leaves to determine 10805which implicit input arguments are used, propagating to each caller of the 10806function. The used implicit arguments are appended to the function arguments 10807after the source language arguments in the following order: 10808 10809.. TODO:: 10810 10811 Is recursion or external functions supported? 10812 108131. Work-Item ID (1 VGPR) 10814 10815 The X, Y and Z work-item ID are packed into a single VGRP with the following 10816 layout. Only fields actually used by the function are set. The other bits 10817 are undefined. 10818 10819 The values come from the initial kernel execution state. See 10820 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10821 10822 .. table:: Work-item implicit argument layout 10823 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10824 10825 ======= ======= ============== 10826 Bits Size Field Name 10827 ======= ======= ============== 10828 9:0 10 bits X Work-Item ID 10829 19:10 10 bits Y Work-Item ID 10830 29:20 10 bits Z Work-Item ID 10831 31:30 2 bits Unused 10832 ======= ======= ============== 10833 108342. Dispatch Ptr (2 SGPRs) 10835 10836 The value comes from the initial kernel execution state. See 10837 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10838 108393. Queue Ptr (2 SGPRs) 10840 10841 The value comes from the initial kernel execution state. See 10842 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10843 108444. Kernarg Segment Ptr (2 SGPRs) 10845 10846 The value comes from the initial kernel execution state. See 10847 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10848 108495. Dispatch id (2 SGPRs) 10850 10851 The value comes from the initial kernel execution state. See 10852 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10853 108546. Work-Group ID X (1 SGPR) 10855 10856 The value comes from the initial kernel execution state. See 10857 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10858 108597. Work-Group ID Y (1 SGPR) 10860 10861 The value comes from the initial kernel execution state. See 10862 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10863 108648. Work-Group ID Z (1 SGPR) 10865 10866 The value comes from the initial kernel execution state. See 10867 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10868 108699. Implicit Argument Ptr (2 SGPRs) 10870 10871 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10872 global address space pointer to the first kernarg implicit argument. 10873 10874The input and result arguments are assigned in order in the following manner: 10875 10876.. note:: 10877 10878 There are likely some errors and omissions in the following description that 10879 need correction. 10880 10881 .. TODO:: 10882 10883 Check the Clang source code to decipher how function arguments and return 10884 results are handled. Also see the AMDGPU specific values used. 10885 10886* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 10887 VGPR31. 10888 10889 If there are more arguments than will fit in these registers, the remaining 10890 arguments are allocated on the stack in order on naturally aligned 10891 addresses. 10892 10893 .. TODO:: 10894 10895 How are overly aligned structures allocated on the stack? 10896 10897* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 10898 SGPR29. 10899 10900 If there are more arguments than will fit in these registers, the remaining 10901 arguments are allocated on the stack in order on naturally aligned 10902 addresses. 10903 10904Note that decomposed struct type arguments may have some fields passed in 10905registers and some in memory. 10906 10907.. TODO:: 10908 10909 So, a struct which can pass some fields as decomposed register arguments, will 10910 pass the rest as decomposed stack elements? But an argument that will not start 10911 in registers will not be decomposed and will be passed as a non-decomposed 10912 stack value? 10913 10914The following is not part of the AMDGPU function calling convention but 10915describes how the AMDGPU implements function calls: 10916 109171. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 10918 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 10919 are used, or for the reasons defined in ``SIFrameLowering``. 109202. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 10921 to access the incoming stack arguments in the function. The BP is needed 10922 only when the function requires the runtime stack alignment. 10923 109243. Allocating SGPR arguments on the stack are not supported. 10925 109264. No CFI is currently generated. See 10927 :ref:`amdgpu-dwarf-call-frame-information`. 10928 10929 .. note:: 10930 10931 CFI will be generated that defines the CFA as the unswizzled address 10932 relative to the wave scratch base in the unswizzled private address space 10933 of the lowest address stack allocated local variable. 10934 10935 ``DW_AT_frame_base`` will be defined as the swizzled address in the 10936 swizzled private address space by dividing the CFA by the wavefront size 10937 (since CFA is always at least dword aligned which matches the scratch 10938 swizzle element size). 10939 10940 If no dynamic stack alignment was performed, the stack allocated arguments 10941 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 10942 local variables and register spill slots are accessed as positive offsets 10943 relative to ``DW_AT_frame_base``. 10944 109455. Function argument passing is implemented by copying the input physical 10946 registers to virtual registers on entry. The register allocator can spill if 10947 necessary. These are copied back to physical registers at call sites. The 10948 net effect is that each function call can have these values in entirely 10949 distinct locations. The IPRA can help avoid shuffling argument registers. 109506. Call sites are implemented by setting up the arguments at positive offsets 10951 from SP. Then SP is incremented to account for the known frame size before 10952 the call and decremented after the call. 10953 10954 .. note:: 10955 10956 The CFI will reflect the changed calculation needed to compute the CFA 10957 from SP. 10958 109597. 4 byte spill slots are used in the stack frame. One slot is allocated for an 10960 emergency spill slot. Buffer instructions are used for stack accesses and 10961 not the ``flat_scratch`` instruction. 10962 10963 .. TODO:: 10964 10965 Explain when the emergency spill slot is used. 10966 10967.. TODO:: 10968 10969 Possible broken issues: 10970 10971 - Stack arguments must be aligned to required alignment. 10972 - Stack is aligned to max(16, max formal argument alignment) 10973 - Direct argument < 64 bits should check register budget. 10974 - Register budget calculation should respect ``inreg`` for SGPR. 10975 - SGPR overflow is not handled. 10976 - struct with 1 member unpeeling is not checking size of member. 10977 - ``sret`` is after ``this`` pointer. 10978 - Caller is not implementing stack realignment: need an extra pointer. 10979 - Should say AMDGPU passes FP rather than SP. 10980 - Should CFI define CFA as address of locals or arguments. Difference is 10981 apparent when have implemented dynamic alignment. 10982 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 10983 highest address of stack frame and use negative offset for locals. Would 10984 allow SP to be the same as FP and could support signal-handler-like as now 10985 have a real SP for the top of the stack. 10986 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 10987 arguments? 10988 10989AMDPAL 10990------ 10991 10992This section provides code conventions used when the target triple OS is 10993``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 10994from the application/runtime to each invocation of a hardware shader. These 10995parameters include both generic, application-controlled parameters called 10996*user data* as well as system-generated parameters that are a product of the 10997draw or dispatch execution. 10998 10999User Data 11000~~~~~~~~~ 11001 11002Each hardware stage has a set of 32-bit *user data registers* which can be 11003written from a command buffer and then loaded into SGPRs when waves are launched 11004via a subsequent dispatch or draw operation. This is the way most arguments are 11005passed from the application/runtime to a hardware shader. 11006 11007Compute User Data 11008~~~~~~~~~~~~~~~~~ 11009 11010Compute shader user data mappings are simpler than graphics shaders and have a 11011fixed mapping. 11012 11013Note that there are always 10 available *user data entries* in registers - 11014entries beyond that limit must be fetched from memory (via the spill table 11015pointer) by the shader. 11016 11017 .. table:: PAL Compute Shader User Data Registers 11018 :name: pal-compute-user-data-registers 11019 11020 ============= ================================ 11021 User Register Description 11022 ============= ================================ 11023 0 Global Internal Table (32-bit pointer) 11024 1 Per-Shader Internal Table (32-bit pointer) 11025 2 - 11 Application-Controlled User Data (10 32-bit values) 11026 12 Spill Table (32-bit pointer) 11027 13 - 14 Thread Group Count (64-bit pointer) 11028 15 GDS Range 11029 ============= ================================ 11030 11031Graphics User Data 11032~~~~~~~~~~~~~~~~~~ 11033 11034Graphics pipelines support a much more flexible user data mapping: 11035 11036 .. table:: PAL Graphics Shader User Data Registers 11037 :name: pal-graphics-user-data-registers 11038 11039 ============= ================================ 11040 User Register Description 11041 ============= ================================ 11042 0 Global Internal Table (32-bit pointer) 11043 + Per-Shader Internal Table (32-bit pointer) 11044 + 1-15 Application Controlled User Data 11045 (1-15 Contiguous 32-bit Values in Registers) 11046 + Spill Table (32-bit pointer) 11047 + Draw Index (First Stage Only) 11048 + Vertex Offset (First Stage Only) 11049 + Instance Offset (First Stage Only) 11050 ============= ================================ 11051 11052 The placement of the global internal table remains fixed in the first *user 11053 data SGPR register*. Otherwise all parameters are optional, and can be mapped 11054 to any desired *user data SGPR register*, with the following restrictions: 11055 11056 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 11057 active hardware stage in a graphics pipeline (i.e. where the API vertex 11058 shader runs). 11059 11060 * Application-controlled user data must be mapped into a contiguous range of 11061 user data registers. 11062 11063 * The application-controlled user data range supports compaction remapping, so 11064 only *entries* that are actually consumed by the shader must be assigned to 11065 corresponding *registers*. Note that in order to support an efficient runtime 11066 implementation, the remapping must pack *registers* in the same order as 11067 *entries*, with unused *entries* removed. 11068 11069.. _pal_global_internal_table: 11070 11071Global Internal Table 11072~~~~~~~~~~~~~~~~~~~~~ 11073 11074The global internal table is a table of *shader resource descriptors* (SRDs) 11075that define how certain engine-wide, runtime-managed resources should be 11076accessed from a shader. The majority of these resources have HW-defined formats, 11077and it is up to the compiler to write/read data as required by the target 11078hardware. 11079 11080The following table illustrates the required format: 11081 11082 .. table:: PAL Global Internal Table 11083 :name: pal-git-table 11084 11085 ============= ================================ 11086 Offset Description 11087 ============= ================================ 11088 0-3 Graphics Scratch SRD 11089 4-7 Compute Scratch SRD 11090 8-11 ES/GS Ring Output SRD 11091 12-15 ES/GS Ring Input SRD 11092 16-19 GS/VS Ring Output #0 11093 20-23 GS/VS Ring Output #1 11094 24-27 GS/VS Ring Output #2 11095 28-31 GS/VS Ring Output #3 11096 32-35 GS/VS Ring Input SRD 11097 36-39 Tessellation Factor Buffer SRD 11098 40-43 Off-Chip LDS Buffer SRD 11099 44-47 Off-Chip Param Cache Buffer SRD 11100 48-51 Sample Position Buffer SRD 11101 52 vaRange::ShadowDescriptorTable High Bits 11102 ============= ================================ 11103 11104 The pointer to the global internal table passed to the shader as user data 11105 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 11106 the top 32 bits of the pipeline, so the shader may use the program 11107 counter's top 32 bits. 11108 11109.. _pal_call-convention: 11110 11111Call Convention 11112~~~~~~~~~~~~~~~ 11113 11114For graphics use cases, the calling convention is `amdgpu_gfx`. 11115 11116.. note:: 11117 11118 `amdgpu_gfx` Function calls are currently in development and are 11119 subject to major changes. 11120 11121This calling convention shares most properties with calling non-kernel 11122functions (see 11123:ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions`). 11124Differences are: 11125 11126 - Currently there are none, differences will be listed here 11127 11128Unspecified OS 11129-------------- 11130 11131This section provides code conventions used when the target triple OS is 11132empty (see :ref:`amdgpu-target-triples`). 11133 11134Trap Handler ABI 11135~~~~~~~~~~~~~~~~ 11136 11137For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11138not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11139instructions are handled as follows: 11140 11141 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11142 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11143 11144 =============== =============== =========================================== 11145 Usage Code Sequence Description 11146 =============== =============== =========================================== 11147 llvm.trap s_endpgm Causes wavefront to be terminated. 11148 llvm.debugtrap *none* Compiler warning given that there is no 11149 trap handler installed. 11150 =============== =============== =========================================== 11151 11152Source Languages 11153================ 11154 11155.. _amdgpu-opencl: 11156 11157OpenCL 11158------ 11159 11160When the language is OpenCL the following differences occur: 11161 111621. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 111632. The AMDGPU backend appends additional arguments to the kernel's explicit 11164 arguments for the AMDHSA OS (see 11165 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 111663. Additional metadata is generated 11167 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11168 11169 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11170 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11171 11172 ======== ==== ========= =========================================== 11173 Position Byte Byte Description 11174 Size Alignment 11175 ======== ==== ========= =========================================== 11176 1 8 8 OpenCL Global Offset X 11177 2 8 8 OpenCL Global Offset Y 11178 3 8 8 OpenCL Global Offset Z 11179 4 8 8 OpenCL address of printf buffer 11180 5 8 8 OpenCL address of virtual queue used by 11181 enqueue_kernel. 11182 6 8 8 OpenCL address of AqlWrap struct used by 11183 enqueue_kernel. 11184 7 8 8 Pointer argument used for Multi-gird 11185 synchronization. 11186 ======== ==== ========= =========================================== 11187 11188.. _amdgpu-hcc: 11189 11190HCC 11191--- 11192 11193When the language is HCC the following differences occur: 11194 111951. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11196 11197.. _amdgpu-assembler: 11198 11199Assembler 11200--------- 11201 11202AMDGPU backend has LLVM-MC based assembler which is currently in development. 11203It supports AMDGCN GFX6-GFX10. 11204 11205This section describes general syntax for instructions and operands. 11206 11207Instructions 11208~~~~~~~~~~~~ 11209 11210An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11211 11212 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11213 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11214 11215:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11216:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11217 11218The order of operands and modifiers is fixed. 11219Most modifiers are optional and may be omitted. 11220 11221Links to detailed instruction syntax description may be found in the following 11222table. Note that features under development are not included 11223in this description. 11224 11225 =================================== ======================================= 11226 Core ISA ISA Extensions 11227 =================================== ======================================= 11228 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11229 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11230 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11231 11232 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11233 11234 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11235 11236 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11237 11238 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11239 11240 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11241 11242 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11243 11244 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11245 =================================== ======================================= 11246 11247For more information about instructions, their semantics and supported 11248combinations of operands, refer to one of instruction set architecture manuals 11249[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_, 11250[AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11251 11252Operands 11253~~~~~~~~ 11254 11255Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11256 11257Modifiers 11258~~~~~~~~~ 11259 11260Detailed description of modifiers may be found 11261:doc:`here<AMDGPUModifierSyntax>`. 11262 11263Instruction Examples 11264~~~~~~~~~~~~~~~~~~~~ 11265 11266DS 11267++ 11268 11269.. code-block:: nasm 11270 11271 ds_add_u32 v2, v4 offset:16 11272 ds_write_src2_b64 v2 offset0:4 offset1:8 11273 ds_cmpst_f32 v2, v4, v6 11274 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11275 11276For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11277Manual. 11278 11279FLAT 11280++++ 11281 11282.. code-block:: nasm 11283 11284 flat_load_dword v1, v[3:4] 11285 flat_store_dwordx3 v[3:4], v[5:7] 11286 flat_atomic_swap v1, v[3:4], v5 glc 11287 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11288 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11289 11290For full list of supported instructions, refer to "FLAT instructions" in ISA 11291Manual. 11292 11293MUBUF 11294+++++ 11295 11296.. code-block:: nasm 11297 11298 buffer_load_dword v1, off, s[4:7], s1 11299 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11300 buffer_store_format_xy v[1:2], off, s[4:7], s1 11301 buffer_wbinvl1 11302 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11303 11304For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11305Manual. 11306 11307SMRD/SMEM 11308+++++++++ 11309 11310.. code-block:: nasm 11311 11312 s_load_dword s1, s[2:3], 0xfc 11313 s_load_dwordx8 s[8:15], s[2:3], s4 11314 s_load_dwordx16 s[88:103], s[2:3], s4 11315 s_dcache_inv_vol 11316 s_memtime s[4:5] 11317 11318For full list of supported instructions, refer to "Scalar Memory Operations" in 11319ISA Manual. 11320 11321SOP1 11322++++ 11323 11324.. code-block:: nasm 11325 11326 s_mov_b32 s1, s2 11327 s_mov_b64 s[0:1], 0x80000000 11328 s_cmov_b32 s1, 200 11329 s_wqm_b64 s[2:3], s[4:5] 11330 s_bcnt0_i32_b64 s1, s[2:3] 11331 s_swappc_b64 s[2:3], s[4:5] 11332 s_cbranch_join s[4:5] 11333 11334For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11335Manual. 11336 11337SOP2 11338++++ 11339 11340.. code-block:: nasm 11341 11342 s_add_u32 s1, s2, s3 11343 s_and_b64 s[2:3], s[4:5], s[6:7] 11344 s_cselect_b32 s1, s2, s3 11345 s_andn2_b32 s2, s4, s6 11346 s_lshr_b64 s[2:3], s[4:5], s6 11347 s_ashr_i32 s2, s4, s6 11348 s_bfm_b64 s[2:3], s4, s6 11349 s_bfe_i64 s[2:3], s[4:5], s6 11350 s_cbranch_g_fork s[4:5], s[6:7] 11351 11352For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11353Manual. 11354 11355SOPC 11356++++ 11357 11358.. code-block:: nasm 11359 11360 s_cmp_eq_i32 s1, s2 11361 s_bitcmp1_b32 s1, s2 11362 s_bitcmp0_b64 s[2:3], s4 11363 s_setvskip s3, s5 11364 11365For full list of supported instructions, refer to "SOPC Instructions" in ISA 11366Manual. 11367 11368SOPP 11369++++ 11370 11371.. code-block:: nasm 11372 11373 s_barrier 11374 s_nop 2 11375 s_endpgm 11376 s_waitcnt 0 ; Wait for all counters to be 0 11377 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11378 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11379 s_sethalt 9 11380 s_sleep 10 11381 s_sendmsg 0x1 11382 s_sendmsg sendmsg(MSG_INTERRUPT) 11383 s_trap 1 11384 11385For full list of supported instructions, refer to "SOPP Instructions" in ISA 11386Manual. 11387 11388Unless otherwise mentioned, little verification is performed on the operands 11389of SOPP Instructions, so it is up to the programmer to be familiar with the 11390range or acceptable values. 11391 11392VALU 11393++++ 11394 11395For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11396the assembler will automatically use optimal encoding based on its operands. To 11397force specific encoding, one can add a suffix to the opcode of the instruction: 11398 11399* _e32 for 32-bit VOP1/VOP2/VOPC 11400* _e64 for 64-bit VOP3 11401* _dpp for VOP_DPP 11402* _sdwa for VOP_SDWA 11403 11404VOP1/VOP2/VOP3/VOPC examples: 11405 11406.. code-block:: nasm 11407 11408 v_mov_b32 v1, v2 11409 v_mov_b32_e32 v1, v2 11410 v_nop 11411 v_cvt_f64_i32_e32 v[1:2], v2 11412 v_floor_f32_e32 v1, v2 11413 v_bfrev_b32_e32 v1, v2 11414 v_add_f32_e32 v1, v2, v3 11415 v_mul_i32_i24_e64 v1, v2, 3 11416 v_mul_i32_i24_e32 v1, -3, v3 11417 v_mul_i32_i24_e32 v1, -100, v3 11418 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11419 v_max_f16_e32 v1, v2, v3 11420 11421VOP_DPP examples: 11422 11423.. code-block:: nasm 11424 11425 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11426 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11427 v_mov_b32 v0, v0 wave_shl:1 11428 v_mov_b32 v0, v0 row_mirror 11429 v_mov_b32 v0, v0 row_bcast:31 11430 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11431 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11432 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11433 11434VOP_SDWA examples: 11435 11436.. code-block:: nasm 11437 11438 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11439 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11440 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11441 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11442 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11443 11444For full list of supported instructions, refer to "Vector ALU instructions". 11445 11446.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11447 11448Code Object V2 Predefined Symbols 11449~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11450 11451.. warning:: 11452 Code object V2 is not the default code object version emitted by 11453 this version of LLVM. 11454 11455The AMDGPU assembler defines and updates some symbols automatically. These 11456symbols do not affect code generation. 11457 11458.option.machine_version_major 11459+++++++++++++++++++++++++++++ 11460 11461Set to the GFX major generation number of the target being assembled for. For 11462example, when assembling for a "GFX9" target this will be set to the integer 11463value "9". The possible GFX major generation numbers are presented in 11464:ref:`amdgpu-processors`. 11465 11466.option.machine_version_minor 11467+++++++++++++++++++++++++++++ 11468 11469Set to the GFX minor generation number of the target being assembled for. For 11470example, when assembling for a "GFX810" target this will be set to the integer 11471value "1". The possible GFX minor generation numbers are presented in 11472:ref:`amdgpu-processors`. 11473 11474.option.machine_version_stepping 11475++++++++++++++++++++++++++++++++ 11476 11477Set to the GFX stepping generation number of the target being assembled for. 11478For example, when assembling for a "GFX704" target this will be set to the 11479integer value "4". The possible GFX stepping generation numbers are presented 11480in :ref:`amdgpu-processors`. 11481 11482.kernel.vgpr_count 11483++++++++++++++++++ 11484 11485Set to zero each time a 11486:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11487encountered. At each instruction, if the current value of this symbol is less 11488than or equal to the maximum VGPR number explicitly referenced within that 11489instruction then the symbol value is updated to equal that VGPR number plus 11490one. 11491 11492.kernel.sgpr_count 11493++++++++++++++++++ 11494 11495Set to zero each time a 11496:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11497encountered. At each instruction, if the current value of this symbol is less 11498than or equal to the maximum VGPR number explicitly referenced within that 11499instruction then the symbol value is updated to equal that SGPR number plus 11500one. 11501 11502.. _amdgpu-amdhsa-assembler-directives-v2: 11503 11504Code Object V2 Directives 11505~~~~~~~~~~~~~~~~~~~~~~~~~ 11506 11507.. warning:: 11508 Code object V2 is not the default code object version emitted by 11509 this version of LLVM. 11510 11511AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11512one can specify them with assembler directives. 11513 11514.hsa_code_object_version major, minor 11515+++++++++++++++++++++++++++++++++++++ 11516 11517*major* and *minor* are integers that specify the version of the HSA code 11518object that will be generated by the assembler. 11519 11520.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11521+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11522 11523 11524*major*, *minor*, and *stepping* are all integers that describe the instruction 11525set architecture (ISA) version of the assembly program. 11526 11527*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11528"AMD" and *arch* should always be equal to "AMDGPU". 11529 11530By default, the assembler will derive the ISA version, *vendor*, and *arch* 11531from the value of the -mcpu option that is passed to the assembler. 11532 11533.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11534 11535.amdgpu_hsa_kernel (name) 11536+++++++++++++++++++++++++ 11537 11538This directives specifies that the symbol with given name is a kernel entry 11539point (label) and the object should contain corresponding symbol of type 11540STT_AMDGPU_HSA_KERNEL. 11541 11542.amd_kernel_code_t 11543++++++++++++++++++ 11544 11545This directive marks the beginning of a list of key / value pairs that are used 11546to specify the amd_kernel_code_t object that will be emitted by the assembler. 11547The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11548amd_kernel_code_t values that are unspecified a default value will be used. The 11549default value for all keys is 0, with the following exceptions: 11550 11551- *amd_code_version_major* defaults to 1. 11552- *amd_kernel_code_version_minor* defaults to 2. 11553- *amd_machine_kind* defaults to 1. 11554- *amd_machine_version_major*, *machine_version_minor*, and 11555 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11556 that is passed to the assembler. 11557- *kernel_code_entry_byte_offset* defaults to 256. 11558- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11559 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11560 Note that wavefront size is specified as a power of two, so a value of **n** 11561 means a size of 2^ **n**. 11562- *call_convention* defaults to -1. 11563- *kernarg_segment_alignment*, *group_segment_alignment*, and 11564 *private_segment_alignment* default to 4. Note that alignments are specified 11565 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11566- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11567 GFX90A onwards. 11568- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11569 GFX10 onwards. 11570- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11571 11572The *.amd_kernel_code_t* directive must be placed immediately after the 11573function label and before any instructions. 11574 11575For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11576comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11577 11578.. _amdgpu-amdhsa-assembler-example-v2: 11579 11580Code Object V2 Example Source Code 11581~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11582 11583.. warning:: 11584 Code Object V2 is not the default code object version emitted by 11585 this version of LLVM. 11586 11587Here is an example of a minimal assembly source file, defining one HSA kernel: 11588 11589.. code:: 11590 :number-lines: 11591 11592 .hsa_code_object_version 1,0 11593 .hsa_code_object_isa 11594 11595 .hsatext 11596 .globl hello_world 11597 .p2align 8 11598 .amdgpu_hsa_kernel hello_world 11599 11600 hello_world: 11601 11602 .amd_kernel_code_t 11603 enable_sgpr_kernarg_segment_ptr = 1 11604 is_ptr64 = 1 11605 compute_pgm_rsrc1_vgprs = 0 11606 compute_pgm_rsrc1_sgprs = 0 11607 compute_pgm_rsrc2_user_sgpr = 2 11608 compute_pgm_rsrc1_wgp_mode = 0 11609 compute_pgm_rsrc1_mem_ordered = 0 11610 compute_pgm_rsrc1_fwd_progress = 1 11611 .end_amd_kernel_code_t 11612 11613 s_load_dwordx2 s[0:1], s[0:1] 0x0 11614 v_mov_b32 v0, 3.14159 11615 s_waitcnt lgkmcnt(0) 11616 v_mov_b32 v1, s0 11617 v_mov_b32 v2, s1 11618 flat_store_dword v[1:2], v0 11619 s_endpgm 11620 .Lfunc_end0: 11621 .size hello_world, .Lfunc_end0-hello_world 11622 11623.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11624 11625Code Object V3 to V4 Predefined Symbols 11626~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11627 11628The AMDGPU assembler defines and updates some symbols automatically. These 11629symbols do not affect code generation. 11630 11631.amdgcn.gfx_generation_number 11632+++++++++++++++++++++++++++++ 11633 11634Set to the GFX major generation number of the target being assembled for. For 11635example, when assembling for a "GFX9" target this will be set to the integer 11636value "9". The possible GFX major generation numbers are presented in 11637:ref:`amdgpu-processors`. 11638 11639.amdgcn.gfx_generation_minor 11640++++++++++++++++++++++++++++ 11641 11642Set to the GFX minor generation number of the target being assembled for. For 11643example, when assembling for a "GFX810" target this will be set to the integer 11644value "1". The possible GFX minor generation numbers are presented in 11645:ref:`amdgpu-processors`. 11646 11647.amdgcn.gfx_generation_stepping 11648+++++++++++++++++++++++++++++++ 11649 11650Set to the GFX stepping generation number of the target being assembled for. 11651For example, when assembling for a "GFX704" target this will be set to the 11652integer value "4". The possible GFX stepping generation numbers are presented 11653in :ref:`amdgpu-processors`. 11654 11655.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 11656 11657.amdgcn.next_free_vgpr 11658++++++++++++++++++++++ 11659 11660Set to zero before assembly begins. At each instruction, if the current value 11661of this symbol is less than or equal to the maximum VGPR number explicitly 11662referenced within that instruction then the symbol value is updated to equal 11663that VGPR number plus one. 11664 11665May be used to set the `.amdhsa_next_free_vgpr` directive in 11666:ref:`amdhsa-kernel-directives-table`. 11667 11668May be set at any time, e.g. manually set to zero at the start of each kernel. 11669 11670.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 11671 11672.amdgcn.next_free_sgpr 11673++++++++++++++++++++++ 11674 11675Set to zero before assembly begins. At each instruction, if the current value 11676of this symbol is less than or equal the maximum SGPR number explicitly 11677referenced within that instruction then the symbol value is updated to equal 11678that SGPR number plus one. 11679 11680May be used to set the `.amdhsa_next_free_spgr` directive in 11681:ref:`amdhsa-kernel-directives-table`. 11682 11683May be set at any time, e.g. manually set to zero at the start of each kernel. 11684 11685.. _amdgpu-amdhsa-assembler-directives-v3-v4: 11686 11687Code Object V3 to V4 Directives 11688~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11689 11690Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 11691architecture processors, and are not OS-specific. Directives which begin with 11692``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 11693``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 11694:ref:`amdgpu-processors`. 11695 11696.. _amdgpu-assembler-directive-amdgcn-target: 11697 11698.amdgcn_target <target-triple> "-" <target-id> 11699++++++++++++++++++++++++++++++++++++++++++++++ 11700 11701Optional directive which declares the ``<target-triple>-<target-id>`` supported 11702by the containing assembler source file. Used by the assembler to validate 11703command-line options such as ``-triple``, ``-mcpu``, and 11704``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 11705:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 11706 11707.. note:: 11708 11709 The target ID syntax used for code object V2 to V3 for this directive differs 11710 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 11711 11712.amdhsa_kernel <name> 11713+++++++++++++++++++++ 11714 11715Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 11716``<name>.kd``, in the current location of the current section. Only valid when 11717the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 11718instruction to execute, and does not need to be previously defined. 11719 11720Marks the beginning of a list of directives used to generate the bytes of a 11721kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 11722Directives which may appear in this list are described in 11723:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 11724be valid for the target being assembled for, and cannot be repeated. Directives 11725support the range of values specified by the field they reference in 11726:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 11727assumed to have its default value, unless it is marked as "Required", in which 11728case it is an error to omit the directive. This list of directives is 11729terminated by an ``.end_amdhsa_kernel`` directive. 11730 11731 .. table:: AMDHSA Kernel Assembler Directives 11732 :name: amdhsa-kernel-directives-table 11733 11734 ======================================================== =================== ============ =================== 11735 Directive Default Supported On Description 11736 ======================================================== =================== ============ =================== 11737 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 11738 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11739 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 11740 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11741 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 11742 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11743 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 11744 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11745 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 11746 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11747 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 11748 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11749 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 11750 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11751 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 11752 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11753 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 11754 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11755 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 11756 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11757 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 11758 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11759 Specific 11760 (wavefrontsize64) 11761 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 11762 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11763 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 11764 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11765 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 11766 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11767 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 11768 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11769 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 11770 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11771 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 11772 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11773 Possible values are defined in 11774 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 11775 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 11776 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 11777 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11778 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 11779 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11780 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11781 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 11782 Used to calculate ACCUM_OFFSET in 11783 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11784 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 11785 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11786 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11787 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 11788 scratch memory. Used to calculate 11789 GRANULATED_WAVEFRONT_SGPR_COUNT in 11790 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11791 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 11792 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11793 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11794 (xnack) 11795 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 11796 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11797 Possible values are defined in 11798 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11799 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 11800 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11801 Possible values are defined in 11802 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11803 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 11804 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11805 Possible values are defined in 11806 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11807 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 11808 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11809 Possible values are defined in 11810 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11811 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 11812 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11813 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 11814 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11815 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 11816 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11817 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 11818 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11819 Specific 11820 (tgsplit) 11821 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 11822 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11823 Specific 11824 (cumode) 11825 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 11826 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11827 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 11828 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11829 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 11830 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11831 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 11832 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11833 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 11834 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11835 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 11836 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11837 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 11838 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11839 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 11840 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11841 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 11842 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11843 ======================================================== =================== ============ =================== 11844 11845.amdgpu_metadata 11846++++++++++++++++ 11847 11848Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 11849note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 11850 11851The contents must be in the [YAML]_ markup format, with the same structure and 11852semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 11853:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 11854 11855This directive is terminated by an ``.end_amdgpu_metadata`` directive. 11856 11857.. _amdgpu-amdhsa-assembler-example-v3-v4: 11858 11859Code Object V3 to V4 Example Source Code 11860~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11861 11862Here is an example of a minimal assembly source file, defining one HSA kernel: 11863 11864.. code:: 11865 :number-lines: 11866 11867 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 11868 11869 .text 11870 .globl hello_world 11871 .p2align 8 11872 .type hello_world,@function 11873 hello_world: 11874 s_load_dwordx2 s[0:1], s[0:1] 0x0 11875 v_mov_b32 v0, 3.14159 11876 s_waitcnt lgkmcnt(0) 11877 v_mov_b32 v1, s0 11878 v_mov_b32 v2, s1 11879 flat_store_dword v[1:2], v0 11880 s_endpgm 11881 .Lfunc_end0: 11882 .size hello_world, .Lfunc_end0-hello_world 11883 11884 .rodata 11885 .p2align 6 11886 .amdhsa_kernel hello_world 11887 .amdhsa_user_sgpr_kernarg_segment_ptr 1 11888 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11889 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11890 .end_amdhsa_kernel 11891 11892 .amdgpu_metadata 11893 --- 11894 amdhsa.version: 11895 - 1 11896 - 0 11897 amdhsa.kernels: 11898 - .name: hello_world 11899 .symbol: hello_world.kd 11900 .kernarg_segment_size: 48 11901 .group_segment_fixed_size: 0 11902 .private_segment_fixed_size: 0 11903 .kernarg_segment_align: 4 11904 .wavefront_size: 64 11905 .sgpr_count: 2 11906 .vgpr_count: 3 11907 .max_flat_workgroup_size: 256 11908 ... 11909 .end_amdgpu_metadata 11910 11911If an assembly source file contains multiple kernels and/or functions, the 11912:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 11913:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 11914the ``.set <symbol>, <expression>`` directive. For example, in the case of two 11915kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 11916to group the function with the kernel that calls it and reset the symbols 11917between the two connected components: 11918 11919.. code:: 11920 :number-lines: 11921 11922 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 11923 11924 // gpr tracking symbols are implicitly set to zero 11925 11926 .text 11927 .globl kern0 11928 .p2align 8 11929 .type kern0,@function 11930 kern0: 11931 // ... 11932 s_endpgm 11933 .Lkern0_end: 11934 .size kern0, .Lkern0_end-kern0 11935 11936 .rodata 11937 .p2align 6 11938 .amdhsa_kernel kern0 11939 // ... 11940 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11941 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11942 .end_amdhsa_kernel 11943 11944 // reset symbols to begin tracking usage in func1 and kern1 11945 .set .amdgcn.next_free_vgpr, 0 11946 .set .amdgcn.next_free_sgpr, 0 11947 11948 .text 11949 .hidden func1 11950 .global func1 11951 .p2align 2 11952 .type func1,@function 11953 func1: 11954 // ... 11955 s_setpc_b64 s[30:31] 11956 .Lfunc1_end: 11957 .size func1, .Lfunc1_end-func1 11958 11959 .globl kern1 11960 .p2align 8 11961 .type kern1,@function 11962 kern1: 11963 // ... 11964 s_getpc_b64 s[4:5] 11965 s_add_u32 s4, s4, func1@rel32@lo+4 11966 s_addc_u32 s5, s5, func1@rel32@lo+4 11967 s_swappc_b64 s[30:31], s[4:5] 11968 // ... 11969 s_endpgm 11970 .Lkern1_end: 11971 .size kern1, .Lkern1_end-kern1 11972 11973 .rodata 11974 .p2align 6 11975 .amdhsa_kernel kern1 11976 // ... 11977 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11978 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11979 .end_amdhsa_kernel 11980 11981These symbols cannot identify connected components in order to automatically 11982track the usage for each kernel. However, in some cases careful organization of 11983the kernels and functions in the source file means there is minimal additional 11984effort required to accurately calculate GPR usage. 11985 11986Additional Documentation 11987======================== 11988 11989.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 11990.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 11991.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 11992.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 11993.. [AMD-GCN-GFX10-RDNA1] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 11994.. [AMD-GCN-GFX10-RDNA2] `AMD "RDNA 2" Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 11995.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 11996.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 11997.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 11998.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 11999.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 12000.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12001.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12002.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12003.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12004.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12005.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12006.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12007.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12008.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12009.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12010.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12011