1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 42to specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as: 73 74 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 75 loader on Linux. See *AMD ROCm Platform Release Notes* 76 [AMD-ROCm-Release-Notes]_ for supported hardware and 77 software. 78 - AMD's PAL runtime using the *pal-amdhsa* loader on 79 Windows. 80 81 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 82 runtime using the *pal-amdpal* loader on Windows and Linux 83 Pro. 84 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 85 3D runtime using the *mesa-mesa3d* loader on Linux. 86 ============== ============================================================ 87 88 .. table:: AMDGPU Environments 89 :name: amdgpu-environment-table 90 91 ============ ============================================================== 92 Environment Description 93 ============ ============================================================== 94 *<empty>* Default. 95 ============ ============================================================== 96 97.. _amdgpu-processors: 98 99Processors 100---------- 101 102Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 103specify the AMDGPU processor together with optional target features. See 104:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 105specific information. 106 107Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 108 109* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 110 111 112 .. table:: AMDGPU Processors 113 :name: amdgpu-processor-table 114 115 =========== =============== ============ ===== ================= =============== =============== ====================== 116 Processor Alternative Target dGPU/ Target Target OS Support Example 117 Processor Triple APU Features Properties *(see* Products 118 Architecture Supported `amdgpu-os`_ 119 *and 120 corresponding 121 runtime release 122 notes for 123 current 124 information and 125 level of 126 support)* 127 =========== =============== ============ ===== ================= =============== =============== ====================== 128 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 129 ----------------------------------------------------------------------------------------------------------------------- 130 ``r600`` ``r600`` dGPU - Does not 131 support 132 generic 133 address 134 space 135 ``r630`` ``r600`` dGPU - Does not 136 support 137 generic 138 address 139 space 140 ``rs880`` ``r600`` dGPU - Does not 141 support 142 generic 143 address 144 space 145 ``rv670`` ``r600`` dGPU - Does not 146 support 147 generic 148 address 149 space 150 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 151 ----------------------------------------------------------------------------------------------------------------------- 152 ``rv710`` ``r600`` dGPU - Does not 153 support 154 generic 155 address 156 space 157 ``rv730`` ``r600`` dGPU - Does not 158 support 159 generic 160 address 161 space 162 ``rv770`` ``r600`` dGPU - Does not 163 support 164 generic 165 address 166 space 167 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 168 ----------------------------------------------------------------------------------------------------------------------- 169 ``cedar`` ``r600`` dGPU - Does not 170 support 171 generic 172 address 173 space 174 ``cypress`` ``r600`` dGPU - Does not 175 support 176 generic 177 address 178 space 179 ``juniper`` ``r600`` dGPU - Does not 180 support 181 generic 182 address 183 space 184 ``redwood`` ``r600`` dGPU - Does not 185 support 186 generic 187 address 188 space 189 ``sumo`` ``r600`` dGPU - Does not 190 support 191 generic 192 address 193 space 194 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 195 ----------------------------------------------------------------------------------------------------------------------- 196 ``barts`` ``r600`` dGPU - Does not 197 support 198 generic 199 address 200 space 201 ``caicos`` ``r600`` dGPU - Does not 202 support 203 generic 204 address 205 space 206 ``cayman`` ``r600`` dGPU - Does not 207 support 208 generic 209 address 210 space 211 ``turks`` ``r600`` dGPU - Does not 212 support 213 generic 214 address 215 space 216 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 217 ----------------------------------------------------------------------------------------------------------------------- 218 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 219 support 220 generic 221 address 222 space 223 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 224 - ``verde`` support 225 generic 226 address 227 space 228 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 229 - ``oland`` support 230 generic 231 address 232 space 233 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 234 ----------------------------------------------------------------------------------------------------------------------- 235 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 236 flat - *pal-amdhsa* - A6 Pro-7050B 237 scratch - *pal-amdpal* - A8-7100 238 - A8 Pro-7150B 239 - A10-7300 240 - A10 Pro-7350B 241 - FX-7500 242 - A8-7200P 243 - A10-7400P 244 - FX-7600P 245 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 246 flat - *pal-amdhsa* - FirePro W9100 247 scratch - *pal-amdpal* - FirePro S9150 248 - FirePro S9170 249 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 250 flat - *pal-amdhsa* - Radeon R9 290x 251 scratch - *pal-amdpal* - Radeon R390 252 - Radeon R390x 253 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 254 - ``mullins`` flat - *pal-amdpal* - E1-2200 255 scratch - E1-2500 256 - E2-3000 257 - E2-3800 258 - A4-5000 259 - A4-5100 260 - A6-5200 261 - A4 Pro-3340B 262 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 263 flat - *pal-amdpal* - Radeon HD 8770 264 scratch - R7 260 265 - R7 260X 266 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 267 flat - *pal-amdpal* 268 scratch .. TODO:: 269 270 Add product 271 names. 272 273 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 274 ----------------------------------------------------------------------------------------------------------------------- 275 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 276 flat - *pal-amdhsa* - Pro A6-8500B 277 scratch - *pal-amdpal* - A8-8600P 278 - Pro A8-8600B 279 - FX-8800P 280 - Pro A12-8800B 281 - A10-8700P 282 - Pro A10-8700B 283 - A10-8780P 284 - A10-9600P 285 - A10-9630P 286 - A12-9700P 287 - A12-9730P 288 - FX-9800P 289 - FX-9830P 290 - E2-9010 291 - A6-9210 292 - A9-9410 293 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 294 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 295 scratch - *pal-amdpal* - Radeon R9 385 296 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 297 - *pal-amdhsa* - Radeon R9 Fury 298 - *pal-amdpal* - Radeon R9 FuryX 299 - Radeon Pro Duo 300 - FirePro S9300x2 301 - Radeon Instinct MI8 302 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 303 flat - *pal-amdhsa* - Radeon RX 480 304 scratch - *pal-amdpal* - Radeon Instinct MI6 305 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 306 flat - *pal-amdhsa* 307 scratch - *pal-amdpal* 308 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 309 flat - *pal-amdhsa* - FirePro S7100 310 scratch - *pal-amdpal* - FirePro W7100 311 - Mobile FirePro 312 M7170 313 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 314 flat - *pal-amdhsa* 315 scratch - *pal-amdpal* .. TODO:: 316 317 Add product 318 names. 319 320 **GCN GFX9 (Vega)** [AMD-GCN-GFX9]_ 321 ----------------------------------------------------------------------------------------------------------------------- 322 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 323 flat - *pal-amdhsa* Frontier Edition 324 scratch - *pal-amdpal* - Radeon RX Vega 56 325 - Radeon RX Vega 64 326 - Radeon RX Vega 64 327 Liquid 328 - Radeon Instinct MI25 329 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 330 flat - *pal-amdhsa* - Ryzen 5 2400G 331 scratch - *pal-amdpal* 332 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 333 - *pal-amdhsa* 334 - *pal-amdpal* .. TODO:: 335 336 Add product 337 names. 338 339 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 340 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 341 scratch - *pal-amdpal* - Radeon VII 342 - Radeon Pro VII 343 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* *TBA* 344 - xnack - Absolute 345 flat .. TODO:: 346 scratch 347 Add product 348 names. 349 350 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 351 flat 352 scratch .. TODO:: 353 354 Add product 355 names. 356 357 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 358 - tgsplit flat 359 - xnack scratch .. TODO:: 360 - Packed 361 work-item Add product 362 IDs names. 363 364 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 365 flat - Ryzen 7 4700GE 366 scratch - Ryzen 5 4600G 367 - Ryzen 5 4600GE 368 - Ryzen 3 4300G 369 - Ryzen 3 4300GE 370 - Ryzen Pro 4000G 371 - Ryzen 7 Pro 4700G 372 - Ryzen 7 Pro 4750GE 373 - Ryzen 5 Pro 4650G 374 - Ryzen 5 Pro 4650GE 375 - Ryzen 3 Pro 4350G 376 - Ryzen 3 Pro 4350GE 377 378 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 379 ----------------------------------------------------------------------------------------------------------------------- 380 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 381 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 382 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 383 - Radeon Pro 5600M 384 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* *TBA* 385 - wavefrontsize64 - Absolute - *pal-amdhsa* 386 - xnack flat - *pal-amdpal* 387 scratch .. TODO:: 388 389 Add product 390 names. 391 392 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 393 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 394 - xnack scratch - *pal-amdpal* 395 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 396 ----------------------------------------------------------------------------------------------------------------------- 397 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 398 - wavefrontsize64 flat - *pal-amdhsa* 399 scratch - *pal-amdpal* .. TODO:: 400 401 Add product 402 names. 403 404 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 405 - wavefrontsize64 flat - *pal-amdhsa* 406 scratch - *pal-amdpal* .. TODO:: 407 408 Add product 409 names. 410 411 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 412 - wavefrontsize64 flat - *pal-amdhsa* 413 scratch - *pal-amdpal* .. TODO:: 414 415 Add product 416 names. 417 418 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 419 - wavefrontsize64 flat 420 scratch .. TODO:: 421 422 Add product 423 names. 424 425 =========== =============== ============ ===== ================= =============== =============== ====================== 426 427.. _amdgpu-target-features: 428 429Target Features 430--------------- 431 432Target features control how code is generated to support certain 433processor specific features. Not all target features are supported by 434all processors. The runtime must ensure that the features supported by 435the device used to execute the code match the features enabled when 436generating the code. A mismatch of features may result in incorrect 437execution, or a reduction in performance. 438 439The target features supported by each processor is listed in 440:ref:`amdgpu-processor-table`. 441 442Target features are controlled by exactly one of the following Clang 443options: 444 445``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 446 447 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 448 optional components of the target ID. If omitted, the target feature has the 449 ``any`` value. See :ref:`amdgpu-target-id`. 450 451``-m[no-]<target-feature>`` 452 453 Target features not specified by the target ID are specified using a 454 separate option. These target features can have an ``on`` or ``off`` 455 value. ``on`` is specified by omitting the ``no-`` prefix, and 456 ``off`` is specified by including the ``no-`` prefix. The default 457 if not specified is ``off``. 458 459For example: 460 461``-mcpu=gfx908:xnack+`` 462 Enable the ``xnack`` feature. 463``-mcpu=gfx908:xnack-`` 464 Disable the ``xnack`` feature. 465``-mcumode`` 466 Enable the ``cumode`` feature. 467``-mno-cumode`` 468 Disable the ``cumode`` feature. 469 470 .. table:: AMDGPU Target Features 471 :name: amdgpu-target-features-table 472 473 =============== ============================ ================================================== 474 Target Feature Clang Option to Control Description 475 Name 476 =============== ============================ ================================================== 477 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 478 when generating code for kernels. When disabled 479 native WGP wavefront execution mode is used, 480 when enabled CU wavefront execution mode is used 481 (see :ref:`amdgpu-amdhsa-memory-model`). 482 483 sramecc - ``-mcpu`` If specified, generate code that can only be 484 - ``--offload-arch`` loaded and executed in a process that has a 485 matching setting for SRAMECC. 486 487 If not specified for code object V2 to V3, generate 488 code that can be loaded and executed in a process 489 with SRAMECC enabled. 490 491 If not specified for code object V4, generate 492 code that can be loaded and executed in a process 493 with either setting of SRAMECC. 494 495 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 496 work-groups are launched in threadgroup split mode. 497 When enabled the waves of a work-group may be 498 launched in different CUs. 499 500 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 501 generating code for kernels. When disabled 502 native wavefront size 32 is used, when enabled 503 wavefront size 64 is used. 504 505 xnack - ``-mcpu`` If specified, generate code that can only be 506 - ``--offload-arch`` loaded and executed in a process that has a 507 matching setting for XNACK replay. 508 509 If not specified for code object V2 to V3, generate 510 code that can be loaded and executed in a process 511 with XNACK replay enabled. 512 513 If not specified for code object V4, generate 514 code that can be loaded and executed in a process 515 with either setting of XNACK replay. 516 517 XNACK replay can be used for demand paging and 518 page migration. If enabled in the device, then if 519 a page fault occurs the code may execute 520 incorrectly unless generated with XNACK replay 521 enabled, or generated for code object V4 without 522 specifying XNACK replay. Executing code that was 523 generated with XNACK replay enabled, or generated 524 for code object V4 without specifying XNACK replay, 525 on a device that does not have XNACK replay 526 enabled will execute correctly but may be less 527 performant than code generated for XNACK replay 528 disabled. 529 =============== ============================ ================================================== 530 531.. _amdgpu-target-id: 532 533Target ID 534--------- 535 536AMDGPU supports target IDs. See `Clang Offload Bundler 537<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 538description. The AMDGPU target specific information is: 539 540**processor** 541 Is an AMDGPU processor or alternative processor name specified in 542 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 543 the primary processor and alternative processor names. The canonical form 544 target ID only allow the primary processor name. 545 546**target-feature** 547 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 548 is supported by the processor. The target features supported by each processor 549 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 550 a target ID are marked as being controlled by ``-mcpu`` and 551 ``--offload-arch``. Each target feature must appear at most once in a target 552 ID. The non-canonical form target ID allows the target features to be 553 specified in any order. The canonical form target ID requires the target 554 features to be specified in alphabetic order. 555 556.. _amdgpu-target-id-v2-v3: 557 558Code Object V2 to V3 Target ID 559~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 560 561The target ID syntax for code object V2 to V3 is the same as defined in `Clang 562Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 563when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 564directive and the bundle entry ID. In those cases it has the following BNF 565syntax: 566 567.. code:: 568 569 <target-id> ::== <processor> ( "+" <target-feature> )* 570 571Where a target feature is omitted if *Off* and present if *On* or *Any*. 572 573.. note:: 574 575 The code object V2 to V3 cannot represent *Any* and treats it the same as 576 *On*. 577 578.. _amdgpu-embedding-bundled-objects: 579 580Embedding Bundled Code Objects 581------------------------------ 582 583AMDGPU supports the HIP and OpenMP languages that perform code object embedding 584as described in `Clang Offload Bundler 585<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 586 587.. note:: 588 589 The target ID syntax used for code object V2 to V3 for a bundle entry ID 590 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 591 592.. _amdgpu-address-spaces: 593 594Address Spaces 595-------------- 596 597The AMDGPU architecture supports a number of memory address spaces. The address 598space names use the OpenCL standard names, with some additions. 599 600The AMDGPU address spaces correspond to target architecture specific LLVM 601address space numbers used in LLVM IR. 602 603The AMDGPU address spaces are described in 604:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 605supported for the ``amdgcn`` target. 606 607 .. table:: AMDGPU Address Spaces 608 :name: amdgpu-address-spaces-table 609 610 ================================= =============== =========== ================ ======= ============================ 611 .. 64-Bit Process Address Space 612 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 613 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 614 Space Number Name Name Size 615 ================================= =============== =========== ================ ======= ============================ 616 Generic 0 flat flat 64 0x0000000000000000 617 Global 1 global global 64 0x0000000000000000 618 Region 2 N/A GDS 32 *not implemented for AMDHSA* 619 Local 3 group LDS 32 0xFFFFFFFF 620 Constant 4 constant *same as global* 64 0x0000000000000000 621 Private 5 private scratch 32 0xFFFFFFFF 622 Constant 32-bit 6 *TODO* 0x00000000 623 Buffer Fat Pointer (experimental) 7 *TODO* 624 ================================= =============== =========== ================ ======= ============================ 625 626**Generic** 627 The generic address space is supported unless the *Target Properties* column 628 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 629 space*. 630 631 The generic address space uses the hardware flat address support for two fixed 632 ranges of virtual addresses (the private and local apertures), that are 633 outside the range of addressable global memory, to map from a flat address to 634 a private or local address. This uses FLAT instructions that can take a flat 635 address and access global, private (scratch), and group (LDS) memory depending 636 on if the address is within one of the aperture ranges. 637 638 Flat access to scratch requires hardware aperture setup and setup in the 639 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 640 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 641 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 642 643 To convert between a private or group address space address (termed a segment 644 address) and a flat address the base address of the corresponding aperture 645 can be used. For GFX7-GFX8 these are available in the 646 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 647 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 648 GFX9-GFX10 the aperture base addresses are directly available as inline 649 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 650 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 651 aligned to 2^32 which makes it easier to convert from flat to segment or 652 segment to flat. 653 654 A global address space address has the same value when used as a flat address 655 so no conversion is needed. 656 657**Global and Constant** 658 The global and constant address spaces both use global virtual addresses, 659 which are the same virtual address space used by the CPU. However, some 660 virtual addresses may only be accessible to the CPU, some only accessible 661 by the GPU, and some by both. 662 663 Using the constant address space indicates that the data will not change 664 during the execution of the kernel. This allows scalar read instructions to 665 be used. As the constant address space could only be modified on the host 666 side, a generic pointer loaded from the constant address space is safe to be 667 assumed as a global pointer since only the device global memory is visible 668 and managed on the host side. The vector and scalar L1 caches are invalidated 669 of volatile data before each kernel dispatch execution to allow constant 670 memory to change values between kernel dispatches. 671 672**Region** 673 The region address space uses the hardware Global Data Store (GDS). All 674 wavefronts executing on the same device will access the same memory for any 675 given region address. However, the same region address accessed by wavefronts 676 executing on different devices will access different memory. It is higher 677 performance than global memory. It is allocated by the runtime. The data 678 store (DS) instructions can be used to access it. 679 680**Local** 681 The local address space uses the hardware Local Data Store (LDS) which is 682 automatically allocated when the hardware creates the wavefronts of a 683 work-group, and freed when all the wavefronts of a work-group have 684 terminated. All wavefronts belonging to the same work-group will access the 685 same memory for any given local address. However, the same local address 686 accessed by wavefronts belonging to different work-groups will access 687 different memory. It is higher performance than global memory. The data store 688 (DS) instructions can be used to access it. 689 690**Private** 691 The private address space uses the hardware scratch memory support which 692 automatically allocates memory when it creates a wavefront and frees it when 693 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 694 given private address will be different to the memory accessed by another lane 695 of the same or different wavefront for the same private address. 696 697 If a kernel dispatch uses scratch, then the hardware allocates memory from a 698 pool of backing memory allocated by the runtime for each wavefront. The lanes 699 of the wavefront access this using dword (4 byte) interleaving. The mapping 700 used from private address to backing memory address is: 701 702 ``wavefront-scratch-base + 703 ((private-address / 4) * wavefront-size * 4) + 704 (wavefront-lane-id * 4) + (private-address % 4)`` 705 706 If each lane of a wavefront accesses the same private address, the 707 interleaving results in adjacent dwords being accessed and hence requires 708 fewer cache lines to be fetched. 709 710 There are different ways that the wavefront scratch base address is 711 determined by a wavefront (see 712 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 713 714 Scratch memory can be accessed in an interleaved manner using buffer 715 instructions with the scratch buffer descriptor and per wavefront scratch 716 offset, by the scratch instructions, or by flat instructions. Multi-dword 717 access is not supported except by flat and scratch instructions in 718 GFX9-GFX10. 719 720**Constant 32-bit** 721 *TODO* 722 723**Buffer Fat Pointer** 724 The buffer fat pointer is an experimental address space that is currently 725 unsupported in the backend. It exposes a non-integral pointer that is in 726 the future intended to support the modelling of 128-bit buffer descriptors 727 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 728 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 729 model the buffer descriptors used heavily in graphics workloads targeting 730 the backend. 731 732.. _amdgpu-memory-scopes: 733 734Memory Scopes 735------------- 736 737This section provides LLVM memory synchronization scopes supported by the AMDGPU 738backend memory model when the target triple OS is ``amdhsa`` (see 739:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 740 741The memory model supported is based on the HSA memory model [HSA]_ which is 742based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 743relation is transitive over the synchronizes-with relation independent of scope 744and synchronizes-with allows the memory scope instances to be inclusive (see 745table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 746 747This is different to the OpenCL [OpenCL]_ memory model which does not have scope 748inclusion and requires the memory scopes to exactly match. However, this 749is conservatively correct for OpenCL. 750 751 .. table:: AMDHSA LLVM Sync Scopes 752 :name: amdgpu-amdhsa-llvm-sync-scopes-table 753 754 ======================= =================================================== 755 LLVM Sync Scope Description 756 ======================= =================================================== 757 *none* The default: ``system``. 758 759 Synchronizes with, and participates in modification 760 and seq_cst total orderings with, other operations 761 (except image operations) for all address spaces 762 (except private, or generic that accesses private) 763 provided the other operation's sync scope is: 764 765 - ``system``. 766 - ``agent`` and executed by a thread on the same 767 agent. 768 - ``workgroup`` and executed by a thread in the 769 same work-group. 770 - ``wavefront`` and executed by a thread in the 771 same wavefront. 772 773 ``agent`` Synchronizes with, and participates in modification 774 and seq_cst total orderings with, other operations 775 (except image operations) for all address spaces 776 (except private, or generic that accesses private) 777 provided the other operation's sync scope is: 778 779 - ``system`` or ``agent`` and executed by a thread 780 on the same agent. 781 - ``workgroup`` and executed by a thread in the 782 same work-group. 783 - ``wavefront`` and executed by a thread in the 784 same wavefront. 785 786 ``workgroup`` Synchronizes with, and participates in modification 787 and seq_cst total orderings with, other operations 788 (except image operations) for all address spaces 789 (except private, or generic that accesses private) 790 provided the other operation's sync scope is: 791 792 - ``system``, ``agent`` or ``workgroup`` and 793 executed by a thread in the same work-group. 794 - ``wavefront`` and executed by a thread in the 795 same wavefront. 796 797 ``wavefront`` Synchronizes with, and participates in modification 798 and seq_cst total orderings with, other operations 799 (except image operations) for all address spaces 800 (except private, or generic that accesses private) 801 provided the other operation's sync scope is: 802 803 - ``system``, ``agent``, ``workgroup`` or 804 ``wavefront`` and executed by a thread in the 805 same wavefront. 806 807 ``singlethread`` Only synchronizes with and participates in 808 modification and seq_cst total orderings with, 809 other operations (except image operations) running 810 in the same thread for all address spaces (for 811 example, in signal handlers). 812 813 ``one-as`` Same as ``system`` but only synchronizes with other 814 operations within the same address space. 815 816 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 817 operations within the same address space. 818 819 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 820 other operations within the same address space. 821 822 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 823 other operations within the same address space. 824 825 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 826 other operations within the same address space. 827 ======================= =================================================== 828 829LLVM IR Intrinsics 830------------------ 831 832The AMDGPU backend implements the following LLVM IR intrinsics. 833 834*This section is WIP.* 835 836.. TODO:: 837 838 List AMDGPU intrinsics. 839 840LLVM IR Attributes 841------------------ 842 843The AMDGPU backend supports the following LLVM IR attributes. 844 845 .. table:: AMDGPU LLVM IR Attributes 846 :name: amdgpu-llvm-ir-attributes-table 847 848 ======================================= ========================================================== 849 LLVM Attribute Description 850 ======================================= ========================================================== 851 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 852 will be specified when the kernel is dispatched. Generated 853 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 854 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 855 argument block size for the implicit arguments. This 856 varies by OS and language (for OpenCL see 857 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 858 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 859 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 860 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 861 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 862 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 863 execution unit. Generated by the ``amdgpu_waves_per_eu`` 864 CLANG attribute [CLANG-ATTR]_. 865 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 866 mode register to be set on entry. Overrides the default for 867 the calling convention. 868 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 869 the mode register to be set on entry. Overrides the default 870 for the calling convention. 871 ======================================= ========================================================== 872 873.. _amdgpu-elf-code-object: 874 875ELF Code Object 876=============== 877 878The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 879can be linked by ``lld`` to produce a standard ELF shared code object which can 880be loaded and executed on an AMDGPU target. 881 882.. _amdgpu-elf-header: 883 884Header 885------ 886 887The AMDGPU backend uses the following ELF header: 888 889 .. table:: AMDGPU ELF Header 890 :name: amdgpu-elf-header-table 891 892 ========================== =============================== 893 Field Value 894 ========================== =============================== 895 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 896 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 897 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 898 - ``ELFOSABI_AMDGPU_HSA`` 899 - ``ELFOSABI_AMDGPU_PAL`` 900 - ``ELFOSABI_AMDGPU_MESA3D`` 901 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 902 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 903 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 904 - ``ELFABIVERSION_AMDGPU_PAL`` 905 - ``ELFABIVERSION_AMDGPU_MESA3D`` 906 ``e_type`` - ``ET_REL`` 907 - ``ET_DYN`` 908 ``e_machine`` ``EM_AMDGPU`` 909 ``e_entry`` 0 910 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 911 :ref:`amdgpu-elf-header-e_flags-table-v3`, 912 and :ref:`amdgpu-elf-header-e_flags-table-v4` 913 ========================== =============================== 914 915.. 916 917 .. table:: AMDGPU ELF Header Enumeration Values 918 :name: amdgpu-elf-header-enumeration-values-table 919 920 =============================== ===== 921 Name Value 922 =============================== ===== 923 ``EM_AMDGPU`` 224 924 ``ELFOSABI_NONE`` 0 925 ``ELFOSABI_AMDGPU_HSA`` 64 926 ``ELFOSABI_AMDGPU_PAL`` 65 927 ``ELFOSABI_AMDGPU_MESA3D`` 66 928 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 929 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 930 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 931 ``ELFABIVERSION_AMDGPU_PAL`` 0 932 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 933 =============================== ===== 934 935``e_ident[EI_CLASS]`` 936 The ELF class is: 937 938 * ``ELFCLASS32`` for ``r600`` architecture. 939 940 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 941 process address space applications. 942 943``e_ident[EI_DATA]`` 944 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 945 946``e_ident[EI_OSABI]`` 947 One of the following AMDGPU target architecture specific OS ABIs 948 (see :ref:`amdgpu-os`): 949 950 * ``ELFOSABI_NONE`` for *unknown* OS. 951 952 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 953 954 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 955 956 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 957 958``e_ident[EI_ABIVERSION]`` 959 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 960 object conforms: 961 962 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 963 runtime ABI for code object V2. Specify using the Clang option 964 ``-mcode-object-version=2``. 965 966 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 967 runtime ABI for code object V3. Specify using the Clang option 968 ``-mcode-object-version=3``. This is the default code object 969 version if not specified. 970 971 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 972 runtime ABI for code object V4. Specify using the Clang option 973 ``-mcode-object-version=4``. 974 975 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 976 runtime ABI. 977 978 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 979 3D runtime ABI. 980 981``e_type`` 982 Can be one of the following values: 983 984 985 ``ET_REL`` 986 The type produced by the AMDGPU backend compiler as it is relocatable code 987 object. 988 989 ``ET_DYN`` 990 The type produced by the linker as it is a shared code object. 991 992 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 993 994``e_machine`` 995 The value ``EM_AMDGPU`` is used for the machine for all processors supported 996 by the ``r600`` and ``amdgcn`` architectures (see 997 :ref:`amdgpu-processor-table`). The specific processor is specified in the 998 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 999 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1000 ``e_flags`` for code object V3 to V4 (see 1001 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1002 :ref:`amdgpu-elf-header-e_flags-table-v4`). 1003 1004``e_entry`` 1005 The entry point is 0 as the entry points for individual kernels must be 1006 selected in order to invoke them through AQL packets. 1007 1008``e_flags`` 1009 The AMDGPU backend uses the following ELF header flags: 1010 1011 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1012 :name: amdgpu-elf-header-e_flags-v2-table 1013 1014 ===================================== ===== ============================= 1015 Name Value Description 1016 ===================================== ===== ============================= 1017 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1018 target feature is 1019 enabled for all code 1020 contained in the code object. 1021 If the processor 1022 does not support the 1023 ``xnack`` target 1024 feature then must 1025 be 0. 1026 See 1027 :ref:`amdgpu-target-features`. 1028 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1029 handler is enabled for all 1030 code contained in the code 1031 object. If the processor 1032 does not support a trap 1033 handler then must be 0. 1034 See 1035 :ref:`amdgpu-target-features`. 1036 ===================================== ===== ============================= 1037 1038 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1039 :name: amdgpu-elf-header-e_flags-table-v3 1040 1041 ================================= ===== ============================= 1042 Name Value Description 1043 ================================= ===== ============================= 1044 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1045 mask for 1046 ``EF_AMDGPU_MACH_xxx`` values 1047 defined in 1048 :ref:`amdgpu-ef-amdgpu-mach-table`. 1049 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1050 target feature is 1051 enabled for all code 1052 contained in the code object. 1053 If the processor 1054 does not support the 1055 ``xnack`` target 1056 feature then must 1057 be 0. 1058 See 1059 :ref:`amdgpu-target-features`. 1060 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1061 target feature is 1062 enabled for all code 1063 contained in the code object. 1064 If the processor 1065 does not support the 1066 ``sramecc`` target 1067 feature then must 1068 be 0. 1069 See 1070 :ref:`amdgpu-target-features`. 1071 ================================= ===== ============================= 1072 1073 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1074 :name: amdgpu-elf-header-e_flags-table-v4 1075 1076 ============================================ ===== =================================== 1077 Name Value Description 1078 ============================================ ===== =================================== 1079 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1080 mask for 1081 ``EF_AMDGPU_MACH_xxx`` values 1082 defined in 1083 :ref:`amdgpu-ef-amdgpu-mach-table`. 1084 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1085 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1086 values. 1087 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1088 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1089 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1090 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1091 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1092 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1093 values. 1094 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1095 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1096 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1097 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1098 ============================================ ===== =================================== 1099 1100 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1101 :name: amdgpu-ef-amdgpu-mach-table 1102 1103 ==================================== ========== ============================= 1104 Name Value Description (see 1105 :ref:`amdgpu-processor-table`) 1106 ==================================== ========== ============================= 1107 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1108 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1109 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1110 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1111 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1112 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1113 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1114 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1115 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1116 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1117 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1118 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1119 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1120 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1121 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1122 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1123 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1124 *reserved* 0x011 - Reserved for ``r600`` 1125 0x01f architecture processors. 1126 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1127 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1128 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1129 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1130 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1131 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1132 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1133 *reserved* 0x027 Reserved. 1134 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1135 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1136 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1137 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1138 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1139 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1140 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1141 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1142 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1143 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1144 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1145 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1146 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1147 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1148 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1149 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1150 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1151 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1152 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1153 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1154 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1155 *reserved* 0x03d Reserved. 1156 *reserved* 0x03e Reserved. 1157 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1158 *reserved* 0x040 Reserved. 1159 ==================================== ========== ============================= 1160 1161Sections 1162-------- 1163 1164An AMDGPU target ELF code object has the standard ELF sections which include: 1165 1166 .. table:: AMDGPU ELF Sections 1167 :name: amdgpu-elf-sections-table 1168 1169 ================== ================ ================================= 1170 Name Type Attributes 1171 ================== ================ ================================= 1172 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1173 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1174 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1175 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1176 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1177 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1178 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1179 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1180 ``.note`` ``SHT_NOTE`` *none* 1181 ``.rela``\ *name* ``SHT_RELA`` *none* 1182 ``.rela.dyn`` ``SHT_RELA`` *none* 1183 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1184 ``.shstrtab`` ``SHT_STRTAB`` *none* 1185 ``.strtab`` ``SHT_STRTAB`` *none* 1186 ``.symtab`` ``SHT_SYMTAB`` *none* 1187 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1188 ================== ================ ================================= 1189 1190These sections have their standard meanings (see [ELF]_) and are only generated 1191if needed. 1192 1193``.debug``\ *\** 1194 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1195 information on the DWARF produced by the AMDGPU backend. 1196 1197``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1198 The standard sections used by a dynamic loader. 1199 1200``.note`` 1201 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1202 backend. 1203 1204``.rela``\ *name*, ``.rela.dyn`` 1205 For relocatable code objects, *name* is the name of the section that the 1206 relocation records apply. For example, ``.rela.text`` is the section name for 1207 relocation records associated with the ``.text`` section. 1208 1209 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1210 records from each of the relocatable code object's ``.rela``\ *name* sections. 1211 1212 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1213 the AMDGPU backend. 1214 1215``.text`` 1216 The executable machine code for the kernels and functions they call. Generated 1217 as position independent code. See :ref:`amdgpu-code-conventions` for 1218 information on conventions used in the isa generation. 1219 1220.. _amdgpu-note-records: 1221 1222Note Records 1223------------ 1224 1225The AMDGPU backend code object contains ELF note records in the ``.note`` 1226section. The set of generated notes and their semantics depend on the code 1227object version; see :ref:`amdgpu-note-records-v2` and 1228:ref:`amdgpu-note-records-v3-v4`. 1229 1230As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1231must be generated after the ``name`` field to ensure the ``desc`` field is 4 1232byte aligned. In addition, minimal zero-byte padding must be generated to 1233ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1234field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1235alignment. 1236 1237.. _amdgpu-note-records-v2: 1238 1239Code Object V2 Note Records 1240~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1241 1242.. warning:: 1243 Code object V2 is not the default code object version emitted by 1244 this version of LLVM. 1245 1246The AMDGPU backend code object uses the following ELF note record in the 1247``.note`` section when compiling for code object V2. 1248 1249The note record vendor field is "AMD". 1250 1251Additional note records may be present, but any which are not documented here 1252are deprecated and should not be used. 1253 1254 .. table:: AMDGPU Code Object V2 ELF Note Records 1255 :name: amdgpu-elf-note-records-v2-table 1256 1257 ===== ===================================== ====================================== 1258 Name Type Description 1259 ===== ===================================== ====================================== 1260 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1261 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1262 Finalizer and not the LLVM compiler. 1263 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1264 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1265 YAML [YAML]_ textual format. 1266 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1267 ===== ===================================== ====================================== 1268 1269.. 1270 1271 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1272 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1273 1274 ===================================== ===== 1275 Name Value 1276 ===================================== ===== 1277 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1278 ``NT_AMD_HSA_HSAIL`` 2 1279 ``NT_AMD_HSA_ISA_VERSION`` 3 1280 *reserved* 4-9 1281 ``NT_AMD_HSA_METADATA`` 10 1282 ``NT_AMD_HSA_ISA_NAME`` 11 1283 ===================================== ===== 1284 1285``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1286 Specifies the code object version number. The description field has the 1287 following layout: 1288 1289 .. code:: 1290 1291 struct amdgpu_hsa_note_code_object_version_s { 1292 uint32_t major_version; 1293 uint32_t minor_version; 1294 }; 1295 1296 The ``major_version`` has a value less than or equal to 2. 1297 1298``NT_AMD_HSA_HSAIL`` 1299 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1300 field has the following layout: 1301 1302 .. code:: 1303 1304 struct amdgpu_hsa_note_hsail_s { 1305 uint32_t hsail_major_version; 1306 uint32_t hsail_minor_version; 1307 uint8_t profile; 1308 uint8_t machine_model; 1309 uint8_t default_float_round; 1310 }; 1311 1312``NT_AMD_HSA_ISA_VERSION`` 1313 Specifies the target ISA version. The description field has the following layout: 1314 1315 .. code:: 1316 1317 struct amdgpu_hsa_note_isa_s { 1318 uint16_t vendor_name_size; 1319 uint16_t architecture_name_size; 1320 uint32_t major; 1321 uint32_t minor; 1322 uint32_t stepping; 1323 char vendor_and_architecture_name[1]; 1324 }; 1325 1326 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1327 vendor and architecture names respectively, including the NUL character. 1328 1329 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1330 vendor, immediately followed by the NUL terminated string for the 1331 architecture. 1332 1333 This note record is used by the HSA runtime loader. 1334 1335 Code object V2 only supports a limited number of processors and has fixed 1336 settings for target features. See 1337 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1338 processors and the corresponding target ID. In the table the note record ISA 1339 name is a concatenation of the vendor name, architecture name, major, minor, 1340 and stepping separated by a ":". 1341 1342 The target ID column shows the processor name and fixed target features used 1343 by the LLVM compiler. The LLVM compiler does not generate a 1344 ``NT_AMD_HSA_HSAIL`` note record. 1345 1346 A code object generated by the Finalizer also uses code object V2 and always 1347 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1348 ``sramecc`` target feature is as shown in 1349 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1350 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1351 bit. 1352 1353``NT_AMD_HSA_ISA_NAME`` 1354 Specifies the target ISA name as a non-NUL terminated string. 1355 1356 This note record is not used by the HSA runtime loader. 1357 1358 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1359 V2's limited support of processors and fixed settings for target features. 1360 1361 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1362 from the string to the corresponding target ID. If the ``xnack`` target 1363 feature is supported and enabled, the string produced by the LLVM compiler 1364 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1365 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1366 1367``NT_AMD_HSA_METADATA`` 1368 Specifies extensible metadata associated with the code objects executed on HSA 1369 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1370 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1371 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1372 metadata string. 1373 1374 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1375 :name: amdgpu-elf-note-record-supported_processors-v2-table 1376 1377 ==================== ========================== 1378 Note Record ISA Name Target ID 1379 ==================== ========================== 1380 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1381 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1382 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1383 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1384 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1385 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1386 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1387 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1388 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1389 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1390 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1391 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1392 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1393 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1394 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1395 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1396 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1397 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1398 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1399 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1400 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1401 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1402 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1403 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1404 ==================== ========================== 1405 1406.. _amdgpu-note-records-v3-v4: 1407 1408Code Object V3 to V4 Note Records 1409~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1410 1411The AMDGPU backend code object uses the following ELF note record in the 1412``.note`` section when compiling for code object V3 to V4. 1413 1414The note record vendor field is "AMDGPU". 1415 1416Additional note records may be present, but any which are not documented here 1417are deprecated and should not be used. 1418 1419 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1420 :name: amdgpu-elf-note-records-table-v3-v4 1421 1422 ======== ============================== ====================================== 1423 Name Type Description 1424 ======== ============================== ====================================== 1425 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1426 binary format. 1427 ======== ============================== ====================================== 1428 1429.. 1430 1431 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1432 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1433 1434 ============================== ===== 1435 Name Value 1436 ============================== ===== 1437 *reserved* 0-31 1438 ``NT_AMDGPU_METADATA`` 32 1439 ============================== ===== 1440 1441``NT_AMDGPU_METADATA`` 1442 Specifies extensible metadata associated with an AMDGPU code object. It is 1443 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1444 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1445 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1446 ``amdhsa`` OS. 1447 1448.. _amdgpu-symbols: 1449 1450Symbols 1451------- 1452 1453Symbols include the following: 1454 1455 .. table:: AMDGPU ELF Symbols 1456 :name: amdgpu-elf-symbols-table 1457 1458 ===================== ================== ================ ================== 1459 Name Type Section Description 1460 ===================== ================== ================ ================== 1461 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1462 - ``.rodata`` 1463 - ``.bss`` 1464 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1465 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1466 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1467 ===================== ================== ================ ================== 1468 1469Global variable 1470 Global variables both used and defined by the compilation unit. 1471 1472 If the symbol is defined in the compilation unit then it is allocated in the 1473 appropriate section according to if it has initialized data or is readonly. 1474 1475 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1476 will resolve relocations using the definition provided by another code object 1477 or explicitly defined by the runtime. 1478 1479 If the symbol resides in local/group memory (LDS) then its section is the 1480 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1481 ``st_value`` field describes alignment requirements as it does for common 1482 symbols. 1483 1484 .. TODO:: 1485 1486 Add description of linked shared object symbols. Seems undefined symbols 1487 are marked as STT_NOTYPE. 1488 1489Kernel descriptor 1490 Every HSA kernel has an associated kernel descriptor. It is the address of the 1491 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1492 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1493 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1494 1495Kernel entry point 1496 Every HSA kernel also has a symbol for its machine code entry point. 1497 1498.. _amdgpu-relocation-records: 1499 1500Relocation Records 1501------------------ 1502 1503AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1504relocatable fields are: 1505 1506``word32`` 1507 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1508 alignment. These values use the same byte order as other word values in the 1509 AMDGPU architecture. 1510 1511``word64`` 1512 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1513 alignment. These values use the same byte order as other word values in the 1514 AMDGPU architecture. 1515 1516Following notations are used for specifying relocation calculations: 1517 1518**A** 1519 Represents the addend used to compute the value of the relocatable field. 1520 1521**G** 1522 Represents the offset into the global offset table at which the relocation 1523 entry's symbol will reside during execution. 1524 1525**GOT** 1526 Represents the address of the global offset table. 1527 1528**P** 1529 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1530 of the storage unit being relocated (computed using ``r_offset``). 1531 1532**S** 1533 Represents the value of the symbol whose index resides in the relocation 1534 entry. Relocations not using this must specify a symbol index of 1535 ``STN_UNDEF``. 1536 1537**B** 1538 Represents the base address of a loaded executable or shared object which is 1539 the difference between the ELF address and the actual load address. 1540 Relocations using this are only valid in executable or shared objects. 1541 1542The following relocation types are supported: 1543 1544 .. table:: AMDGPU ELF Relocation Records 1545 :name: amdgpu-elf-relocation-records-table 1546 1547 ========================== ======= ===== ========== ============================== 1548 Relocation Type Kind Value Field Calculation 1549 ========================== ======= ===== ========== ============================== 1550 ``R_AMDGPU_NONE`` 0 *none* *none* 1551 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1552 Dynamic 1553 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1554 Dynamic 1555 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1556 Dynamic 1557 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1558 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1559 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1560 Dynamic 1561 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1562 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1563 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1564 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1565 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1566 *reserved* 12 1567 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1568 ========================== ======= ===== ========== ============================== 1569 1570``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1571the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1572 1573There is no current OS loader support for 32-bit programs and so 1574``R_AMDGPU_ABS32`` is not used. 1575 1576.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1577 1578Loaded Code Object Path Uniform Resource Identifier (URI) 1579--------------------------------------------------------- 1580 1581The AMD GPU code object loader represents the path of the ELF shared object from 1582which the code object was loaded as a textual Unifom Resource Identifier (URI). 1583Note that the code object is the in memory loaded relocated form of the ELF 1584shared object. Multiple code objects may be loaded at different memory 1585addresses in the same process from the same ELF shared object. 1586 1587The loaded code object path URI syntax is defined by the following BNF syntax: 1588 1589.. code:: 1590 1591 code_object_uri ::== file_uri | memory_uri 1592 file_uri ::== "file://" file_path [ range_specifier ] 1593 memory_uri ::== "memory://" process_id range_specifier 1594 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1595 file_path ::== URI_ENCODED_OS_FILE_PATH 1596 process_id ::== DECIMAL_NUMBER 1597 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1598 1599**number** 1600 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1601 and octal values by "0". 1602 1603**file_path** 1604 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1605 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1606 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1607 the path are separated by "/". 1608 1609**offset** 1610 Is a 0-based byte offset to the start of the code object. For a file URI, it 1611 is from the start of the file specified by the ``file_path``, and if omitted 1612 defaults to 0. For a memory URI, it is the memory address and is required. 1613 1614**size** 1615 Is the number of bytes in the code object. For a file URI, if omitted it 1616 defaults to the size of the file. It is required for a memory URI. 1617 1618**process_id** 1619 Is the identity of the process owning the memory. For Linux it is the C 1620 unsigned integral decimal literal for the process ID (PID). 1621 1622For example: 1623 1624.. code:: 1625 1626 file:///dir1/dir2/file1 1627 file:///dir3/dir4/file2#offset=0x2000&size=3000 1628 memory://1234#offset=0x20000&size=3000 1629 1630.. _amdgpu-dwarf-debug-information: 1631 1632DWARF Debug Information 1633======================= 1634 1635.. warning:: 1636 1637 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1638 is not currently fully implemented and is subject to change. 1639 1640AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1641:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1642object executable code and data to the source language constructs. It can be 1643used by tools such as debuggers and profilers. It uses features defined in 1644:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1645DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1646 1647This section defines the AMDGPU target architecture specific DWARF mappings. 1648 1649.. _amdgpu-dwarf-register-identifier: 1650 1651Register Identifier 1652------------------- 1653 1654This section defines the AMDGPU target architecture register numbers used in 1655DWARF operation expressions (see DWARF Version 5 section 2.5 and 1656:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1657instructions (see DWARF Version 5 section 6.4 and 1658:ref:`amdgpu-dwarf-call-frame-information`). 1659 1660A single code object can contain code for kernels that have different wavefront 1661sizes. The vector registers and some scalar registers are based on the wavefront 1662size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1663simplifies the consumer of the DWARF so that each register has a fixed size, 1664rather than being dynamic according to the wavefront size mode. Similarly, 1665distinct DWARF registers are defined for those registers that vary in size 1666according to the process address size. This allows a consumer to treat a 1667specific AMDGPU processor as a single architecture regardless of how it is 1668configured at run time. The compiler explicitly specifies the DWARF registers 1669that match the mode in which the code it is generating will be executed. 1670 1671DWARF registers are encoded as numbers, which are mapped to architecture 1672registers. The mapping for AMDGPU is defined in 1673:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1674mapping. 1675 1676.. table:: AMDGPU DWARF Register Mapping 1677 :name: amdgpu-dwarf-register-mapping-table 1678 1679 ============== ================= ======== ================================== 1680 DWARF Register AMDGPU Register Bit Size Description 1681 ============== ================= ======== ================================== 1682 0 PC_32 32 Program Counter (PC) when 1683 executing in a 32-bit process 1684 address space. Used in the CFI to 1685 describe the PC of the calling 1686 frame. 1687 1 EXEC_MASK_32 32 Execution Mask Register when 1688 executing in wavefront 32 mode. 1689 2-15 *Reserved* *Reserved for highly accessed 1690 registers using DWARF shortcut.* 1691 16 PC_64 64 Program Counter (PC) when 1692 executing in a 64-bit process 1693 address space. Used in the CFI to 1694 describe the PC of the calling 1695 frame. 1696 17 EXEC_MASK_64 64 Execution Mask Register when 1697 executing in wavefront 64 mode. 1698 18-31 *Reserved* *Reserved for highly accessed 1699 registers using DWARF shortcut.* 1700 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1701 Registers. 1702 96-127 *Reserved* *Reserved for frequently accessed 1703 registers using DWARF 1-byte ULEB.* 1704 128 STATUS 32 Status Register. 1705 129-511 *Reserved* *Reserved for future Scalar 1706 Architectural Registers.* 1707 512 VCC_32 32 Vector Condition Code Register 1708 when executing in wavefront 32 1709 mode. 1710 513-1023 *Reserved* *Reserved for future Vector 1711 Architectural Registers when 1712 executing in wavefront 32 mode.* 1713 768 VCC_64 64 Vector Condition Code Register 1714 when executing in wavefront 64 1715 mode. 1716 769-1023 *Reserved* *Reserved for future Vector 1717 Architectural Registers when 1718 executing in wavefront 64 mode.* 1719 1024-1087 *Reserved* *Reserved for padding.* 1720 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1721 1130-1535 *Reserved* *Reserved for future Scalar 1722 General Purpose Registers.* 1723 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1724 when executing in wavefront 32 1725 mode. 1726 1792-2047 *Reserved* *Reserved for future Vector 1727 General Purpose Registers when 1728 executing in wavefront 32 mode.* 1729 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1730 when executing in wavefront 32 1731 mode. 1732 2304-2559 *Reserved* *Reserved for future Vector 1733 Accumulation Registers when 1734 executing in wavefront 32 mode.* 1735 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1736 when executing in wavefront 64 1737 mode. 1738 2816-3071 *Reserved* *Reserved for future Vector 1739 General Purpose Registers when 1740 executing in wavefront 64 mode.* 1741 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1742 when executing in wavefront 64 1743 mode. 1744 3328-3583 *Reserved* *Reserved for future Vector 1745 Accumulation Registers when 1746 executing in wavefront 64 mode.* 1747 ============== ================= ======== ================================== 1748 1749The vector registers are represented as the full size for the wavefront. They 1750are organized as consecutive dwords (32-bits), one per lane, with the dword at 1751the least significant bit position corresponding to lane 0 and so forth. DWARF 1752location expressions involving the ``DW_OP_LLVM_offset`` and 1753``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1754register corresponding to the lane that is executing the current thread of 1755execution in languages that are implemented using a SIMD or SIMT execution 1756model. 1757 1758If the wavefront size is 32 lanes then the wavefront 32 mode register 1759definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1760mode register definitions are used. Some AMDGPU targets support executing in 1761both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1762to the wavefront mode of the generated code will be used. 1763 1764If code is generated to execute in a 32-bit process address space, then the 176532-bit process address space register definitions are used. If code is generated 1766to execute in a 64-bit process address space, then the 64-bit process address 1767space register definitions are used. The ``amdgcn`` target only supports the 176864-bit process address space. 1769 1770.. _amdgpu-dwarf-address-class-identifier: 1771 1772Address Class Identifier 1773------------------------ 1774 1775The DWARF address class represents the source language memory space. See DWARF 1776Version 5 section 2.12 which is updated by the *DWARF Extensions For 1777Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1778 1779The DWARF address class mapping used for AMDGPU is defined in 1780:ref:`amdgpu-dwarf-address-class-mapping-table`. 1781 1782.. table:: AMDGPU DWARF Address Class Mapping 1783 :name: amdgpu-dwarf-address-class-mapping-table 1784 1785 ========================= ====== ================= 1786 DWARF AMDGPU 1787 -------------------------------- ----------------- 1788 Address Class Name Value Address Space 1789 ========================= ====== ================= 1790 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1791 ``DW_ADDR_LLVM_global`` 0x0001 Global 1792 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1793 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1794 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1795 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1796 ========================= ====== ================= 1797 1798The DWARF address class values defined in the *DWARF Extensions For 1799Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1800 1801In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1802available for use for the AMD extension for access to the hardware GDS memory 1803which is scratchpad memory allocated per device. 1804 1805For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1806address class of ``DW_ADDR_none`` is used. 1807 1808See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1809mapping of DWARF address classes to DWARF address spaces, including address size 1810and NULL value. 1811 1812.. _amdgpu-dwarf-address-space-identifier: 1813 1814Address Space Identifier 1815------------------------ 1816 1817DWARF address spaces correspond to target architecture specific linear 1818addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1819For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1820 1821The DWARF address space mapping used for AMDGPU is defined in 1822:ref:`amdgpu-dwarf-address-space-mapping-table`. 1823 1824.. table:: AMDGPU DWARF Address Space Mapping 1825 :name: amdgpu-dwarf-address-space-mapping-table 1826 1827 ======================================= ===== ======= ======== ================= ======================= 1828 DWARF AMDGPU Notes 1829 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1830 Address Space Name Value Address Bit Size Address Space 1831 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1832 .. 64-bit 32-bit 1833 process process 1834 address address 1835 space space 1836 ======================================= ===== ======= ======== ================= ======================= 1837 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1838 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1839 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1840 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1841 *Reserved* 0x04 1842 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1843 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1844 ======================================= ===== ======= ======== ================= ======================= 1845 1846See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1847including address size and NULL value. 1848 1849The ``DW_ASPACE_none`` address space is the default target architecture address 1850space used in DWARF operations that do not specify an address space. It 1851therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1852related operations can refer to addresses in the program code. 1853 1854The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1855specify the flat address space. If the address corresponds to an address in the 1856local address space, then it corresponds to the wavefront that is executing the 1857focused thread of execution. If the address corresponds to an address in the 1858private address space, then it corresponds to the lane that is executing the 1859focused thread of execution for languages that are implemented using a SIMD or 1860SIMT execution model. 1861 1862.. note:: 1863 1864 CUDA-like languages such as HIP that do not have address spaces in the 1865 language type system, but do allow variables to be allocated in different 1866 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1867 address space in the DWARF expression operations as the default address space 1868 is the global address space. 1869 1870The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1871specify the local address space corresponding to the wavefront that is executing 1872the focused thread of execution. 1873 1874The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1875to specify the private address space corresponding to the lane that is executing 1876the focused thread of execution for languages that are implemented using a SIMD 1877or SIMT execution model. 1878 1879The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1880to specify the unswizzled private address space corresponding to the wavefront 1881that is executing the focused thread of execution. The wavefront view of private 1882memory is the per wavefront unswizzled backing memory layout defined in 1883:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1884location for the backing memory of the wavefront (namely the address is not 1885offset by ``wavefront-scratch-base``). The following formula can be used to 1886convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1887``DW_ASPACE_AMDGPU_private_wave`` address: 1888 1889:: 1890 1891 private-address-wavefront = 1892 ((private-address-lane / 4) * wavefront-size * 4) + 1893 (wavefront-lane-id * 4) + (private-address-lane % 4) 1894 1895If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1896of the dwords for each lane starting with lane 0 is required, then this 1897simplifies to: 1898 1899:: 1900 1901 private-address-wavefront = 1902 private-address-lane * wavefront-size 1903 1904A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1905complete spilled vector register back into a complete vector register in the 1906CFI. The frame pointer can be a private lane address which is dword aligned, 1907which can be shifted to multiply by the wavefront size, and then used to form a 1908private wavefront address that gives a location for a contiguous set of dwords, 1909one per lane, where the vector register dwords are spilled. The compiler knows 1910the wavefront size since it generates the code. Note that the type of the 1911address may have to be converted as the size of a 1912``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1913``DW_ASPACE_AMDGPU_private_wave`` address. 1914 1915.. _amdgpu-dwarf-lane-identifier: 1916 1917Lane identifier 1918--------------- 1919 1920DWARF lane identifies specify a target architecture lane position for hardware 1921that executes in a SIMD or SIMT manner, and on which a source language maps its 1922threads of execution onto those lanes. The DWARF lane identifier is pushed by 1923the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1924section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1925section :ref:`amdgpu-dwarf-operation-expressions`. 1926 1927For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1928wavefront. It is numbered from 0 to the wavefront size minus 1. 1929 1930Operation Expressions 1931--------------------- 1932 1933DWARF expressions are used to compute program values and the locations of 1934program objects. See DWARF Version 5 section 2.5 and 1935:ref:`amdgpu-dwarf-operation-expressions`. 1936 1937DWARF location descriptions describe how to access storage which includes memory 1938and registers. When accessing storage on AMDGPU, bytes are ordered with least 1939significant bytes first, and bits are ordered within bytes with least 1940significant bits first. 1941 1942For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1943unwinding vector registers that are spilled under the execution mask to memory: 1944the zero-single location description is the vector register, and the one-single 1945location description is the spilled memory location description. The 1946``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1947memory location description. 1948 1949In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1950``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1951controlled by the execution mask. An undefined location description together 1952with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1953to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1954 1955Debugger Information Entry Attributes 1956------------------------------------- 1957 1958This section describes how certain debugger information entry attributes are 1959used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1960by *DWARF Extensions For Heterogeneous Debugging* section 1961:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1962 1963.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1964 1965``DW_AT_LLVM_lane_pc`` 1966~~~~~~~~~~~~~~~~~~~~~~ 1967 1968For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1969location of the separate lanes of a SIMT thread. 1970 1971If the lane is an active lane then this will be the same as the current program 1972location. 1973 1974If the lane is inactive, but was active on entry to the subprogram, then this is 1975the program location in the subprogram at which execution of the lane is 1976conceptual positioned. 1977 1978If the lane was not active on entry to the subprogram, then this will be the 1979undefined location. A client debugger can check if the lane is part of a valid 1980work-group by checking that the lane is in the range of the associated 1981work-group within the grid, accounting for partial work-groups. If it is not, 1982then the debugger can omit any information for the lane. Otherwise, the debugger 1983may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1984calling subprogram until it finds a non-undefined location. Conceptually the 1985lane only has the call frames that it has a non-undefined 1986``DW_AT_LLVM_lane_pc``. 1987 1988The following example illustrates how the AMDGPU backend can generate a DWARF 1989location list expression for the nested ``IF/THEN/ELSE`` structures of the 1990following subprogram pseudo code for a target with 64 lanes per wavefront. 1991 1992.. code:: 1993 :number-lines: 1994 1995 SUBPROGRAM X 1996 BEGIN 1997 a; 1998 IF (c1) THEN 1999 b; 2000 IF (c2) THEN 2001 c; 2002 ELSE 2003 d; 2004 ENDIF 2005 e; 2006 ELSE 2007 f; 2008 ENDIF 2009 g; 2010 END 2011 2012The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2013execution mask (``EXEC``) to linearize the control flow. The condition is 2014evaluated to make a mask of the lanes for which the condition evaluates to true. 2015First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2016logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2017``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2018the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2019region the ``EXEC`` mask is restored to the value it had at the beginning of the 2020region. This is shown below. Other approaches are possible, but the basic 2021concept is the same. 2022 2023.. code:: 2024 :number-lines: 2025 2026 $lex_start: 2027 a; 2028 %1 = EXEC 2029 %2 = c1 2030 $lex_1_start: 2031 EXEC = %1 & %2 2032 $if_1_then: 2033 b; 2034 %3 = EXEC 2035 %4 = c2 2036 $lex_1_1_start: 2037 EXEC = %3 & %4 2038 $lex_1_1_then: 2039 c; 2040 EXEC = ~EXEC & %3 2041 $lex_1_1_else: 2042 d; 2043 EXEC = %3 2044 $lex_1_1_end: 2045 e; 2046 EXEC = ~EXEC & %1 2047 $lex_1_else: 2048 f; 2049 EXEC = %1 2050 $lex_1_end: 2051 g; 2052 $lex_end: 2053 2054To create the DWARF location list expression that defines the location 2055description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2056pseudo instruction can be used to annotate the linearized control flow. This can 2057be done by defining an artificial variable for the lane PC. The DWARF location 2058list expression created for it is used as the value of the 2059``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2060 2061A DWARF procedure is defined for each well nested structured control flow region 2062which provides the conceptual lane program location for a lane if it is not 2063active (namely it is divergent). The DWARF operation expression for each region 2064conceptually inherits the value of the immediately enclosing region and modifies 2065it according to the semantics of the region. 2066 2067For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2068the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2069region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2070region since the ``THEN`` region has completed. 2071 2072The lane PC artificial variable is assigned at each region transition. It uses 2073the immediately enclosing region's DWARF procedure to compute the program 2074location for each lane assuming they are divergent, and then modifies the result 2075by inserting the current program location for each lane that the ``EXEC`` mask 2076indicates is active. 2077 2078By having separate DWARF procedures for each region, they can be reused to 2079define the value for any nested region. This reduces the total size of the DWARF 2080operation expressions. 2081 2082The following provides an example using pseudo LLVM MIR. 2083 2084.. code:: 2085 :number-lines: 2086 2087 $lex_start: 2088 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2089 DW_AT_name = "__uint64"; 2090 DW_AT_byte_size = 8; 2091 DW_AT_encoding = DW_ATE_unsigned; 2092 ]; 2093 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2094 DW_AT_name = "__active_lane_pc"; 2095 DW_AT_location = [ 2096 DW_OP_regx PC; 2097 DW_OP_LLVM_extend 64, 64; 2098 DW_OP_regval_type EXEC, %uint_64; 2099 DW_OP_LLVM_select_bit_piece 64, 64; 2100 ]; 2101 ]; 2102 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2103 DW_AT_name = "__divergent_lane_pc"; 2104 DW_AT_location = [ 2105 DW_OP_LLVM_undefined; 2106 DW_OP_LLVM_extend 64, 64; 2107 ]; 2108 ]; 2109 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2110 DW_OP_call_ref %__divergent_lane_pc; 2111 DW_OP_call_ref %__active_lane_pc; 2112 ]; 2113 a; 2114 %1 = EXEC; 2115 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2116 %2 = c1; 2117 $lex_1_start: 2118 EXEC = %1 & %2; 2119 $lex_1_then: 2120 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2121 DW_AT_name = "__divergent_lane_pc_1_then"; 2122 DW_AT_location = DIExpression[ 2123 DW_OP_call_ref %__divergent_lane_pc; 2124 DW_OP_addrx &lex_1_start; 2125 DW_OP_stack_value; 2126 DW_OP_LLVM_extend 64, 64; 2127 DW_OP_call_ref %__lex_1_save_exec; 2128 DW_OP_deref_type 64, %__uint_64; 2129 DW_OP_LLVM_select_bit_piece 64, 64; 2130 ]; 2131 ]; 2132 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2133 DW_OP_call_ref %__divergent_lane_pc_1_then; 2134 DW_OP_call_ref %__active_lane_pc; 2135 ]; 2136 b; 2137 %3 = EXEC; 2138 DBG_VALUE %3, %__lex_1_1_save_exec; 2139 %4 = c2; 2140 $lex_1_1_start: 2141 EXEC = %3 & %4; 2142 $lex_1_1_then: 2143 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2144 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2145 DW_AT_location = DIExpression[ 2146 DW_OP_call_ref %__divergent_lane_pc_1_then; 2147 DW_OP_addrx &lex_1_1_start; 2148 DW_OP_stack_value; 2149 DW_OP_LLVM_extend 64, 64; 2150 DW_OP_call_ref %__lex_1_1_save_exec; 2151 DW_OP_deref_type 64, %__uint_64; 2152 DW_OP_LLVM_select_bit_piece 64, 64; 2153 ]; 2154 ]; 2155 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2156 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2157 DW_OP_call_ref %__active_lane_pc; 2158 ]; 2159 c; 2160 EXEC = ~EXEC & %3; 2161 $lex_1_1_else: 2162 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2163 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2164 DW_AT_location = DIExpression[ 2165 DW_OP_call_ref %__divergent_lane_pc_1_then; 2166 DW_OP_addrx &lex_1_1_end; 2167 DW_OP_stack_value; 2168 DW_OP_LLVM_extend 64, 64; 2169 DW_OP_call_ref %__lex_1_1_save_exec; 2170 DW_OP_deref_type 64, %__uint_64; 2171 DW_OP_LLVM_select_bit_piece 64, 64; 2172 ]; 2173 ]; 2174 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2175 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2176 DW_OP_call_ref %__active_lane_pc; 2177 ]; 2178 d; 2179 EXEC = %3; 2180 $lex_1_1_end: 2181 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2182 DW_OP_call_ref %__divergent_lane_pc; 2183 DW_OP_call_ref %__active_lane_pc; 2184 ]; 2185 e; 2186 EXEC = ~EXEC & %1; 2187 $lex_1_else: 2188 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2189 DW_AT_name = "__divergent_lane_pc_1_else"; 2190 DW_AT_location = DIExpression[ 2191 DW_OP_call_ref %__divergent_lane_pc; 2192 DW_OP_addrx &lex_1_end; 2193 DW_OP_stack_value; 2194 DW_OP_LLVM_extend 64, 64; 2195 DW_OP_call_ref %__lex_1_save_exec; 2196 DW_OP_deref_type 64, %__uint_64; 2197 DW_OP_LLVM_select_bit_piece 64, 64; 2198 ]; 2199 ]; 2200 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2201 DW_OP_call_ref %__divergent_lane_pc_1_else; 2202 DW_OP_call_ref %__active_lane_pc; 2203 ]; 2204 f; 2205 EXEC = %1; 2206 $lex_1_end: 2207 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2208 DW_OP_call_ref %__divergent_lane_pc; 2209 DW_OP_call_ref %__active_lane_pc; 2210 ]; 2211 g; 2212 $lex_end: 2213 2214The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2215that are active, with the current program location. 2216 2217Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2218the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2219instruction, location list entries will be created that describe where the 2220artificial variables are allocated at any given program location. The compiler 2221may allocate them to registers or spill them to memory. 2222 2223The DWARF procedures for each region use the values of the saved execution mask 2224artificial variables to only update the lanes that are active on entry to the 2225region. All other lanes retain the value of the enclosing region where they were 2226last active. If they were not active on entry to the subprogram, then will have 2227the undefined location description. 2228 2229Other structured control flow regions can be handled similarly. For example, 2230loops would set the divergent program location for the region at the end of the 2231loop. Any lanes active will be in the loop, and any lanes not active must have 2232exited the loop. 2233 2234An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2235``IF/THEN/ELSE`` regions. 2236 2237The DWARF procedures can use the active lane artificial variable described in 2238:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2239``EXEC`` mask in order to support whole or quad wavefront mode. 2240 2241.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2242 2243``DW_AT_LLVM_active_lane`` 2244~~~~~~~~~~~~~~~~~~~~~~~~~~ 2245 2246The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2247entry is used to specify the lanes that are conceptually active for a SIMT 2248thread. 2249 2250The execution mask may be modified to implement whole or quad wavefront mode 2251operations. For example, all lanes may need to temporarily be made active to 2252execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2253update it to enable the necessary lanes, perform the operations, and then 2254restore the ``EXEC`` mask from the saved value. While executing the whole 2255wavefront region, the conceptual execution mask is the saved value, not the 2256``EXEC`` value. 2257 2258This is handled by defining an artificial variable for the active lane mask. The 2259active lane mask artificial variable would be the actual ``EXEC`` mask for 2260normal regions, and the saved execution mask for regions where the mask is 2261temporarily updated. The location list expression created for this artificial 2262variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2263attribute. 2264 2265``DW_AT_LLVM_augmentation`` 2266~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2267 2268For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2269debugger information entry has the following value for the augmentation string: 2270 2271:: 2272 2273 [amdgpu:v0.0] 2274 2275The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2276extensions used in the DWARF of the compilation unit. The version number 2277conforms to [SEMVER]_. 2278 2279Call Frame Information 2280---------------------- 2281 2282DWARF Call Frame Information (CFI) describes how a consumer can virtually 2283*unwind* call frames in a running process or core dump. See DWARF Version 5 2284section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2285 2286For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2287 22881. ``augmentation`` string contains the following null-terminated UTF-8 string: 2289 2290 :: 2291 2292 [amd:v0.0] 2293 2294 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2295 extensions used in this CIE or to the FDEs that use it. The version number 2296 conforms to [SEMVER]_. 2297 22982. ``address_size`` for the ``Global`` address space is defined in 2299 :ref:`amdgpu-dwarf-address-space-identifier`. 2300 23013. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2302 23034. ``code_alignment_factor`` is 4 bytes. 2304 2305 .. TODO:: 2306 2307 Add to :ref:`amdgpu-processor-table` table. 2308 23095. ``data_alignment_factor`` is 4 bytes. 2310 2311 .. TODO:: 2312 2313 Add to :ref:`amdgpu-processor-table` table. 2314 23156. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2316 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2317 23187. ``initial_instructions`` Since a subprogram X with fewer registers can be 2319 called from subprogram Y that has more allocated, X will not change any of 2320 the extra registers as it cannot access them. Therefore, the default rule 2321 for all columns is ``same value``. 2322 2323For AMDGPU the register number follows the numbering defined in 2324:ref:`amdgpu-dwarf-register-identifier`. 2325 2326For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2327the return address to get the address of a byte within the call site 2328instructions. See DWARF Version 5 section 6.4.4. 2329 2330Accelerated Access 2331------------------ 2332 2333See DWARF Version 5 section 6.1. 2334 2335Lookup By Name Section Header 2336~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2337 2338See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2339 2340For AMDGPU the lookup by name section header table: 2341 2342``augmentation_string_size`` (uword) 2343 2344 Set to the length of the ``augmentation_string`` value which is always a 2345 multiple of 4. 2346 2347``augmentation_string`` (sequence of UTF-8 characters) 2348 2349 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2350 2351 :: 2352 2353 [amdgpu:v0.0] 2354 2355 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2356 extensions used in the DWARF of this index. The version number conforms to 2357 [SEMVER]_. 2358 2359 .. note:: 2360 2361 This is different to the DWARF Version 5 definition that requires the first 2362 4 characters to be the vendor ID. But this is consistent with the other 2363 augmentation strings and does allow multiple vendor contributions. However, 2364 backwards compatibility may be more desirable. 2365 2366Lookup By Address Section Header 2367~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2368 2369See DWARF Version 5 section 6.1.2. 2370 2371For AMDGPU the lookup by address section header table: 2372 2373``address_size`` (ubyte) 2374 2375 Match the address size for the ``Global`` address space defined in 2376 :ref:`amdgpu-dwarf-address-space-identifier`. 2377 2378``segment_selector_size`` (ubyte) 2379 2380 AMDGPU does not use a segment selector so this is 0. The entries in the 2381 ``.debug_aranges`` do not have a segment selector. 2382 2383Line Number Information 2384----------------------- 2385 2386See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2387 2388AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2389The instruction set must be obtained from the ELF file header ``e_flags`` field 2390in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2391<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2392 2393.. TODO:: 2394 2395 Should the ``isa`` state machine register be used to indicate if the code is 2396 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2397 2398For AMDGPU the line number program header fields have the following values (see 2399DWARF Version 5 section 6.2.4): 2400 2401``address_size`` (ubyte) 2402 Matches the address size for the ``Global`` address space defined in 2403 :ref:`amdgpu-dwarf-address-space-identifier`. 2404 2405``segment_selector_size`` (ubyte) 2406 AMDGPU does not use a segment selector so this is 0. 2407 2408``minimum_instruction_length`` (ubyte) 2409 For GFX9-GFX10 this is 4. 2410 2411``maximum_operations_per_instruction`` (ubyte) 2412 For GFX9-GFX10 this is 1. 2413 2414Source text for online-compiled programs (for example, those compiled by the 2415OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2416See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2417Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2418<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2419 2420The Clang option used to control source embedding in AMDGPU is defined in 2421:ref:`amdgpu-clang-debug-options-table`. 2422 2423 .. table:: AMDGPU Clang Debug Options 2424 :name: amdgpu-clang-debug-options-table 2425 2426 ==================== ================================================== 2427 Debug Flag Description 2428 ==================== ================================================== 2429 -g[no-]embed-source Enable/disable embedding source text in DWARF 2430 debug sections. Useful for environments where 2431 source cannot be written to disk, such as 2432 when performing online compilation. 2433 ==================== ================================================== 2434 2435For example: 2436 2437``-gembed-source`` 2438 Enable the embedded source. 2439 2440``-gno-embed-source`` 2441 Disable the embedded source. 2442 244332-Bit and 64-Bit DWARF Formats 2444------------------------------- 2445 2446See DWARF Version 5 section 7.4 and 2447:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2448 2449For AMDGPU: 2450 2451* For the ``amdgcn`` target architecture only the 64-bit process address space 2452 is supported. 2453 2454* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2455 the 32-bit DWARF format. 2456 2457Unit Headers 2458------------ 2459 2460For AMDGPU the following values apply for each of the unit headers described in 2461DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2462 2463``address_size`` (ubyte) 2464 Matches the address size for the ``Global`` address space defined in 2465 :ref:`amdgpu-dwarf-address-space-identifier`. 2466 2467.. _amdgpu-code-conventions: 2468 2469Code Conventions 2470================ 2471 2472This section provides code conventions used for each supported target triple OS 2473(see :ref:`amdgpu-target-triples`). 2474 2475AMDHSA 2476------ 2477 2478This section provides code conventions used when the target triple OS is 2479``amdhsa`` (see :ref:`amdgpu-target-triples`). 2480 2481.. _amdgpu-amdhsa-code-object-metadata: 2482 2483Code Object Metadata 2484~~~~~~~~~~~~~~~~~~~~ 2485 2486The code object metadata specifies extensible metadata associated with the code 2487objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2488encoding and semantics of this metadata depends on the code object version; see 2489:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2490:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2491:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2492 2493Code object metadata is specified in a note record (see 2494:ref:`amdgpu-note-records`) and is required when the target triple OS is 2495``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2496information necessary to support the HSA compatible runtime kernel queries. For 2497example, the segment sizes needed in a dispatch packet. In addition, a 2498high-level language runtime may require other information to be included. For 2499example, the AMD OpenCL runtime records kernel argument information. 2500 2501.. _amdgpu-amdhsa-code-object-metadata-v2: 2502 2503Code Object V2 Metadata 2504+++++++++++++++++++++++ 2505 2506.. warning:: 2507 Code object V2 is not the default code object version emitted by this version 2508 of LLVM. 2509 2510Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2511(see :ref:`amdgpu-note-records-v2`). 2512 2513The metadata is specified as a YAML formatted string (see [YAML]_ and 2514:doc:`YamlIO`). 2515 2516.. TODO:: 2517 2518 Is the string null terminated? It probably should not if YAML allows it to 2519 contain null characters, otherwise it should be. 2520 2521The metadata is represented as a single YAML document comprised of the mapping 2522defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2523referenced tables. 2524 2525For boolean values, the string values of ``false`` and ``true`` are used for 2526false and true respectively. 2527 2528Additional information can be added to the mappings. To avoid conflicts, any 2529non-AMD key names should be prefixed by "*vendor-name*.". 2530 2531 .. table:: AMDHSA Code Object V2 Metadata Map 2532 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2533 2534 ========== ============== ========= ======================================= 2535 String Key Value Type Required? Description 2536 ========== ============== ========= ======================================= 2537 "Version" sequence of Required - The first integer is the major 2538 2 integers version. Currently 1. 2539 - The second integer is the minor 2540 version. Currently 0. 2541 "Printf" sequence of Each string is encoded information 2542 strings about a printf function call. The 2543 encoded information is organized as 2544 fields separated by colon (':'): 2545 2546 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2547 2548 where: 2549 2550 ``ID`` 2551 A 32-bit integer as a unique id for 2552 each printf function call 2553 2554 ``N`` 2555 A 32-bit integer equal to the number 2556 of arguments of printf function call 2557 minus 1 2558 2559 ``S[i]`` (where i = 0, 1, ... , N-1) 2560 32-bit integers for the size in bytes 2561 of the i-th FormatString argument of 2562 the printf function call 2563 2564 FormatString 2565 The format string passed to the 2566 printf function call. 2567 "Kernels" sequence of Required Sequence of the mappings for each 2568 mapping kernel in the code object. See 2569 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2570 for the definition of the mapping. 2571 ========== ============== ========= ======================================= 2572 2573.. 2574 2575 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2576 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2577 2578 ================= ============== ========= ================================ 2579 String Key Value Type Required? Description 2580 ================= ============== ========= ================================ 2581 "Name" string Required Source name of the kernel. 2582 "SymbolName" string Required Name of the kernel 2583 descriptor ELF symbol. 2584 "Language" string Source language of the kernel. 2585 Values include: 2586 2587 - "OpenCL C" 2588 - "OpenCL C++" 2589 - "HCC" 2590 - "OpenMP" 2591 2592 "LanguageVersion" sequence of - The first integer is the major 2593 2 integers version. 2594 - The second integer is the 2595 minor version. 2596 "Attrs" mapping Mapping of kernel attributes. 2597 See 2598 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2599 for the mapping definition. 2600 "Args" sequence of Sequence of mappings of the 2601 mapping kernel arguments. See 2602 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2603 for the definition of the mapping. 2604 "CodeProps" mapping Mapping of properties related to 2605 the kernel code. See 2606 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2607 for the mapping definition. 2608 ================= ============== ========= ================================ 2609 2610.. 2611 2612 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2613 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2614 2615 =================== ============== ========= ============================== 2616 String Key Value Type Required? Description 2617 =================== ============== ========= ============================== 2618 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2619 3 integers must be >=1 and the dispatch 2620 work-group size X, Y, Z must 2621 correspond to the specified 2622 values. Defaults to 0, 0, 0. 2623 2624 Corresponds to the OpenCL 2625 ``reqd_work_group_size`` 2626 attribute. 2627 "WorkGroupSizeHint" sequence of The dispatch work-group size 2628 3 integers X, Y, Z is likely to be the 2629 specified values. 2630 2631 Corresponds to the OpenCL 2632 ``work_group_size_hint`` 2633 attribute. 2634 "VecTypeHint" string The name of a scalar or vector 2635 type. 2636 2637 Corresponds to the OpenCL 2638 ``vec_type_hint`` attribute. 2639 2640 "RuntimeHandle" string The external symbol name 2641 associated with a kernel. 2642 OpenCL runtime allocates a 2643 global buffer for the symbol 2644 and saves the kernel's address 2645 to it, which is used for 2646 device side enqueueing. Only 2647 available for device side 2648 enqueued kernels. 2649 =================== ============== ========= ============================== 2650 2651.. 2652 2653 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2654 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2655 2656 ================= ============== ========= ================================ 2657 String Key Value Type Required? Description 2658 ================= ============== ========= ================================ 2659 "Name" string Kernel argument name. 2660 "TypeName" string Kernel argument type name. 2661 "Size" integer Required Kernel argument size in bytes. 2662 "Align" integer Required Kernel argument alignment in 2663 bytes. Must be a power of two. 2664 "ValueKind" string Required Kernel argument kind that 2665 specifies how to set up the 2666 corresponding argument. 2667 Values include: 2668 2669 "ByValue" 2670 The argument is copied 2671 directly into the kernarg. 2672 2673 "GlobalBuffer" 2674 A global address space pointer 2675 to the buffer data is passed 2676 in the kernarg. 2677 2678 "DynamicSharedPointer" 2679 A group address space pointer 2680 to dynamically allocated LDS 2681 is passed in the kernarg. 2682 2683 "Sampler" 2684 A global address space 2685 pointer to a S# is passed in 2686 the kernarg. 2687 2688 "Image" 2689 A global address space 2690 pointer to a T# is passed in 2691 the kernarg. 2692 2693 "Pipe" 2694 A global address space pointer 2695 to an OpenCL pipe is passed in 2696 the kernarg. 2697 2698 "Queue" 2699 A global address space pointer 2700 to an OpenCL device enqueue 2701 queue is passed in the 2702 kernarg. 2703 2704 "HiddenGlobalOffsetX" 2705 The OpenCL grid dispatch 2706 global offset for the X 2707 dimension is passed in the 2708 kernarg. 2709 2710 "HiddenGlobalOffsetY" 2711 The OpenCL grid dispatch 2712 global offset for the Y 2713 dimension is passed in the 2714 kernarg. 2715 2716 "HiddenGlobalOffsetZ" 2717 The OpenCL grid dispatch 2718 global offset for the Z 2719 dimension is passed in the 2720 kernarg. 2721 2722 "HiddenNone" 2723 An argument that is not used 2724 by the kernel. Space needs to 2725 be left for it, but it does 2726 not need to be set up. 2727 2728 "HiddenPrintfBuffer" 2729 A global address space pointer 2730 to the runtime printf buffer 2731 is passed in kernarg. 2732 2733 "HiddenHostcallBuffer" 2734 A global address space pointer 2735 to the runtime hostcall buffer 2736 is passed in kernarg. 2737 2738 "HiddenDefaultQueue" 2739 A global address space pointer 2740 to the OpenCL device enqueue 2741 queue that should be used by 2742 the kernel by default is 2743 passed in the kernarg. 2744 2745 "HiddenCompletionAction" 2746 A global address space pointer 2747 to help link enqueued kernels into 2748 the ancestor tree for determining 2749 when the parent kernel has finished. 2750 2751 "HiddenMultiGridSyncArg" 2752 A global address space pointer for 2753 multi-grid synchronization is 2754 passed in the kernarg. 2755 2756 "ValueType" string Unused and deprecated. This should no longer 2757 be emitted, but is accepted for compatibility. 2758 2759 2760 "PointeeAlign" integer Alignment in bytes of pointee 2761 type for pointer type kernel 2762 argument. Must be a power 2763 of 2. Only present if 2764 "ValueKind" is 2765 "DynamicSharedPointer". 2766 "AddrSpaceQual" string Kernel argument address space 2767 qualifier. Only present if 2768 "ValueKind" is "GlobalBuffer" or 2769 "DynamicSharedPointer". Values 2770 are: 2771 2772 - "Private" 2773 - "Global" 2774 - "Constant" 2775 - "Local" 2776 - "Generic" 2777 - "Region" 2778 2779 .. TODO:: 2780 2781 Is GlobalBuffer only Global 2782 or Constant? Is 2783 DynamicSharedPointer always 2784 Local? Can HCC allow Generic? 2785 How can Private or Region 2786 ever happen? 2787 2788 "AccQual" string Kernel argument access 2789 qualifier. Only present if 2790 "ValueKind" is "Image" or 2791 "Pipe". Values 2792 are: 2793 2794 - "ReadOnly" 2795 - "WriteOnly" 2796 - "ReadWrite" 2797 2798 .. TODO:: 2799 2800 Does this apply to 2801 GlobalBuffer? 2802 2803 "ActualAccQual" string The actual memory accesses 2804 performed by the kernel on the 2805 kernel argument. Only present if 2806 "ValueKind" is "GlobalBuffer", 2807 "Image", or "Pipe". This may be 2808 more restrictive than indicated 2809 by "AccQual" to reflect what the 2810 kernel actual does. If not 2811 present then the runtime must 2812 assume what is implied by 2813 "AccQual" and "IsConst". Values 2814 are: 2815 2816 - "ReadOnly" 2817 - "WriteOnly" 2818 - "ReadWrite" 2819 2820 "IsConst" boolean Indicates if the kernel argument 2821 is const qualified. Only present 2822 if "ValueKind" is 2823 "GlobalBuffer". 2824 2825 "IsRestrict" boolean Indicates if the kernel argument 2826 is restrict qualified. Only 2827 present if "ValueKind" is 2828 "GlobalBuffer". 2829 2830 "IsVolatile" boolean Indicates if the kernel argument 2831 is volatile qualified. Only 2832 present if "ValueKind" is 2833 "GlobalBuffer". 2834 2835 "IsPipe" boolean Indicates if the kernel argument 2836 is pipe qualified. Only present 2837 if "ValueKind" is "Pipe". 2838 2839 .. TODO:: 2840 2841 Can GlobalBuffer be pipe 2842 qualified? 2843 2844 ================= ============== ========= ================================ 2845 2846.. 2847 2848 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2849 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2850 2851 ============================ ============== ========= ===================== 2852 String Key Value Type Required? Description 2853 ============================ ============== ========= ===================== 2854 "KernargSegmentSize" integer Required The size in bytes of 2855 the kernarg segment 2856 that holds the values 2857 of the arguments to 2858 the kernel. 2859 "GroupSegmentFixedSize" integer Required The amount of group 2860 segment memory 2861 required by a 2862 work-group in 2863 bytes. This does not 2864 include any 2865 dynamically allocated 2866 group segment memory 2867 that may be added 2868 when the kernel is 2869 dispatched. 2870 "PrivateSegmentFixedSize" integer Required The amount of fixed 2871 private address space 2872 memory required for a 2873 work-item in 2874 bytes. If the kernel 2875 uses a dynamic call 2876 stack then additional 2877 space must be added 2878 to this value for the 2879 call stack. 2880 "KernargSegmentAlign" integer Required The maximum byte 2881 alignment of 2882 arguments in the 2883 kernarg segment. Must 2884 be a power of 2. 2885 "WavefrontSize" integer Required Wavefront size. Must 2886 be a power of 2. 2887 "NumSGPRs" integer Required Number of scalar 2888 registers used by a 2889 wavefront for 2890 GFX6-GFX10. This 2891 includes the special 2892 SGPRs for VCC, Flat 2893 Scratch (GFX7-GFX10) 2894 and XNACK (for 2895 GFX8-GFX10). It does 2896 not include the 16 2897 SGPR added if a trap 2898 handler is 2899 enabled. It is not 2900 rounded up to the 2901 allocation 2902 granularity. 2903 "NumVGPRs" integer Required Number of vector 2904 registers used by 2905 each work-item for 2906 GFX6-GFX10 2907 "MaxFlatWorkGroupSize" integer Required Maximum flat 2908 work-group size 2909 supported by the 2910 kernel in work-items. 2911 Must be >=1 and 2912 consistent with 2913 ReqdWorkGroupSize if 2914 not 0, 0, 0. 2915 "NumSpilledSGPRs" integer Number of stores from 2916 a scalar register to 2917 a register allocator 2918 created spill 2919 location. 2920 "NumSpilledVGPRs" integer Number of stores from 2921 a vector register to 2922 a register allocator 2923 created spill 2924 location. 2925 ============================ ============== ========= ===================== 2926 2927.. _amdgpu-amdhsa-code-object-metadata-v3: 2928 2929Code Object V3 Metadata 2930+++++++++++++++++++++++ 2931 2932Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2933record (see :ref:`amdgpu-note-records-v3-v4`). 2934 2935The metadata is represented as Message Pack formatted binary data (see 2936[MsgPack]_). The top level is a Message Pack map that includes the 2937keys defined in table 2938:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2939tables. 2940 2941Additional information can be added to the maps. To avoid conflicts, 2942any key names should be prefixed by "*vendor-name*." where 2943``vendor-name`` can be the name of the vendor and specific vendor 2944tool that generates the information. The prefix is abbreviated to 2945simply "." when it appears within a map that has been added by the 2946same *vendor-name*. 2947 2948 .. table:: AMDHSA Code Object V3 Metadata Map 2949 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2950 2951 ================= ============== ========= ======================================= 2952 String Key Value Type Required? Description 2953 ================= ============== ========= ======================================= 2954 "amdhsa.version" sequence of Required - The first integer is the major 2955 2 integers version. Currently 1. 2956 - The second integer is the minor 2957 version. Currently 0. 2958 "amdhsa.printf" sequence of Each string is encoded information 2959 strings about a printf function call. The 2960 encoded information is organized as 2961 fields separated by colon (':'): 2962 2963 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2964 2965 where: 2966 2967 ``ID`` 2968 A 32-bit integer as a unique id for 2969 each printf function call 2970 2971 ``N`` 2972 A 32-bit integer equal to the number 2973 of arguments of printf function call 2974 minus 1 2975 2976 ``S[i]`` (where i = 0, 1, ... , N-1) 2977 32-bit integers for the size in bytes 2978 of the i-th FormatString argument of 2979 the printf function call 2980 2981 FormatString 2982 The format string passed to the 2983 printf function call. 2984 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2985 map kernel in the code object. See 2986 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2987 for the definition of the keys included 2988 in that map. 2989 ================= ============== ========= ======================================= 2990 2991.. 2992 2993 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2994 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2995 2996 =================================== ============== ========= ================================ 2997 String Key Value Type Required? Description 2998 =================================== ============== ========= ================================ 2999 ".name" string Required Source name of the kernel. 3000 ".symbol" string Required Name of the kernel 3001 descriptor ELF symbol. 3002 ".language" string Source language of the kernel. 3003 Values include: 3004 3005 - "OpenCL C" 3006 - "OpenCL C++" 3007 - "HCC" 3008 - "HIP" 3009 - "OpenMP" 3010 - "Assembler" 3011 3012 ".language_version" sequence of - The first integer is the major 3013 2 integers version. 3014 - The second integer is the 3015 minor version. 3016 ".args" sequence of Sequence of maps of the 3017 map kernel arguments. See 3018 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3019 for the definition of the keys 3020 included in that map. 3021 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3022 3 integers must be >=1 and the dispatch 3023 work-group size X, Y, Z must 3024 correspond to the specified 3025 values. Defaults to 0, 0, 0. 3026 3027 Corresponds to the OpenCL 3028 ``reqd_work_group_size`` 3029 attribute. 3030 ".workgroup_size_hint" sequence of The dispatch work-group size 3031 3 integers X, Y, Z is likely to be the 3032 specified values. 3033 3034 Corresponds to the OpenCL 3035 ``work_group_size_hint`` 3036 attribute. 3037 ".vec_type_hint" string The name of a scalar or vector 3038 type. 3039 3040 Corresponds to the OpenCL 3041 ``vec_type_hint`` attribute. 3042 3043 ".device_enqueue_symbol" string The external symbol name 3044 associated with a kernel. 3045 OpenCL runtime allocates a 3046 global buffer for the symbol 3047 and saves the kernel's address 3048 to it, which is used for 3049 device side enqueueing. Only 3050 available for device side 3051 enqueued kernels. 3052 ".kernarg_segment_size" integer Required The size in bytes of 3053 the kernarg segment 3054 that holds the values 3055 of the arguments to 3056 the kernel. 3057 ".group_segment_fixed_size" integer Required The amount of group 3058 segment memory 3059 required by a 3060 work-group in 3061 bytes. This does not 3062 include any 3063 dynamically allocated 3064 group segment memory 3065 that may be added 3066 when the kernel is 3067 dispatched. 3068 ".private_segment_fixed_size" integer Required The amount of fixed 3069 private address space 3070 memory required for a 3071 work-item in 3072 bytes. If the kernel 3073 uses a dynamic call 3074 stack then additional 3075 space must be added 3076 to this value for the 3077 call stack. 3078 ".kernarg_segment_align" integer Required The maximum byte 3079 alignment of 3080 arguments in the 3081 kernarg segment. Must 3082 be a power of 2. 3083 ".wavefront_size" integer Required Wavefront size. Must 3084 be a power of 2. 3085 ".sgpr_count" integer Required Number of scalar 3086 registers required by a 3087 wavefront for 3088 GFX6-GFX9. A register 3089 is required if it is 3090 used explicitly, or 3091 if a higher numbered 3092 register is used 3093 explicitly. This 3094 includes the special 3095 SGPRs for VCC, Flat 3096 Scratch (GFX7-GFX9) 3097 and XNACK (for 3098 GFX8-GFX9). It does 3099 not include the 16 3100 SGPR added if a trap 3101 handler is 3102 enabled. It is not 3103 rounded up to the 3104 allocation 3105 granularity. 3106 ".vgpr_count" integer Required Number of vector 3107 registers required by 3108 each work-item for 3109 GFX6-GFX9. A register 3110 is required if it is 3111 used explicitly, or 3112 if a higher numbered 3113 register is used 3114 explicitly. 3115 ".max_flat_workgroup_size" integer Required Maximum flat 3116 work-group size 3117 supported by the 3118 kernel in work-items. 3119 Must be >=1 and 3120 consistent with 3121 ReqdWorkGroupSize if 3122 not 0, 0, 0. 3123 ".sgpr_spill_count" integer Number of stores from 3124 a scalar register to 3125 a register allocator 3126 created spill 3127 location. 3128 ".vgpr_spill_count" integer Number of stores from 3129 a vector register to 3130 a register allocator 3131 created spill 3132 location. 3133 =================================== ============== ========= ================================ 3134 3135.. 3136 3137 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3138 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3139 3140 ====================== ============== ========= ================================ 3141 String Key Value Type Required? Description 3142 ====================== ============== ========= ================================ 3143 ".name" string Kernel argument name. 3144 ".type_name" string Kernel argument type name. 3145 ".size" integer Required Kernel argument size in bytes. 3146 ".offset" integer Required Kernel argument offset in 3147 bytes. The offset must be a 3148 multiple of the alignment 3149 required by the argument. 3150 ".value_kind" string Required Kernel argument kind that 3151 specifies how to set up the 3152 corresponding argument. 3153 Values include: 3154 3155 "by_value" 3156 The argument is copied 3157 directly into the kernarg. 3158 3159 "global_buffer" 3160 A global address space pointer 3161 to the buffer data is passed 3162 in the kernarg. 3163 3164 "dynamic_shared_pointer" 3165 A group address space pointer 3166 to dynamically allocated LDS 3167 is passed in the kernarg. 3168 3169 "sampler" 3170 A global address space 3171 pointer to a S# is passed in 3172 the kernarg. 3173 3174 "image" 3175 A global address space 3176 pointer to a T# is passed in 3177 the kernarg. 3178 3179 "pipe" 3180 A global address space pointer 3181 to an OpenCL pipe is passed in 3182 the kernarg. 3183 3184 "queue" 3185 A global address space pointer 3186 to an OpenCL device enqueue 3187 queue is passed in the 3188 kernarg. 3189 3190 "hidden_global_offset_x" 3191 The OpenCL grid dispatch 3192 global offset for the X 3193 dimension is passed in the 3194 kernarg. 3195 3196 "hidden_global_offset_y" 3197 The OpenCL grid dispatch 3198 global offset for the Y 3199 dimension is passed in the 3200 kernarg. 3201 3202 "hidden_global_offset_z" 3203 The OpenCL grid dispatch 3204 global offset for the Z 3205 dimension is passed in the 3206 kernarg. 3207 3208 "hidden_none" 3209 An argument that is not used 3210 by the kernel. Space needs to 3211 be left for it, but it does 3212 not need to be set up. 3213 3214 "hidden_printf_buffer" 3215 A global address space pointer 3216 to the runtime printf buffer 3217 is passed in kernarg. 3218 3219 "hidden_hostcall_buffer" 3220 A global address space pointer 3221 to the runtime hostcall buffer 3222 is passed in kernarg. 3223 3224 "hidden_default_queue" 3225 A global address space pointer 3226 to the OpenCL device enqueue 3227 queue that should be used by 3228 the kernel by default is 3229 passed in the kernarg. 3230 3231 "hidden_completion_action" 3232 A global address space pointer 3233 to help link enqueued kernels into 3234 the ancestor tree for determining 3235 when the parent kernel has finished. 3236 3237 "hidden_multigrid_sync_arg" 3238 A global address space pointer for 3239 multi-grid synchronization is 3240 passed in the kernarg. 3241 3242 ".value_type" string Unused and deprecated. This should no longer 3243 be emitted, but is accepted for compatibility. 3244 3245 ".pointee_align" integer Alignment in bytes of pointee 3246 type for pointer type kernel 3247 argument. Must be a power 3248 of 2. Only present if 3249 ".value_kind" is 3250 "dynamic_shared_pointer". 3251 ".address_space" string Kernel argument address space 3252 qualifier. Only present if 3253 ".value_kind" is "global_buffer" or 3254 "dynamic_shared_pointer". Values 3255 are: 3256 3257 - "private" 3258 - "global" 3259 - "constant" 3260 - "local" 3261 - "generic" 3262 - "region" 3263 3264 .. TODO:: 3265 3266 Is "global_buffer" only "global" 3267 or "constant"? Is 3268 "dynamic_shared_pointer" always 3269 "local"? Can HCC allow "generic"? 3270 How can "private" or "region" 3271 ever happen? 3272 3273 ".access" string Kernel argument access 3274 qualifier. Only present if 3275 ".value_kind" is "image" or 3276 "pipe". Values 3277 are: 3278 3279 - "read_only" 3280 - "write_only" 3281 - "read_write" 3282 3283 .. TODO:: 3284 3285 Does this apply to 3286 "global_buffer"? 3287 3288 ".actual_access" string The actual memory accesses 3289 performed by the kernel on the 3290 kernel argument. Only present if 3291 ".value_kind" is "global_buffer", 3292 "image", or "pipe". This may be 3293 more restrictive than indicated 3294 by ".access" to reflect what the 3295 kernel actual does. If not 3296 present then the runtime must 3297 assume what is implied by 3298 ".access" and ".is_const" . Values 3299 are: 3300 3301 - "read_only" 3302 - "write_only" 3303 - "read_write" 3304 3305 ".is_const" boolean Indicates if the kernel argument 3306 is const qualified. Only present 3307 if ".value_kind" is 3308 "global_buffer". 3309 3310 ".is_restrict" boolean Indicates if the kernel argument 3311 is restrict qualified. Only 3312 present if ".value_kind" is 3313 "global_buffer". 3314 3315 ".is_volatile" boolean Indicates if the kernel argument 3316 is volatile qualified. Only 3317 present if ".value_kind" is 3318 "global_buffer". 3319 3320 ".is_pipe" boolean Indicates if the kernel argument 3321 is pipe qualified. Only present 3322 if ".value_kind" is "pipe". 3323 3324 .. TODO:: 3325 3326 Can "global_buffer" be pipe 3327 qualified? 3328 3329 ====================== ============== ========= ================================ 3330 3331.. _amdgpu-amdhsa-code-object-metadata-v4: 3332 3333Code Object V4 Metadata 3334+++++++++++++++++++++++ 3335 3336.. warning:: 3337 Code object V4 is not the default code object version emitted by this version 3338 of LLVM. 3339 3340Code object V4 metadata is the same as 3341:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3342defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3343 3344 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3345 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3346 3347 ================= ============== ========= ======================================= 3348 String Key Value Type Required? Description 3349 ================= ============== ========= ======================================= 3350 "amdhsa.version" sequence of Required - The first integer is the major 3351 2 integers version. Currently 1. 3352 - The second integer is the minor 3353 version. Currently 1. 3354 "amdhsa.target" string Required The target name of the code using the syntax: 3355 3356 .. code:: 3357 3358 <target-triple> [ "-" <target-id> ] 3359 3360 A canonical target ID must be 3361 used. See :ref:`amdgpu-target-triples` 3362 and :ref:`amdgpu-target-id`. 3363 ================= ============== ========= ======================================= 3364 3365.. 3366 3367Kernel Dispatch 3368~~~~~~~~~~~~~~~ 3369 3370The HSA architected queuing language (AQL) defines a user space memory interface 3371that can be used to control the dispatch of kernels, in an agent independent 3372way. An agent can have zero or more AQL queues created for it using an HSA 3373compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3374are 64 bytes) can be placed. See the *HSA Platform System Architecture 3375Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3376 3377The packet processor of a kernel agent is responsible for detecting and 3378dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3379packet processor is implemented by the hardware command processor (CP), 3380asynchronous dispatch controller (ADC) and shader processor input controller 3381(SPI). 3382 3383An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3384the kernel mode driver to initialize and register the AQL queue with CP. 3385 3386To dispatch a kernel the following actions are performed. This can occur in the 3387CPU host program, or from an HSA kernel executing on a GPU. 3388 33891. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3390 executed is obtained. 33912. A pointer to the kernel descriptor (see 3392 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3393 It must be for a kernel that is contained in a code object that that was 3394 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3395 queue is associated. 33963. Space is allocated for the kernel arguments using the HSA compatible runtime 3397 allocator for a memory region with the kernarg property for the kernel agent 3398 that will execute the kernel. It must be at least 16-byte aligned. 33994. Kernel argument values are assigned to the kernel argument memory 3400 allocation. The layout is defined in the *HSA Programmer's Language 3401 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3402 kernel argument memory in the same way constant memory is accessed. (Note 3403 that the HSA specification allows an implementation to copy the kernel 3404 argument contents to another location that is accessed by the kernel.) 34055. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3406 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3407 for the packet. The packet must be set up, and the final write must use an 3408 atomic store release to set the packet kind to ensure the packet contents are 3409 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3410 notify the kernel agent that the AQL queue has been updated. These rules, and 3411 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3412 System Architecture Specification* [HSA]_. 34136. A kernel dispatch packet includes information about the actual dispatch, 3414 such as grid and work-group size, together with information from the code 3415 object about the kernel, such as segment sizes. The HSA compatible runtime 3416 queries on the kernel symbol can be used to obtain the code object values 3417 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 34187. CP executes micro-code and is responsible for detecting and setting up the 3419 GPU to execute the wavefronts of a kernel dispatch. 34208. CP ensures that when the a wavefront starts executing the kernel machine 3421 code, the scalar general purpose registers (SGPR) and vector general purpose 3422 registers (VGPR) are set up as required by the machine code. The required 3423 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3424 register state is defined in 3425 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 34269. The prolog of the kernel machine code (see 3427 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3428 before continuing executing the machine code that corresponds to the kernel. 342910. When the kernel dispatch has completed execution, CP signals the completion 3430 signal specified in the kernel dispatch packet if not 0. 3431 3432.. _amdgpu-amdhsa-memory-spaces: 3433 3434Memory Spaces 3435~~~~~~~~~~~~~ 3436 3437The memory space properties are: 3438 3439 .. table:: AMDHSA Memory Spaces 3440 :name: amdgpu-amdhsa-memory-spaces-table 3441 3442 ================= =========== ======== ======= ================== 3443 Memory Space Name HSA Segment Hardware Address NULL Value 3444 Name Name Size 3445 ================= =========== ======== ======= ================== 3446 Private private scratch 32 0x00000000 3447 Local group LDS 32 0xFFFFFFFF 3448 Global global global 64 0x0000000000000000 3449 Constant constant *same as 64 0x0000000000000000 3450 global* 3451 Generic flat flat 64 0x0000000000000000 3452 Region N/A GDS 32 *not implemented 3453 for AMDHSA* 3454 ================= =========== ======== ======= ================== 3455 3456The global and constant memory spaces both use global virtual addresses, which 3457are the same virtual address space used by the CPU. However, some virtual 3458addresses may only be accessible to the CPU, some only accessible by the GPU, 3459and some by both. 3460 3461Using the constant memory space indicates that the data will not change during 3462the execution of the kernel. This allows scalar read instructions to be 3463used. The vector and scalar L1 caches are invalidated of volatile data before 3464each kernel dispatch execution to allow constant memory to change values between 3465kernel dispatches. 3466 3467The local memory space uses the hardware Local Data Store (LDS) which is 3468automatically allocated when the hardware creates work-groups of wavefronts, and 3469freed when all the wavefronts of a work-group have terminated. The data store 3470(DS) instructions can be used to access it. 3471 3472The private memory space uses the hardware scratch memory support. If the kernel 3473uses scratch, then the hardware allocates memory that is accessed using 3474wavefront lane dword (4 byte) interleaving. The mapping used from private 3475address to physical address is: 3476 3477 ``wavefront-scratch-base + 3478 (private-address * wavefront-size * 4) + 3479 (wavefront-lane-id * 4)`` 3480 3481There are different ways that the wavefront scratch base address is determined 3482by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3483memory can be accessed in an interleaved manner using buffer instruction with 3484the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3485instructions, or by flat instructions. If each lane of a wavefront accesses the 3486same private address, the interleaving results in adjacent dwords being accessed 3487and hence requires fewer cache lines to be fetched. Multi-dword access is not 3488supported except by flat and scratch instructions in GFX9-GFX10. 3489 3490The generic address space uses the hardware flat address support available in 3491GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3492local apertures), that are outside the range of addressible global memory, to 3493map from a flat address to a private or local address. 3494 3495FLAT instructions can take a flat address and access global, private (scratch) 3496and group (LDS) memory depending in if the address is within one of the 3497aperture ranges. Flat access to scratch requires hardware aperture setup and 3498setup in the kernel prologue (see 3499:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3500hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3501:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3502 3503To convert between a segment address and a flat address the base address of the 3504apertures address can be used. For GFX7-GFX8 these are available in the 3505:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3506Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3507GFX9-GFX10 the aperture base addresses are directly available as inline constant 3508registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3509address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3510which makes it easier to convert from flat to segment or segment to flat. 3511 3512Image and Samplers 3513~~~~~~~~~~~~~~~~~~ 3514 3515Image and sample handles created by an HSA compatible runtime (see 3516:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3517object respectively. In order to support the HSA ``query_sampler`` operations 3518two extra dwords are used to store the HSA BRIG enumeration values for the 3519queries that are not trivially deducible from the S# representation. 3520 3521HSA Signals 3522~~~~~~~~~~~ 3523 3524HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3525are 64-bit addresses of a structure allocated in memory accessible from both the 3526CPU and GPU. The structure is defined by the runtime and subject to change 3527between releases. For example, see [AMD-ROCm-github]_. 3528 3529.. _amdgpu-amdhsa-hsa-aql-queue: 3530 3531HSA AQL Queue 3532~~~~~~~~~~~~~ 3533 3534The HSA AQL queue structure is defined by an HSA compatible runtime (see 3535:ref:`amdgpu-os`) and subject to change between releases. For example, see 3536[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3537certain language features such as the flat address aperture bases. It also 3538contains fields used by CP such as managing the allocation of scratch memory. 3539 3540.. _amdgpu-amdhsa-kernel-descriptor: 3541 3542Kernel Descriptor 3543~~~~~~~~~~~~~~~~~ 3544 3545A kernel descriptor consists of the information needed by CP to initiate the 3546execution of a kernel, including the entry point address of the machine code 3547that implements the kernel. 3548 3549Code Object V3 Kernel Descriptor 3550++++++++++++++++++++++++++++++++ 3551 3552CP microcode requires the Kernel descriptor to be allocated on 64-byte 3553alignment. 3554 3555The fields used by CP for code objects before V3 also match those specified in 3556:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3557 3558 .. table:: Code Object V3 Kernel Descriptor 3559 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3560 3561 ======= ======= =============================== ============================ 3562 Bits Size Field Name Description 3563 ======= ======= =============================== ============================ 3564 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3565 address space memory 3566 required for a work-group 3567 in bytes. This does not 3568 include any dynamically 3569 allocated local address 3570 space memory that may be 3571 added when the kernel is 3572 dispatched. 3573 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3574 private address space 3575 memory required for a 3576 work-item in bytes. 3577 Additional space may need to 3578 be added to this value if 3579 the call stack has 3580 non-inlined function calls. 3581 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3582 memory pointed to by the 3583 AQL dispatch packet. The 3584 kernarg memory is used to 3585 pass arguments to the 3586 kernel. 3587 3588 * If the kernarg pointer in 3589 the dispatch packet is NULL 3590 then there are no kernel 3591 arguments. 3592 * If the kernarg pointer in 3593 the dispatch packet is 3594 not NULL and this value 3595 is 0 then the kernarg 3596 memory size is 3597 unspecified. 3598 * If the kernarg pointer in 3599 the dispatch packet is 3600 not NULL and this value 3601 is not 0 then the value 3602 specifies the kernarg 3603 memory size in bytes. It 3604 is recommended to provide 3605 a value as it may be used 3606 by CP to optimize making 3607 the kernarg memory 3608 visible to the kernel 3609 code. 3610 3611 127:96 4 bytes Reserved, must be 0. 3612 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3613 negative) from base 3614 address of kernel 3615 descriptor to kernel's 3616 entry point instruction 3617 which must be 256 byte 3618 aligned. 3619 351:272 20 Reserved, must be 0. 3620 bytes 3621 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3622 Reserved, must be 0. 3623 GFX90A 3624 Compute Shader (CS) 3625 program settings used by 3626 CP to set up 3627 ``COMPUTE_PGM_RSRC3`` 3628 configuration 3629 register. See 3630 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3631 GFX10 3632 Compute Shader (CS) 3633 program settings used by 3634 CP to set up 3635 ``COMPUTE_PGM_RSRC3`` 3636 configuration 3637 register. See 3638 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3639 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3640 program settings used by 3641 CP to set up 3642 ``COMPUTE_PGM_RSRC1`` 3643 configuration 3644 register. See 3645 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3646 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3647 program settings used by 3648 CP to set up 3649 ``COMPUTE_PGM_RSRC2`` 3650 configuration 3651 register. See 3652 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3653 458:448 7 bits *See separate bits below.* Enable the setup of the 3654 SGPR user data registers 3655 (see 3656 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3657 3658 The total number of SGPR 3659 user data registers 3660 requested must not exceed 3661 16 and match value in 3662 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3663 Any requests beyond 16 3664 will be ignored. 3665 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3666 _BUFFER 3667 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3668 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3669 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3670 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3671 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT 3672 3673 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3674 _SIZE 3675 457:455 3 bits Reserved, must be 0. 3676 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3677 Reserved, must be 0. 3678 GFX10 3679 - If 0 execute in 3680 wavefront size 64 mode. 3681 - If 1 execute in 3682 native wavefront size 3683 32 mode. 3684 463:459 1 bit Reserved, must be 0. 3685 464 1 bit RESERVED_464 Deprecated, must be 0. 3686 467:465 3 bits Reserved, must be 0. 3687 468 1 bit RESERVED_468 Deprecated, must be 0. 3688 469:471 3 bits Reserved, must be 0. 3689 511:472 5 bytes Reserved, must be 0. 3690 512 **Total size 64 bytes.** 3691 ======= ==================================================================== 3692 3693.. 3694 3695 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3696 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3697 3698 ======= ======= =============================== =========================================================================== 3699 Bits Size Field Name Description 3700 ======= ======= =============================== =========================================================================== 3701 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3702 blocks used by each work-item; 3703 granularity is device 3704 specific: 3705 3706 GFX6-GFX9 3707 - vgprs_used 0..256 3708 - max(0, ceil(vgprs_used / 4) - 1) 3709 GFX90A 3710 - vgprs_used 0..512 3711 - vgprs_used = align(arch_vgprs, 4) 3712 + acc_vgprs 3713 - max(0, ceil(vgprs_used / 8) - 1) 3714 GFX10 (wavefront size 64) 3715 - max_vgpr 1..256 3716 - max(0, ceil(vgprs_used / 4) - 1) 3717 GFX10 (wavefront size 32) 3718 - max_vgpr 1..256 3719 - max(0, ceil(vgprs_used / 8) - 1) 3720 3721 Where vgprs_used is defined 3722 as the highest VGPR number 3723 explicitly referenced plus 3724 one. 3725 3726 Used by CP to set up 3727 ``COMPUTE_PGM_RSRC1.VGPRS``. 3728 3729 The 3730 :ref:`amdgpu-assembler` 3731 calculates this 3732 automatically for the 3733 selected processor from 3734 values provided to the 3735 `.amdhsa_kernel` directive 3736 by the 3737 `.amdhsa_next_free_vgpr` 3738 nested directive (see 3739 :ref:`amdhsa-kernel-directives-table`). 3740 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3741 blocks used by a wavefront; 3742 granularity is device 3743 specific: 3744 3745 GFX6-GFX8 3746 - sgprs_used 0..112 3747 - max(0, ceil(sgprs_used / 8) - 1) 3748 GFX9 3749 - sgprs_used 0..112 3750 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3751 GFX10 3752 Reserved, must be 0. 3753 (128 SGPRs always 3754 allocated.) 3755 3756 Where sgprs_used is 3757 defined as the highest 3758 SGPR number explicitly 3759 referenced plus one, plus 3760 a target specific number 3761 of additional special 3762 SGPRs for VCC, 3763 FLAT_SCRATCH (GFX7+) and 3764 XNACK_MASK (GFX8+), and 3765 any additional 3766 target specific 3767 limitations. It does not 3768 include the 16 SGPRs added 3769 if a trap handler is 3770 enabled. 3771 3772 The target specific 3773 limitations and special 3774 SGPR layout are defined in 3775 the hardware 3776 documentation, which can 3777 be found in the 3778 :ref:`amdgpu-processors` 3779 table. 3780 3781 Used by CP to set up 3782 ``COMPUTE_PGM_RSRC1.SGPRS``. 3783 3784 The 3785 :ref:`amdgpu-assembler` 3786 calculates this 3787 automatically for the 3788 selected processor from 3789 values provided to the 3790 `.amdhsa_kernel` directive 3791 by the 3792 `.amdhsa_next_free_sgpr` 3793 and `.amdhsa_reserve_*` 3794 nested directives (see 3795 :ref:`amdhsa-kernel-directives-table`). 3796 11:10 2 bits PRIORITY Must be 0. 3797 3798 Start executing wavefront 3799 at the specified priority. 3800 3801 CP is responsible for 3802 filling in 3803 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3804 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3805 with specified rounding 3806 mode for single (32 3807 bit) floating point 3808 precision floating point 3809 operations. 3810 3811 Floating point rounding 3812 mode values are defined in 3813 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3814 3815 Used by CP to set up 3816 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3817 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3818 with specified rounding 3819 denorm mode for half/double (16 3820 and 64-bit) floating point 3821 precision floating point 3822 operations. 3823 3824 Floating point rounding 3825 mode values are defined in 3826 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3827 3828 Used by CP to set up 3829 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3830 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3831 with specified denorm mode 3832 for single (32 3833 bit) floating point 3834 precision floating point 3835 operations. 3836 3837 Floating point denorm mode 3838 values are defined in 3839 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3840 3841 Used by CP to set up 3842 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3843 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3844 with specified denorm mode 3845 for half/double (16 3846 and 64-bit) floating point 3847 precision floating point 3848 operations. 3849 3850 Floating point denorm mode 3851 values are defined in 3852 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3853 3854 Used by CP to set up 3855 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3856 20 1 bit PRIV Must be 0. 3857 3858 Start executing wavefront 3859 in privilege trap handler 3860 mode. 3861 3862 CP is responsible for 3863 filling in 3864 ``COMPUTE_PGM_RSRC1.PRIV``. 3865 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3866 with DX10 clamp mode 3867 enabled. Used by the vector 3868 ALU to force DX10 style 3869 treatment of NaN's (when 3870 set, clamp NaN to zero, 3871 otherwise pass NaN 3872 through). 3873 3874 Used by CP to set up 3875 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3876 22 1 bit DEBUG_MODE Must be 0. 3877 3878 Start executing wavefront 3879 in single step mode. 3880 3881 CP is responsible for 3882 filling in 3883 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3884 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3885 with IEEE mode 3886 enabled. Floating point 3887 opcodes that support 3888 exception flag gathering 3889 will quiet and propagate 3890 signaling-NaN inputs per 3891 IEEE 754-2008. Min_dx10 and 3892 max_dx10 become IEEE 3893 754-2008 compliant due to 3894 signaling-NaN propagation 3895 and quieting. 3896 3897 Used by CP to set up 3898 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3899 24 1 bit BULKY Must be 0. 3900 3901 Only one work-group allowed 3902 to execute on a compute 3903 unit. 3904 3905 CP is responsible for 3906 filling in 3907 ``COMPUTE_PGM_RSRC1.BULKY``. 3908 25 1 bit CDBG_USER Must be 0. 3909 3910 Flag that can be used to 3911 control debugging code. 3912 3913 CP is responsible for 3914 filling in 3915 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3916 26 1 bit FP16_OVFL GFX6-GFX8 3917 Reserved, must be 0. 3918 GFX9-GFX10 3919 Wavefront starts execution 3920 with specified fp16 overflow 3921 mode. 3922 3923 - If 0, fp16 overflow generates 3924 +/-INF values. 3925 - If 1, fp16 overflow that is the 3926 result of an +/-INF input value 3927 or divide by 0 produces a +/-INF, 3928 otherwise clamps computed 3929 overflow to +/-MAX_FP16 as 3930 appropriate. 3931 3932 Used by CP to set up 3933 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3934 28:27 2 bits Reserved, must be 0. 3935 29 1 bit WGP_MODE GFX6-GFX9 3936 Reserved, must be 0. 3937 GFX10 3938 - If 0 execute work-groups in 3939 CU wavefront execution mode. 3940 - If 1 execute work-groups on 3941 in WGP wavefront execution mode. 3942 3943 See :ref:`amdgpu-amdhsa-memory-model`. 3944 3945 Used by CP to set up 3946 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3947 30 1 bit MEM_ORDERED GFX6-GFX9 3948 Reserved, must be 0. 3949 GFX10 3950 Controls the behavior of the 3951 s_waitcnt's vmcnt and vscnt 3952 counters. 3953 3954 - If 0 vmcnt reports completion 3955 of load and atomic with return 3956 out of order with sample 3957 instructions, and the vscnt 3958 reports the completion of 3959 store and atomic without 3960 return in order. 3961 - If 1 vmcnt reports completion 3962 of load, atomic with return 3963 and sample instructions in 3964 order, and the vscnt reports 3965 the completion of store and 3966 atomic without return in order. 3967 3968 Used by CP to set up 3969 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3970 31 1 bit FWD_PROGRESS GFX6-GFX9 3971 Reserved, must be 0. 3972 GFX10 3973 - If 0 execute SIMD wavefronts 3974 using oldest first policy. 3975 - If 1 execute SIMD wavefronts to 3976 ensure wavefronts will make some 3977 forward progress. 3978 3979 Used by CP to set up 3980 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3981 32 **Total size 4 bytes** 3982 ======= =================================================================================================================== 3983 3984.. 3985 3986 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3987 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3988 3989 ======= ======= =============================== =========================================================================== 3990 Bits Size Field Name Description 3991 ======= ======= =============================== =========================================================================== 3992 0 1 bit ENABLE_PRIVATE_SEGMENT Enable the setup of the 3993 private segment. 3994 3995 In addition, enable the 3996 setup of the SGPR 3997 wavefront scratch offset 3998 system register (see 3999 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4000 4001 Used by CP to set up 4002 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4003 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4004 user data registers 4005 requested. This number must 4006 match the number of user 4007 data registers enabled. 4008 4009 Used by CP to set up 4010 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4011 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4012 4013 This bit represents 4014 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4015 which is set by the CP if 4016 the runtime has installed a 4017 trap handler. 4018 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4019 system SGPR register for 4020 the work-group id in the X 4021 dimension (see 4022 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4023 4024 Used by CP to set up 4025 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4026 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4027 system SGPR register for 4028 the work-group id in the Y 4029 dimension (see 4030 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4031 4032 Used by CP to set up 4033 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4034 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4035 system SGPR register for 4036 the work-group id in the Z 4037 dimension (see 4038 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4039 4040 Used by CP to set up 4041 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4042 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4043 system SGPR register for 4044 work-group information (see 4045 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4046 4047 Used by CP to set up 4048 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4049 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4050 VGPR system registers used 4051 for the work-item ID. 4052 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4053 defines the values. 4054 4055 Used by CP to set up 4056 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4057 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4058 4059 Wavefront starts execution 4060 with address watch 4061 exceptions enabled which 4062 are generated when L1 has 4063 witnessed a thread access 4064 an *address of 4065 interest*. 4066 4067 CP is responsible for 4068 filling in the address 4069 watch bit in 4070 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4071 according to what the 4072 runtime requests. 4073 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4074 4075 Wavefront starts execution 4076 with memory violation 4077 exceptions exceptions 4078 enabled which are generated 4079 when a memory violation has 4080 occurred for this wavefront from 4081 L1 or LDS 4082 (write-to-read-only-memory, 4083 mis-aligned atomic, LDS 4084 address out of range, 4085 illegal address, etc.). 4086 4087 CP sets the memory 4088 violation bit in 4089 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4090 according to what the 4091 runtime requests. 4092 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4093 4094 CP uses the rounded value 4095 from the dispatch packet, 4096 not this value, as the 4097 dispatch may contain 4098 dynamically allocated group 4099 segment memory. CP writes 4100 directly to 4101 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4102 4103 Amount of group segment 4104 (LDS) to allocate for each 4105 work-group. Granularity is 4106 device specific: 4107 4108 GFX6 4109 roundup(lds-size / (64 * 4)) 4110 GFX7-GFX10 4111 roundup(lds-size / (128 * 4)) 4112 4113 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4114 _INVALID_OPERATION with specified exceptions 4115 enabled. 4116 4117 Used by CP to set up 4118 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4119 (set from bits 0..6). 4120 4121 IEEE 754 FP Invalid 4122 Operation 4123 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4124 _SOURCE input operands is a 4125 denormal number 4126 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4127 _DIVISION_BY_ZERO Zero 4128 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4129 _OVERFLOW 4130 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4131 _UNDERFLOW 4132 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4133 _INEXACT 4134 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4135 _ZERO (rcp_iflag_f32 instruction 4136 only) 4137 31 1 bit Reserved, must be 0. 4138 32 **Total size 4 bytes.** 4139 ======= =================================================================================================================== 4140 4141.. 4142 4143 .. table:: compute_pgm_rsrc3 for GFX90A 4144 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4145 4146 ======= ======= =============================== =========================================================================== 4147 Bits Size Field Name Description 4148 ======= ======= =============================== =========================================================================== 4149 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4150 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4151 63 - accum-offset = 256. 4152 6:15 10 Reserved, must be 0. 4153 bits 4154 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4155 launched in the same CU. 4156 - If 1 the waves of a work-group can be 4157 launched in different CUs. The waves 4158 cannot use S_BARRIER or LDS. 4159 17:31 15 Reserved, must be 0. 4160 bits 4161 32 **Total size 4 bytes.** 4162 ======= =================================================================================================================== 4163 4164.. 4165 4166 .. table:: compute_pgm_rsrc3 for GFX10 4167 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4168 4169 ======= ======= =============================== =========================================================================== 4170 Bits Size Field Name Description 4171 ======= ======= =============================== =========================================================================== 4172 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4173 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4174 31:4 28 Reserved, must be 0. 4175 bits 4176 32 **Total size 4 bytes.** 4177 ======= =================================================================================================================== 4178 4179.. 4180 4181 .. table:: Floating Point Rounding Mode Enumeration Values 4182 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4183 4184 ====================================== ===== ============================== 4185 Enumeration Name Value Description 4186 ====================================== ===== ============================== 4187 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4188 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4189 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4190 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4191 ====================================== ===== ============================== 4192 4193.. 4194 4195 .. table:: Floating Point Denorm Mode Enumeration Values 4196 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4197 4198 ====================================== ===== ============================== 4199 Enumeration Name Value Description 4200 ====================================== ===== ============================== 4201 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4202 Denorms 4203 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4204 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4205 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4206 ====================================== ===== ============================== 4207 4208.. 4209 4210 .. table:: System VGPR Work-Item ID Enumeration Values 4211 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4212 4213 ======================================== ===== ============================ 4214 Enumeration Name Value Description 4215 ======================================== ===== ============================ 4216 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4217 ID. 4218 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4219 dimensions ID. 4220 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4221 dimensions ID. 4222 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4223 ======================================== ===== ============================ 4224 4225.. _amdgpu-amdhsa-initial-kernel-execution-state: 4226 4227Initial Kernel Execution State 4228~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4229 4230This section defines the register state that will be set up by the packet 4231processor prior to the start of execution of every wavefront. This is limited by 4232the constraints of the hardware controllers of CP/ADC/SPI. 4233 4234The order of the SGPR registers is defined, but the compiler can specify which 4235ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4236fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4237for enabled registers are dense starting at SGPR0: the first enabled register is 4238SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4239an SGPR number. 4240 4241The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4242all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4243using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4244actually initialized. These are then immediately followed by the System SGPRs 4245that are set up by ADC/SPI and can have different values for each wavefront of 4246the grid dispatch. 4247 4248SGPR register initial state is defined in 4249:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4250 4251 .. table:: SGPR Register Set Up Order 4252 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4253 4254 ========== ========================== ====== ============================== 4255 SGPR Order Name Number Description 4256 (kernel descriptor enable of 4257 field) SGPRs 4258 ========== ========================== ====== ============================== 4259 First Private Segment Buffer 4 See 4260 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4261 _segment_buffer) 4262 then Dispatch Ptr 2 64-bit address of AQL dispatch 4263 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4264 actually executing. 4265 then Queue Ptr 2 64-bit address of amd_queue_t 4266 (enable_sgpr_queue_ptr) object for AQL queue on which 4267 the dispatch packet was 4268 queued. 4269 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4270 (enable_sgpr_kernarg segment. This is directly 4271 _segment_ptr) copied from the 4272 kernarg_address in the kernel 4273 dispatch packet. 4274 4275 Having CP load it once avoids 4276 loading it at the beginning of 4277 every wavefront. 4278 then Dispatch Id 2 64-bit Dispatch ID of the 4279 (enable_sgpr_dispatch_id) dispatch packet being 4280 executed. 4281 then Flat Scratch Init 2 See 4282 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4283 then Private Segment Size 1 The 32-bit byte size of a 4284 (enable_sgpr_private single 4285 work-item's 4286 scratch_segment_size) memory 4287 allocation. This is the 4288 value from the kernel 4289 dispatch packet Private 4290 Segment Byte Size rounded up 4291 by CP to a multiple of 4292 DWORD. 4293 4294 Having CP load it once avoids 4295 loading it at the beginning of 4296 every wavefront. 4297 4298 This is not used for 4299 GFX7-GFX8 since it is the same 4300 value as the second SGPR of 4301 Flat Scratch Init. However, it 4302 may be needed for GFX9-GFX10 which 4303 changes the meaning of the 4304 Flat Scratch Init value. 4305 then Grid Work-Group Count X 1 32-bit count of the number of 4306 (enable_sgpr_grid work-groups in the X dimension 4307 _workgroup_count_X) for the grid being 4308 executed. Computed from the 4309 fields in the kernel dispatch 4310 packet as ((grid_size.x + 4311 workgroup_size.x - 1) / 4312 workgroup_size.x). 4313 then Grid Work-Group Count Y 1 32-bit count of the number of 4314 (enable_sgpr_grid work-groups in the Y dimension 4315 _workgroup_count_Y && for the grid being 4316 less than 16 previous executed. Computed from the 4317 SGPRs) fields in the kernel dispatch 4318 packet as ((grid_size.y + 4319 workgroup_size.y - 1) / 4320 workgroupSize.y). 4321 4322 Only initialized if <16 4323 previous SGPRs initialized. 4324 then Grid Work-Group Count Z 1 32-bit count of the number of 4325 (enable_sgpr_grid work-groups in the Z dimension 4326 _workgroup_count_Z && for the grid being 4327 less than 16 previous executed. Computed from the 4328 SGPRs) fields in the kernel dispatch 4329 packet as ((grid_size.z + 4330 workgroup_size.z - 1) / 4331 workgroupSize.z). 4332 4333 Only initialized if <16 4334 previous SGPRs initialized. 4335 then Work-Group Id X 1 32-bit work-group id in X 4336 (enable_sgpr_workgroup_id dimension of grid for 4337 _X) wavefront. 4338 then Work-Group Id Y 1 32-bit work-group id in Y 4339 (enable_sgpr_workgroup_id dimension of grid for 4340 _Y) wavefront. 4341 then Work-Group Id Z 1 32-bit work-group id in Z 4342 (enable_sgpr_workgroup_id dimension of grid for 4343 _Z) wavefront. 4344 then Work-Group Info 1 {first_wavefront, 14'b0000, 4345 (enable_sgpr_workgroup ordered_append_term[10:0], 4346 _info) threadgroup_size_in_wavefronts[5:0]} 4347 then Scratch Wavefront Offset 1 See 4348 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4349 _segment_wavefront_offset) and 4350 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4351 ========== ========================== ====== ============================== 4352 4353The order of the VGPR registers is defined, but the compiler can specify which 4354ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4355fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4356for enabled registers are dense starting at VGPR0: the first enabled register is 4357VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4358VGPR number. 4359 4360There are different methods used for the VGPR initial state: 4361 4362* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4363 specifies otherwise, a separate VGPR register is used per work-item ID. The 4364 VGPR register initial state for this method is defined in 4365 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4366* If *Target Properties* column of :ref:`amdgpu-processor-table` 4367 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4368 for all work-item IDs. The register layout for this method is defined in 4369 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4370 4371 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4372 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4373 4374 ========== ========================== ====== ============================== 4375 VGPR Order Name Number Description 4376 (kernel descriptor enable of 4377 field) VGPRs 4378 ========== ========================== ====== ============================== 4379 First Work-Item Id X 1 32-bit work-item id in X 4380 (Always initialized) dimension of work-group for 4381 wavefront lane. 4382 then Work-Item Id Y 1 32-bit work-item id in Y 4383 (enable_vgpr_workitem_id dimension of work-group for 4384 > 0) wavefront lane. 4385 then Work-Item Id Z 1 32-bit work-item id in Z 4386 (enable_vgpr_workitem_id dimension of work-group for 4387 > 1) wavefront lane. 4388 ========== ========================== ====== ============================== 4389 4390.. 4391 4392 .. table:: Register Layout for Packed Work-Item ID Method 4393 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4394 4395 ======= ======= ================ ========================================= 4396 Bits Size Field Name Description 4397 ======= ======= ================ ========================================= 4398 0:9 10 bits Work-Item Id X Work-item id in X 4399 dimension of work-group for 4400 wavefront lane. 4401 4402 Always initialized. 4403 4404 10:19 10 bits Work-Item Id Y Work-item id in Y 4405 dimension of work-group for 4406 wavefront lane. 4407 4408 Initialized if enable_vgpr_workitem_id > 4409 0, otherwise set to 0. 4410 20:29 10 bits Work-Item Id Z Work-item id in Z 4411 dimension of work-group for 4412 wavefront lane. 4413 4414 Initialized if enable_vgpr_workitem_id > 4415 1, otherwise set to 0. 4416 30:31 2 bits Reserved, set to 0. 4417 ======= ======= ================ ========================================= 4418 4419The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4420 44211. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4422 registers. 44232. Work-group Id registers X, Y, Z are set by ADC which supports any 4424 combination including none. 44253. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4426 its value cannot be included with the flat scratch init value which is per 4427 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 44284. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4429 or (X, Y, Z). 44305. Flat Scratch register pair initialization is described in 4431 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4432 4433The global segment can be accessed either using buffer instructions (GFX6 which 4434has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4435instructions (GFX9-GFX10). 4436 4437If buffer operations are used, then the compiler can generate a V# with the 4438following properties: 4439 4440* base address of 0 4441* no swizzle 4442* ATC: 1 if IOMMU present (such as APU) 4443* ptr64: 1 4444* MTYPE set to support memory coherence that matches the runtime (such as CC for 4445 APU and NC for dGPU). 4446 4447.. _amdgpu-amdhsa-kernel-prolog: 4448 4449Kernel Prolog 4450~~~~~~~~~~~~~ 4451 4452The compiler performs initialization in the kernel prologue depending on the 4453target and information about things like stack usage in the kernel and called 4454functions. Some of this initialization requires the compiler to request certain 4455User and System SGPRs be present in the 4456:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4457:ref:`amdgpu-amdhsa-kernel-descriptor`. 4458 4459.. _amdgpu-amdhsa-kernel-prolog-cfi: 4460 4461CFI 4462+++ 4463 44641. The CFI return address is undefined. 4465 44662. The CFI CFA is defined using an expression which evaluates to a location 4467 description that comprises one memory location description for the 4468 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4469 4470.. _amdgpu-amdhsa-kernel-prolog-m0: 4471 4472M0 4473++ 4474 4475GFX6-GFX8 4476 The M0 register must be initialized with a value at least the total LDS size 4477 if the kernel may access LDS via DS or flat operations. Total LDS size is 4478 available in dispatch packet. For M0, it is also possible to use maximum 4479 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4480 GFX7-GFX8). 4481GFX9-GFX10 4482 The M0 register is not used for range checking LDS accesses and so does not 4483 need to be initialized in the prolog. 4484 4485.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4486 4487Stack Pointer 4488+++++++++++++ 4489 4490If the kernel has function calls it must set up the ABI stack pointer described 4491in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4492SGPR32 to the unswizzled scratch offset of the address past the last local 4493allocation. 4494 4495.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4496 4497Frame Pointer 4498+++++++++++++ 4499 4500If the kernel needs a frame pointer for the reasons defined in 4501``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4502kernel prolog. If a frame pointer is not required then all uses of the frame 4503pointer are replaced with immediate ``0`` offsets. 4504 4505.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4506 4507Flat Scratch 4508++++++++++++ 4509 4510There are different methods used for initializing flat scratch: 4511 4512* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4513 specifies *Does not support generic address space*: 4514 4515 Flat scratch is not supported and there is no flat scratch register pair. 4516 4517* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4518 specifies *Offset flat scratch*: 4519 4520 If the kernel or any function it calls may use flat operations to access 4521 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4522 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4523 Scratch Wavefront Offset SGPR registers (see 4524 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4525 4526 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4527 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4528 being managed by SPI for the queue executing the kernel dispatch. This is 4529 the same value used in the Scratch Segment Buffer V# base address. 4530 4531 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4532 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4533 4534 The prolog must add the value of Scratch Wavefront Offset to get the 4535 wavefront's byte scratch backing memory offset from 4536 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4537 4538 The Scratch Wavefront Offset must also be used as an offset with Private 4539 segment address when using the Scratch Segment Buffer. 4540 4541 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4542 shifted by 8 before moving into FLAT_SCRATCH_HI. 4543 4544 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4545 SGPRn is the highest numbered SGPR allocated to the wavefront). 4546 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4547 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4548 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4549 aperture. 4550 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4551 work-items scratch memory usage. 4552 4553 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4554 checks that the value in the kernel dispatch packet Private Segment Byte 4555 Size is not larger and requests the runtime to increase the queue's scratch 4556 size if necessary. 4557 4558 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4559 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4560 loading it at the beginning of every wavefront. 4561 4562 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4563 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4564 in flat memory instructions. 4565 4566* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4567 specifies *Absolute flat scratch*: 4568 4569 If the kernel or any function it calls may use flat operations to access 4570 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4571 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4572 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4573 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4574 4575 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4576 memory being managed by SPI for the queue executing the kernel dispatch. 4577 4578 CP obtains this from the runtime. 4579 4580 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4581 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4582 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4583 memory instructions. 4584 4585 The Scratch Wavefront Offset must also be used as an offset with Private 4586 segment address when using the Scratch Segment Buffer (see 4587 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4588 4589.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4590 4591Private Segment Buffer 4592++++++++++++++++++++++ 4593 4594Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4595that are used as a V# to access scratch. CP uses the value provided by the 4596runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4597access the private memory space using a segment address. See 4598:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4599 4600The scratch V# is a four-aligned SGPR and always selected for the kernel as 4601follows: 4602 4603 - If it is known during instruction selection that there is stack usage, 4604 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4605 optimizations are disabled (``-O0``), if stack objects already exist (for 4606 locals, etc.), or if there are any function calls. 4607 4608 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4609 are reserved for the tentative scratch V#. These will be used if it is 4610 determined that spilling is needed. 4611 4612 - If no use is made of the tentative scratch V#, then it is unreserved, 4613 and the register count is determined ignoring it. 4614 - If use is made of the tentative scratch V#, then its register numbers 4615 are shifted to the first four-aligned SGPR index after the highest one 4616 allocated by the register allocator, and all uses are updated. The 4617 register count includes them in the shifted location. 4618 - In either case, if the processor has the SGPR allocation bug, the 4619 tentative allocation is not shifted or unreserved in order to ensure 4620 the register count is higher to workaround the bug. 4621 4622 .. note:: 4623 4624 This approach of using a tentative scratch V# and shifting the register 4625 numbers if used avoids having to perform register allocation a second 4626 time if the tentative V# is eliminated. This is more efficient and 4627 avoids the problem that the second register allocation may perform 4628 spilling which will fail as there is no longer a scratch V#. 4629 4630When the kernel prolog code is being emitted it is known whether the scratch V# 4631described above is actually used. If it is, the prolog code must set it up by 4632copying the Private Segment Buffer to the scratch V# registers and then adding 4633the Private Segment Wavefront Offset to the queue base address in the V#. The 4634result is a V# with a base address pointing to the beginning of the wavefront 4635scratch backing memory. 4636 4637The Private Segment Buffer is always requested, but the Private Segment 4638Wavefront Offset is only requested if it is used (see 4639:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4640 4641.. _amdgpu-amdhsa-memory-model: 4642 4643Memory Model 4644~~~~~~~~~~~~ 4645 4646This section describes the mapping of the LLVM memory model onto AMDGPU machine 4647code (see :ref:`memmodel`). 4648 4649The AMDGPU backend supports the memory synchronization scopes specified in 4650:ref:`amdgpu-memory-scopes`. 4651 4652The code sequences used to implement the memory model specify the order of 4653instructions that a single thread must execute. The ``s_waitcnt`` and cache 4654management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4655to other memory instructions executed by the same thread. This allows them to be 4656moved earlier or later which can allow them to be combined with other instances 4657of the same instruction, or hoisted/sunk out of loops to improve performance. 4658Only the instructions related to the memory model are given; additional 4659``s_waitcnt`` instructions are required to ensure registers are defined before 4660being used. These may be able to be combined with the memory model ``s_waitcnt`` 4661instructions as described above. 4662 4663The AMDGPU backend supports the following memory models: 4664 4665 HSA Memory Model [HSA]_ 4666 The HSA memory model uses a single happens-before relation for all address 4667 spaces (see :ref:`amdgpu-address-spaces`). 4668 OpenCL Memory Model [OpenCL]_ 4669 The OpenCL memory model which has separate happens-before relations for the 4670 global and local address spaces. Only a fence specifying both global and 4671 local address space, and seq_cst instructions join the relationships. Since 4672 the LLVM ``memfence`` instruction does not allow an address space to be 4673 specified the OpenCL fence has to conservatively assume both local and 4674 global address space was specified. However, optimizations can often be 4675 done to eliminate the additional ``s_waitcnt`` instructions when there are 4676 no intervening memory instructions which access the corresponding address 4677 space. The code sequences in the table indicate what can be omitted for the 4678 OpenCL memory. The target triple environment is used to determine if the 4679 source language is OpenCL (see :ref:`amdgpu-opencl`). 4680 4681``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4682operations. 4683 4684``buffer/global/flat_load/store/atomic`` instructions to global memory are 4685termed vector memory operations. 4686 4687Private address space uses ``buffer_load/store`` using the scratch V# 4688(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4689is accessing the memory, atomic memory orderings are not meaningful, and all 4690accesses are treated as non-atomic. 4691 4692Constant address space uses ``buffer/global_load`` instructions (or equivalent 4693scalar memory instructions). Since the constant address space contents do not 4694change during the execution of a kernel dispatch it is not legal to perform 4695stores, and atomic memory orderings are not meaningful, and all accesses are 4696treated as non-atomic. 4697 4698A memory synchronization scope wider than work-group is not meaningful for the 4699group (LDS) address space and is treated as work-group. 4700 4701The memory model does not support the region address space which is treated as 4702non-atomic. 4703 4704Acquire memory ordering is not meaningful on store atomic instructions and is 4705treated as non-atomic. 4706 4707Release memory ordering is not meaningful on load atomic instructions and is 4708treated a non-atomic. 4709 4710Acquire-release memory ordering is not meaningful on load or store atomic 4711instructions and is treated as acquire and release respectively. 4712 4713The memory order also adds the single thread optimization constraints defined in 4714table 4715:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4716 4717 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4718 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4719 4720 ============ ============================================================== 4721 LLVM Memory Optimization Constraints 4722 Ordering 4723 ============ ============================================================== 4724 unordered *none* 4725 monotonic *none* 4726 acquire - If a load atomic/atomicrmw then no following load/load 4727 atomic/store/store atomic/atomicrmw/fence instruction can be 4728 moved before the acquire. 4729 - If a fence then same as load atomic, plus no preceding 4730 associated fence-paired-atomic can be moved after the fence. 4731 release - If a store atomic/atomicrmw then no preceding load/load 4732 atomic/store/store atomic/atomicrmw/fence instruction can be 4733 moved after the release. 4734 - If a fence then same as store atomic, plus no following 4735 associated fence-paired-atomic can be moved before the 4736 fence. 4737 acq_rel Same constraints as both acquire and release. 4738 seq_cst - If a load atomic then same constraints as acquire, plus no 4739 preceding sequentially consistent load atomic/store 4740 atomic/atomicrmw/fence instruction can be moved after the 4741 seq_cst. 4742 - If a store atomic then the same constraints as release, plus 4743 no following sequentially consistent load atomic/store 4744 atomic/atomicrmw/fence instruction can be moved before the 4745 seq_cst. 4746 - If an atomicrmw/fence then same constraints as acq_rel. 4747 ============ ============================================================== 4748 4749The code sequences used to implement the memory model are defined in the 4750following sections: 4751 4752* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4753* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4754* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4755 4756.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4757 4758Memory Model GFX6-GFX9 4759++++++++++++++++++++++ 4760 4761For GFX6-GFX9: 4762 4763* Each agent has multiple shader arrays (SA). 4764* Each SA has multiple compute units (CU). 4765* Each CU has multiple SIMDs that execute wavefronts. 4766* The wavefronts for a single work-group are executed in the same CU but may be 4767 executed by different SIMDs. 4768* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4769 executing on it. 4770* All LDS operations of a CU are performed as wavefront wide operations in a 4771 global order and involve no caching. Completion is reported to a wavefront in 4772 execution order. 4773* The LDS memory has multiple request queues shared by the SIMDs of a 4774 CU. Therefore, the LDS operations performed by different wavefronts of a 4775 work-group can be reordered relative to each other, which can result in 4776 reordering the visibility of vector memory operations with respect to LDS 4777 operations of other wavefronts in the same work-group. A ``s_waitcnt 4778 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4779 vector memory operations between wavefronts of a work-group, but not between 4780 operations performed by the same wavefront. 4781* The vector memory operations are performed as wavefront wide operations and 4782 completion is reported to a wavefront in execution order. The exception is 4783 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4784 vector memory order if they access LDS memory, and out of LDS operation order 4785 if they access global memory. 4786* The vector memory operations access a single vector L1 cache shared by all 4787 SIMDs a CU. Therefore, no special action is required for coherence between the 4788 lanes of a single wavefront, or for coherence between wavefronts in the same 4789 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4790 wavefronts executing in different work-groups as they may be executing on 4791 different CUs. 4792* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4793 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4794 scalar operations are used in a restricted way so do not impact the memory 4795 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4796* The vector and scalar memory operations use an L2 cache shared by all CUs on 4797 the same agent. 4798* The L2 cache has independent channels to service disjoint ranges of virtual 4799 addresses. 4800* Each CU has a separate request queue per channel. Therefore, the vector and 4801 scalar memory operations performed by wavefronts executing in different 4802 work-groups (which may be executing on different CUs) of an agent can be 4803 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4804 ensure synchronization between vector memory operations of different CUs. It 4805 ensures a previous vector memory operation has completed before executing a 4806 subsequent vector memory or LDS operation and so can be used to meet the 4807 requirements of acquire and release. 4808* The L2 cache can be kept coherent with other agents on some targets, or ranges 4809 of virtual addresses can be set up to bypass it to ensure system coherence. 4810 4811Scalar memory operations are only used to access memory that is proven to not 4812change during the execution of the kernel dispatch. This includes constant 4813address space and global address space for program scope ``const`` variables. 4814Therefore, the kernel machine code does not have to maintain the scalar cache to 4815ensure it is coherent with the vector caches. The scalar and vector caches are 4816invalidated between kernel dispatches by CP since constant address space data 4817may change between kernel dispatch executions. See 4818:ref:`amdgpu-amdhsa-memory-spaces`. 4819 4820The one exception is if scalar writes are used to spill SGPR registers. In this 4821case the AMDGPU backend ensures the memory location used to spill is never 4822accessed by vector memory operations at the same time. If scalar writes are used 4823then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4824return since the locations may be used for vector memory instructions by a 4825future wavefront that uses the same scratch area, or a function call that 4826creates a frame at the same address, respectively. There is no need for a 4827``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4828 4829For kernarg backing memory: 4830 4831* CP invalidates the L1 cache at the start of each kernel dispatch. 4832* On dGPU the kernarg backing memory is allocated in host memory accessed as 4833 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4834 causes it to be treated as non-volatile and so is not invalidated by 4835 ``*_vol``. 4836* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4837 and so the L2 cache will be coherent with the CPU and other agents. 4838 4839Scratch backing memory (which is used for the private address space) is accessed 4840with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4841only accessed by a single thread, and is always write-before-read, there is 4842never a need to invalidate these entries from the L1 cache. Hence all cache 4843invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4844 4845The code sequences used to implement the memory model for GFX6-GFX9 are defined 4846in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4847 4848 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4849 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4850 4851 ============ ============ ============== ========== ================================ 4852 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4853 Ordering Sync Scope Address GFX6-GFX9 4854 Space 4855 ============ ============ ============== ========== ================================ 4856 **Non-Atomic** 4857 ------------------------------------------------------------------------------------ 4858 load *none* *none* - global - !volatile & !nontemporal 4859 - generic 4860 - private 1. buffer/global/flat_load 4861 - constant 4862 - !volatile & nontemporal 4863 4864 1. buffer/global/flat_load 4865 glc=1 slc=1 4866 4867 - volatile 4868 4869 1. buffer/global/flat_load 4870 glc=1 4871 2. s_waitcnt vmcnt(0) 4872 4873 - Must happen before 4874 any following volatile 4875 global/generic 4876 load/store. 4877 - Ensures that 4878 volatile 4879 operations to 4880 different 4881 addresses will not 4882 be reordered by 4883 hardware. 4884 4885 load *none* *none* - local 1. ds_load 4886 store *none* *none* - global - !volatile & !nontemporal 4887 - generic 4888 - private 1. buffer/global/flat_store 4889 - constant 4890 - !volatile & nontemporal 4891 4892 1. buffer/global/flat_store 4893 glc=1 slc=1 4894 4895 - volatile 4896 4897 1. buffer/global/flat_store 4898 2. s_waitcnt vmcnt(0) 4899 4900 - Must happen before 4901 any following volatile 4902 global/generic 4903 load/store. 4904 - Ensures that 4905 volatile 4906 operations to 4907 different 4908 addresses will not 4909 be reordered by 4910 hardware. 4911 4912 store *none* *none* - local 1. ds_store 4913 **Unordered Atomic** 4914 ------------------------------------------------------------------------------------ 4915 load atomic unordered *any* *any* *Same as non-atomic*. 4916 store atomic unordered *any* *any* *Same as non-atomic*. 4917 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4918 **Monotonic Atomic** 4919 ------------------------------------------------------------------------------------ 4920 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4921 - wavefront - local 4922 - workgroup - generic 4923 load atomic monotonic - agent - global 1. buffer/global/flat_load 4924 - system - generic glc=1 4925 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4926 - wavefront - generic 4927 - workgroup 4928 - agent 4929 - system 4930 store atomic monotonic - singlethread - local 1. ds_store 4931 - wavefront 4932 - workgroup 4933 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4934 - wavefront - generic 4935 - workgroup 4936 - agent 4937 - system 4938 atomicrmw monotonic - singlethread - local 1. ds_atomic 4939 - wavefront 4940 - workgroup 4941 **Acquire Atomic** 4942 ------------------------------------------------------------------------------------ 4943 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4944 - wavefront - local 4945 - generic 4946 load atomic acquire - workgroup - global 1. buffer/global_load 4947 load atomic acquire - workgroup - local 1. ds/flat_load 4948 - generic 2. s_waitcnt lgkmcnt(0) 4949 4950 - If OpenCL, omit. 4951 - Must happen before 4952 any following 4953 global/generic 4954 load/load 4955 atomic/store/store 4956 atomic/atomicrmw. 4957 - Ensures any 4958 following global 4959 data read is no 4960 older than a local load 4961 atomic value being 4962 acquired. 4963 4964 load atomic acquire - agent - global 1. buffer/global_load 4965 - system glc=1 4966 2. s_waitcnt vmcnt(0) 4967 4968 - Must happen before 4969 following 4970 buffer_wbinvl1_vol. 4971 - Ensures the load 4972 has completed 4973 before invalidating 4974 the cache. 4975 4976 3. buffer_wbinvl1_vol 4977 4978 - Must happen before 4979 any following 4980 global/generic 4981 load/load 4982 atomic/atomicrmw. 4983 - Ensures that 4984 following 4985 loads will not see 4986 stale global data. 4987 4988 load atomic acquire - agent - generic 1. flat_load glc=1 4989 - system 2. s_waitcnt vmcnt(0) & 4990 lgkmcnt(0) 4991 4992 - If OpenCL omit 4993 lgkmcnt(0). 4994 - Must happen before 4995 following 4996 buffer_wbinvl1_vol. 4997 - Ensures the flat_load 4998 has completed 4999 before invalidating 5000 the cache. 5001 5002 3. buffer_wbinvl1_vol 5003 5004 - Must happen before 5005 any following 5006 global/generic 5007 load/load 5008 atomic/atomicrmw. 5009 - Ensures that 5010 following loads 5011 will not see stale 5012 global data. 5013 5014 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5015 - wavefront - local 5016 - generic 5017 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5018 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5019 - generic 2. s_waitcnt lgkmcnt(0) 5020 5021 - If OpenCL, omit. 5022 - Must happen before 5023 any following 5024 global/generic 5025 load/load 5026 atomic/store/store 5027 atomic/atomicrmw. 5028 - Ensures any 5029 following global 5030 data read is no 5031 older than a local 5032 atomicrmw value 5033 being acquired. 5034 5035 atomicrmw acquire - agent - global 1. buffer/global_atomic 5036 - system 2. s_waitcnt vmcnt(0) 5037 5038 - Must happen before 5039 following 5040 buffer_wbinvl1_vol. 5041 - Ensures the 5042 atomicrmw has 5043 completed before 5044 invalidating the 5045 cache. 5046 5047 3. buffer_wbinvl1_vol 5048 5049 - Must happen before 5050 any following 5051 global/generic 5052 load/load 5053 atomic/atomicrmw. 5054 - Ensures that 5055 following loads 5056 will not see stale 5057 global data. 5058 5059 atomicrmw acquire - agent - generic 1. flat_atomic 5060 - system 2. s_waitcnt vmcnt(0) & 5061 lgkmcnt(0) 5062 5063 - If OpenCL, omit 5064 lgkmcnt(0). 5065 - Must happen before 5066 following 5067 buffer_wbinvl1_vol. 5068 - Ensures the 5069 atomicrmw has 5070 completed before 5071 invalidating the 5072 cache. 5073 5074 3. buffer_wbinvl1_vol 5075 5076 - Must happen before 5077 any following 5078 global/generic 5079 load/load 5080 atomic/atomicrmw. 5081 - Ensures that 5082 following loads 5083 will not see stale 5084 global data. 5085 5086 fence acquire - singlethread *none* *none* 5087 - wavefront 5088 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5089 5090 - If OpenCL and 5091 address space is 5092 not generic, omit. 5093 - However, since LLVM 5094 currently has no 5095 address space on 5096 the fence need to 5097 conservatively 5098 always generate. If 5099 fence had an 5100 address space then 5101 set to address 5102 space of OpenCL 5103 fence flag, or to 5104 generic if both 5105 local and global 5106 flags are 5107 specified. 5108 - Must happen after 5109 any preceding 5110 local/generic load 5111 atomic/atomicrmw 5112 with an equal or 5113 wider sync scope 5114 and memory ordering 5115 stronger than 5116 unordered (this is 5117 termed the 5118 fence-paired-atomic). 5119 - Must happen before 5120 any following 5121 global/generic 5122 load/load 5123 atomic/store/store 5124 atomic/atomicrmw. 5125 - Ensures any 5126 following global 5127 data read is no 5128 older than the 5129 value read by the 5130 fence-paired-atomic. 5131 5132 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5133 - system vmcnt(0) 5134 5135 - If OpenCL and 5136 address space is 5137 not generic, omit 5138 lgkmcnt(0). 5139 - However, since LLVM 5140 currently has no 5141 address space on 5142 the fence need to 5143 conservatively 5144 always generate 5145 (see comment for 5146 previous fence). 5147 - Could be split into 5148 separate s_waitcnt 5149 vmcnt(0) and 5150 s_waitcnt 5151 lgkmcnt(0) to allow 5152 them to be 5153 independently moved 5154 according to the 5155 following rules. 5156 - s_waitcnt vmcnt(0) 5157 must happen after 5158 any preceding 5159 global/generic load 5160 atomic/atomicrmw 5161 with an equal or 5162 wider sync scope 5163 and memory ordering 5164 stronger than 5165 unordered (this is 5166 termed the 5167 fence-paired-atomic). 5168 - s_waitcnt lgkmcnt(0) 5169 must happen after 5170 any preceding 5171 local/generic load 5172 atomic/atomicrmw 5173 with an equal or 5174 wider sync scope 5175 and memory ordering 5176 stronger than 5177 unordered (this is 5178 termed the 5179 fence-paired-atomic). 5180 - Must happen before 5181 the following 5182 buffer_wbinvl1_vol. 5183 - Ensures that the 5184 fence-paired atomic 5185 has completed 5186 before invalidating 5187 the 5188 cache. Therefore 5189 any following 5190 locations read must 5191 be no older than 5192 the value read by 5193 the 5194 fence-paired-atomic. 5195 5196 2. buffer_wbinvl1_vol 5197 5198 - Must happen before any 5199 following global/generic 5200 load/load 5201 atomic/store/store 5202 atomic/atomicrmw. 5203 - Ensures that 5204 following loads 5205 will not see stale 5206 global data. 5207 5208 **Release Atomic** 5209 ------------------------------------------------------------------------------------ 5210 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5211 - wavefront - local 5212 - generic 5213 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5214 - generic 5215 - If OpenCL, omit. 5216 - Must happen after 5217 any preceding 5218 local/generic 5219 load/store/load 5220 atomic/store 5221 atomic/atomicrmw. 5222 - Must happen before 5223 the following 5224 store. 5225 - Ensures that all 5226 memory operations 5227 to local have 5228 completed before 5229 performing the 5230 store that is being 5231 released. 5232 5233 2. buffer/global/flat_store 5234 store atomic release - workgroup - local 1. ds_store 5235 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5236 - system - generic vmcnt(0) 5237 5238 - If OpenCL and 5239 address space is 5240 not generic, omit 5241 lgkmcnt(0). 5242 - Could be split into 5243 separate s_waitcnt 5244 vmcnt(0) and 5245 s_waitcnt 5246 lgkmcnt(0) to allow 5247 them to be 5248 independently moved 5249 according to the 5250 following rules. 5251 - s_waitcnt vmcnt(0) 5252 must happen after 5253 any preceding 5254 global/generic 5255 load/store/load 5256 atomic/store 5257 atomic/atomicrmw. 5258 - s_waitcnt lgkmcnt(0) 5259 must happen after 5260 any preceding 5261 local/generic 5262 load/store/load 5263 atomic/store 5264 atomic/atomicrmw. 5265 - Must happen before 5266 the following 5267 store. 5268 - Ensures that all 5269 memory operations 5270 to memory have 5271 completed before 5272 performing the 5273 store that is being 5274 released. 5275 5276 2. buffer/global/flat_store 5277 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5278 - wavefront - local 5279 - generic 5280 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5281 - generic 5282 - If OpenCL, omit. 5283 - Must happen after 5284 any preceding 5285 local/generic 5286 load/store/load 5287 atomic/store 5288 atomic/atomicrmw. 5289 - Must happen before 5290 the following 5291 atomicrmw. 5292 - Ensures that all 5293 memory operations 5294 to local have 5295 completed before 5296 performing the 5297 atomicrmw that is 5298 being released. 5299 5300 2. buffer/global/flat_atomic 5301 atomicrmw release - workgroup - local 1. ds_atomic 5302 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5303 - system - generic vmcnt(0) 5304 5305 - If OpenCL, omit 5306 lgkmcnt(0). 5307 - Could be split into 5308 separate s_waitcnt 5309 vmcnt(0) and 5310 s_waitcnt 5311 lgkmcnt(0) to allow 5312 them to be 5313 independently moved 5314 according to the 5315 following rules. 5316 - s_waitcnt vmcnt(0) 5317 must happen after 5318 any preceding 5319 global/generic 5320 load/store/load 5321 atomic/store 5322 atomic/atomicrmw. 5323 - s_waitcnt lgkmcnt(0) 5324 must happen after 5325 any preceding 5326 local/generic 5327 load/store/load 5328 atomic/store 5329 atomic/atomicrmw. 5330 - Must happen before 5331 the following 5332 atomicrmw. 5333 - Ensures that all 5334 memory operations 5335 to global and local 5336 have completed 5337 before performing 5338 the atomicrmw that 5339 is being released. 5340 5341 2. buffer/global/flat_atomic 5342 fence release - singlethread *none* *none* 5343 - wavefront 5344 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5345 5346 - If OpenCL and 5347 address space is 5348 not generic, omit. 5349 - However, since LLVM 5350 currently has no 5351 address space on 5352 the fence need to 5353 conservatively 5354 always generate. If 5355 fence had an 5356 address space then 5357 set to address 5358 space of OpenCL 5359 fence flag, or to 5360 generic if both 5361 local and global 5362 flags are 5363 specified. 5364 - Must happen after 5365 any preceding 5366 local/generic 5367 load/load 5368 atomic/store/store 5369 atomic/atomicrmw. 5370 - Must happen before 5371 any following store 5372 atomic/atomicrmw 5373 with an equal or 5374 wider sync scope 5375 and memory ordering 5376 stronger than 5377 unordered (this is 5378 termed the 5379 fence-paired-atomic). 5380 - Ensures that all 5381 memory operations 5382 to local have 5383 completed before 5384 performing the 5385 following 5386 fence-paired-atomic. 5387 5388 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5389 - system vmcnt(0) 5390 5391 - If OpenCL and 5392 address space is 5393 not generic, omit 5394 lgkmcnt(0). 5395 - If OpenCL and 5396 address space is 5397 local, omit 5398 vmcnt(0). 5399 - However, since LLVM 5400 currently has no 5401 address space on 5402 the fence need to 5403 conservatively 5404 always generate. If 5405 fence had an 5406 address space then 5407 set to address 5408 space of OpenCL 5409 fence flag, or to 5410 generic if both 5411 local and global 5412 flags are 5413 specified. 5414 - Could be split into 5415 separate s_waitcnt 5416 vmcnt(0) and 5417 s_waitcnt 5418 lgkmcnt(0) to allow 5419 them to be 5420 independently moved 5421 according to the 5422 following rules. 5423 - s_waitcnt vmcnt(0) 5424 must happen after 5425 any preceding 5426 global/generic 5427 load/store/load 5428 atomic/store 5429 atomic/atomicrmw. 5430 - s_waitcnt lgkmcnt(0) 5431 must happen after 5432 any preceding 5433 local/generic 5434 load/store/load 5435 atomic/store 5436 atomic/atomicrmw. 5437 - Must happen before 5438 any following store 5439 atomic/atomicrmw 5440 with an equal or 5441 wider sync scope 5442 and memory ordering 5443 stronger than 5444 unordered (this is 5445 termed the 5446 fence-paired-atomic). 5447 - Ensures that all 5448 memory operations 5449 have 5450 completed before 5451 performing the 5452 following 5453 fence-paired-atomic. 5454 5455 **Acquire-Release Atomic** 5456 ------------------------------------------------------------------------------------ 5457 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5458 - wavefront - local 5459 - generic 5460 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5461 5462 - If OpenCL, omit. 5463 - Must happen after 5464 any preceding 5465 local/generic 5466 load/store/load 5467 atomic/store 5468 atomic/atomicrmw. 5469 - Must happen before 5470 the following 5471 atomicrmw. 5472 - Ensures that all 5473 memory operations 5474 to local have 5475 completed before 5476 performing the 5477 atomicrmw that is 5478 being released. 5479 5480 2. buffer/global_atomic 5481 5482 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5483 2. s_waitcnt lgkmcnt(0) 5484 5485 - If OpenCL, omit. 5486 - Must happen before 5487 any following 5488 global/generic 5489 load/load 5490 atomic/store/store 5491 atomic/atomicrmw. 5492 - Ensures any 5493 following global 5494 data read is no 5495 older than the local load 5496 atomic value being 5497 acquired. 5498 5499 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5500 5501 - If OpenCL, omit. 5502 - Must happen after 5503 any preceding 5504 local/generic 5505 load/store/load 5506 atomic/store 5507 atomic/atomicrmw. 5508 - Must happen before 5509 the following 5510 atomicrmw. 5511 - Ensures that all 5512 memory operations 5513 to local have 5514 completed before 5515 performing the 5516 atomicrmw that is 5517 being released. 5518 5519 2. flat_atomic 5520 3. s_waitcnt lgkmcnt(0) 5521 5522 - If OpenCL, omit. 5523 - Must happen before 5524 any following 5525 global/generic 5526 load/load 5527 atomic/store/store 5528 atomic/atomicrmw. 5529 - Ensures any 5530 following global 5531 data read is no 5532 older than a local load 5533 atomic value being 5534 acquired. 5535 5536 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5537 - system vmcnt(0) 5538 5539 - If OpenCL, omit 5540 lgkmcnt(0). 5541 - Could be split into 5542 separate s_waitcnt 5543 vmcnt(0) and 5544 s_waitcnt 5545 lgkmcnt(0) to allow 5546 them to be 5547 independently moved 5548 according to the 5549 following rules. 5550 - s_waitcnt vmcnt(0) 5551 must happen after 5552 any preceding 5553 global/generic 5554 load/store/load 5555 atomic/store 5556 atomic/atomicrmw. 5557 - s_waitcnt lgkmcnt(0) 5558 must happen after 5559 any preceding 5560 local/generic 5561 load/store/load 5562 atomic/store 5563 atomic/atomicrmw. 5564 - Must happen before 5565 the following 5566 atomicrmw. 5567 - Ensures that all 5568 memory operations 5569 to global have 5570 completed before 5571 performing the 5572 atomicrmw that is 5573 being released. 5574 5575 2. buffer/global_atomic 5576 3. s_waitcnt vmcnt(0) 5577 5578 - Must happen before 5579 following 5580 buffer_wbinvl1_vol. 5581 - Ensures the 5582 atomicrmw has 5583 completed before 5584 invalidating the 5585 cache. 5586 5587 4. buffer_wbinvl1_vol 5588 5589 - Must happen before 5590 any following 5591 global/generic 5592 load/load 5593 atomic/atomicrmw. 5594 - Ensures that 5595 following loads 5596 will not see stale 5597 global data. 5598 5599 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5600 - system vmcnt(0) 5601 5602 - If OpenCL, omit 5603 lgkmcnt(0). 5604 - Could be split into 5605 separate s_waitcnt 5606 vmcnt(0) and 5607 s_waitcnt 5608 lgkmcnt(0) to allow 5609 them to be 5610 independently moved 5611 according to the 5612 following rules. 5613 - s_waitcnt vmcnt(0) 5614 must happen after 5615 any preceding 5616 global/generic 5617 load/store/load 5618 atomic/store 5619 atomic/atomicrmw. 5620 - s_waitcnt lgkmcnt(0) 5621 must happen after 5622 any preceding 5623 local/generic 5624 load/store/load 5625 atomic/store 5626 atomic/atomicrmw. 5627 - Must happen before 5628 the following 5629 atomicrmw. 5630 - Ensures that all 5631 memory operations 5632 to global have 5633 completed before 5634 performing the 5635 atomicrmw that is 5636 being released. 5637 5638 2. flat_atomic 5639 3. s_waitcnt vmcnt(0) & 5640 lgkmcnt(0) 5641 5642 - If OpenCL, omit 5643 lgkmcnt(0). 5644 - Must happen before 5645 following 5646 buffer_wbinvl1_vol. 5647 - Ensures the 5648 atomicrmw has 5649 completed before 5650 invalidating the 5651 cache. 5652 5653 4. buffer_wbinvl1_vol 5654 5655 - Must happen before 5656 any following 5657 global/generic 5658 load/load 5659 atomic/atomicrmw. 5660 - Ensures that 5661 following loads 5662 will not see stale 5663 global data. 5664 5665 fence acq_rel - singlethread *none* *none* 5666 - wavefront 5667 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5668 5669 - If OpenCL and 5670 address space is 5671 not generic, omit. 5672 - However, 5673 since LLVM 5674 currently has no 5675 address space on 5676 the fence need to 5677 conservatively 5678 always generate 5679 (see comment for 5680 previous fence). 5681 - Must happen after 5682 any preceding 5683 local/generic 5684 load/load 5685 atomic/store/store 5686 atomic/atomicrmw. 5687 - Must happen before 5688 any following 5689 global/generic 5690 load/load 5691 atomic/store/store 5692 atomic/atomicrmw. 5693 - Ensures that all 5694 memory operations 5695 to local have 5696 completed before 5697 performing any 5698 following global 5699 memory operations. 5700 - Ensures that the 5701 preceding 5702 local/generic load 5703 atomic/atomicrmw 5704 with an equal or 5705 wider sync scope 5706 and memory ordering 5707 stronger than 5708 unordered (this is 5709 termed the 5710 acquire-fence-paired-atomic) 5711 has completed 5712 before following 5713 global memory 5714 operations. This 5715 satisfies the 5716 requirements of 5717 acquire. 5718 - Ensures that all 5719 previous memory 5720 operations have 5721 completed before a 5722 following 5723 local/generic store 5724 atomic/atomicrmw 5725 with an equal or 5726 wider sync scope 5727 and memory ordering 5728 stronger than 5729 unordered (this is 5730 termed the 5731 release-fence-paired-atomic). 5732 This satisfies the 5733 requirements of 5734 release. 5735 5736 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5737 - system vmcnt(0) 5738 5739 - If OpenCL and 5740 address space is 5741 not generic, omit 5742 lgkmcnt(0). 5743 - However, since LLVM 5744 currently has no 5745 address space on 5746 the fence need to 5747 conservatively 5748 always generate 5749 (see comment for 5750 previous fence). 5751 - Could be split into 5752 separate s_waitcnt 5753 vmcnt(0) and 5754 s_waitcnt 5755 lgkmcnt(0) to allow 5756 them to be 5757 independently moved 5758 according to the 5759 following rules. 5760 - s_waitcnt vmcnt(0) 5761 must happen after 5762 any preceding 5763 global/generic 5764 load/store/load 5765 atomic/store 5766 atomic/atomicrmw. 5767 - s_waitcnt lgkmcnt(0) 5768 must happen after 5769 any preceding 5770 local/generic 5771 load/store/load 5772 atomic/store 5773 atomic/atomicrmw. 5774 - Must happen before 5775 the following 5776 buffer_wbinvl1_vol. 5777 - Ensures that the 5778 preceding 5779 global/local/generic 5780 load 5781 atomic/atomicrmw 5782 with an equal or 5783 wider sync scope 5784 and memory ordering 5785 stronger than 5786 unordered (this is 5787 termed the 5788 acquire-fence-paired-atomic) 5789 has completed 5790 before invalidating 5791 the cache. This 5792 satisfies the 5793 requirements of 5794 acquire. 5795 - Ensures that all 5796 previous memory 5797 operations have 5798 completed before a 5799 following 5800 global/local/generic 5801 store 5802 atomic/atomicrmw 5803 with an equal or 5804 wider sync scope 5805 and memory ordering 5806 stronger than 5807 unordered (this is 5808 termed the 5809 release-fence-paired-atomic). 5810 This satisfies the 5811 requirements of 5812 release. 5813 5814 2. buffer_wbinvl1_vol 5815 5816 - Must happen before 5817 any following 5818 global/generic 5819 load/load 5820 atomic/store/store 5821 atomic/atomicrmw. 5822 - Ensures that 5823 following loads 5824 will not see stale 5825 global data. This 5826 satisfies the 5827 requirements of 5828 acquire. 5829 5830 **Sequential Consistent Atomic** 5831 ------------------------------------------------------------------------------------ 5832 load atomic seq_cst - singlethread - global *Same as corresponding 5833 - wavefront - local load atomic acquire, 5834 - generic except must generated 5835 all instructions even 5836 for OpenCL.* 5837 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5838 - generic 5839 5840 - Must 5841 happen after 5842 preceding 5843 local/generic load 5844 atomic/store 5845 atomic/atomicrmw 5846 with memory 5847 ordering of seq_cst 5848 and with equal or 5849 wider sync scope. 5850 (Note that seq_cst 5851 fences have their 5852 own s_waitcnt 5853 lgkmcnt(0) and so do 5854 not need to be 5855 considered.) 5856 - Ensures any 5857 preceding 5858 sequential 5859 consistent local 5860 memory instructions 5861 have completed 5862 before executing 5863 this sequentially 5864 consistent 5865 instruction. This 5866 prevents reordering 5867 a seq_cst store 5868 followed by a 5869 seq_cst load. (Note 5870 that seq_cst is 5871 stronger than 5872 acquire/release as 5873 the reordering of 5874 load acquire 5875 followed by a store 5876 release is 5877 prevented by the 5878 s_waitcnt of 5879 the release, but 5880 there is nothing 5881 preventing a store 5882 release followed by 5883 load acquire from 5884 completing out of 5885 order. The s_waitcnt 5886 could be placed after 5887 seq_store or before 5888 the seq_load. We 5889 choose the load to 5890 make the s_waitcnt be 5891 as late as possible 5892 so that the store 5893 may have already 5894 completed.) 5895 5896 2. *Following 5897 instructions same as 5898 corresponding load 5899 atomic acquire, 5900 except must generated 5901 all instructions even 5902 for OpenCL.* 5903 load atomic seq_cst - workgroup - local *Same as corresponding 5904 load atomic acquire, 5905 except must generated 5906 all instructions even 5907 for OpenCL.* 5908 5909 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5910 - system - generic vmcnt(0) 5911 5912 - Could be split into 5913 separate s_waitcnt 5914 vmcnt(0) 5915 and s_waitcnt 5916 lgkmcnt(0) to allow 5917 them to be 5918 independently moved 5919 according to the 5920 following rules. 5921 - s_waitcnt lgkmcnt(0) 5922 must happen after 5923 preceding 5924 global/generic load 5925 atomic/store 5926 atomic/atomicrmw 5927 with memory 5928 ordering of seq_cst 5929 and with equal or 5930 wider sync scope. 5931 (Note that seq_cst 5932 fences have their 5933 own s_waitcnt 5934 lgkmcnt(0) and so do 5935 not need to be 5936 considered.) 5937 - s_waitcnt vmcnt(0) 5938 must happen after 5939 preceding 5940 global/generic load 5941 atomic/store 5942 atomic/atomicrmw 5943 with memory 5944 ordering of seq_cst 5945 and with equal or 5946 wider sync scope. 5947 (Note that seq_cst 5948 fences have their 5949 own s_waitcnt 5950 vmcnt(0) and so do 5951 not need to be 5952 considered.) 5953 - Ensures any 5954 preceding 5955 sequential 5956 consistent global 5957 memory instructions 5958 have completed 5959 before executing 5960 this sequentially 5961 consistent 5962 instruction. This 5963 prevents reordering 5964 a seq_cst store 5965 followed by a 5966 seq_cst load. (Note 5967 that seq_cst is 5968 stronger than 5969 acquire/release as 5970 the reordering of 5971 load acquire 5972 followed by a store 5973 release is 5974 prevented by the 5975 s_waitcnt of 5976 the release, but 5977 there is nothing 5978 preventing a store 5979 release followed by 5980 load acquire from 5981 completing out of 5982 order. The s_waitcnt 5983 could be placed after 5984 seq_store or before 5985 the seq_load. We 5986 choose the load to 5987 make the s_waitcnt be 5988 as late as possible 5989 so that the store 5990 may have already 5991 completed.) 5992 5993 2. *Following 5994 instructions same as 5995 corresponding load 5996 atomic acquire, 5997 except must generated 5998 all instructions even 5999 for OpenCL.* 6000 store atomic seq_cst - singlethread - global *Same as corresponding 6001 - wavefront - local store atomic release, 6002 - workgroup - generic except must generated 6003 - agent all instructions even 6004 - system for OpenCL.* 6005 atomicrmw seq_cst - singlethread - global *Same as corresponding 6006 - wavefront - local atomicrmw acq_rel, 6007 - workgroup - generic except must generated 6008 - agent all instructions even 6009 - system for OpenCL.* 6010 fence seq_cst - singlethread *none* *Same as corresponding 6011 - wavefront fence acq_rel, 6012 - workgroup except must generated 6013 - agent all instructions even 6014 - system for OpenCL.* 6015 ============ ============ ============== ========== ================================ 6016 6017.. _amdgpu-amdhsa-memory-model-gfx90a: 6018 6019Memory Model GFX90A 6020+++++++++++++++++++ 6021 6022For GFX90A: 6023 6024* Each agent has multiple shader arrays (SA). 6025* Each SA has multiple compute units (CU). 6026* Each CU has multiple SIMDs that execute wavefronts. 6027* The wavefronts for a single work-group are executed in the same CU but may be 6028 executed by different SIMDs. The exception is when in tgsplit execution mode 6029 when the wavefronts may be executed by different SIMDs in different CUs. 6030* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6031 executing on it. The exception is when in tgsplit execution mode when no LDS 6032 is allocated as wavefronts of the same work-group can be in different CUs. 6033* All LDS operations of a CU are performed as wavefront wide operations in a 6034 global order and involve no caching. Completion is reported to a wavefront in 6035 execution order. 6036* The LDS memory has multiple request queues shared by the SIMDs of a 6037 CU. Therefore, the LDS operations performed by different wavefronts of a 6038 work-group can be reordered relative to each other, which can result in 6039 reordering the visibility of vector memory operations with respect to LDS 6040 operations of other wavefronts in the same work-group. A ``s_waitcnt 6041 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6042 vector memory operations between wavefronts of a work-group, but not between 6043 operations performed by the same wavefront. 6044* The vector memory operations are performed as wavefront wide operations and 6045 completion is reported to a wavefront in execution order. The exception is 6046 that ``flat_load/store/atomic`` instructions can report out of vector memory 6047 order if they access LDS memory, and out of LDS operation order if they access 6048 global memory. 6049* The vector memory operations access a single vector L1 cache shared by all 6050 SIMDs a CU. Therefore: 6051 6052 * No special action is required for coherence between the lanes of a single 6053 wavefront. 6054 6055 * No special action is required for coherence between wavefronts in the same 6056 work-group since they execute on the same CU. The exception is when in 6057 tgsplit execution mode as wavefronts of the same work-group can be in 6058 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6059 the following item. 6060 6061 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6062 executing in different work-groups as they may be executing on different 6063 CUs. 6064 6065* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6066 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6067 scalar operations are used in a restricted way so do not impact the memory 6068 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6069* The vector and scalar memory operations use an L2 cache shared by all CUs on 6070 the same agent. 6071 6072 * The L2 cache has independent channels to service disjoint ranges of virtual 6073 addresses. 6074 * Each CU has a separate request queue per channel. Therefore, the vector and 6075 scalar memory operations performed by wavefronts executing in different 6076 work-groups (which may be executing on different CUs), or the same 6077 work-group if executing in tgsplit mode, of an agent can be reordered 6078 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6079 synchronization between vector memory operations of different CUs. It 6080 ensures a previous vector memory operation has completed before executing a 6081 subsequent vector memory or LDS operation and so can be used to meet the 6082 requirements of acquire and release. 6083 * The L2 cache of one agent can be kept coherent with other agents by using 6084 the MTYPE RW (read-write) for memory local to the L2, and MTYPE NC 6085 (non-coherent) with the PTE C-bit set for memory not local to the L2. 6086 6087 * Any local memory cache lines will be automatically invalidated by writes 6088 from CUs associated with other L2 caches, or writes from the CPU, due to 6089 the cache probe caused by the PTE C-bit. 6090 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6091 Subsequent access from the GPU will automatically invalidate or writeback 6092 the CPU cache due to the L2 probe filter. 6093 * Since all work-groups on the same agent share the same L2, no L2 6094 invalidation or writeback is required for coherence. 6095 * To ensure coherence of local memory writes of work-groups in different 6096 agents a ``buffer_wbl2`` is required. It will writeback dirty L2 cache 6097 lines. 6098 * To ensure coherence of local memory reads of work-groups in different 6099 agents a ``buffer_invl2`` is required. It will invalidate non-local L2 6100 cache lines. 6101 6102 * PCIe access from the GPU to the CPU memory can be kept coherent by using the 6103 MTYPE UC (uncached) which bypasses the L2. 6104 6105Scalar memory operations are only used to access memory that is proven to not 6106change during the execution of the kernel dispatch. This includes constant 6107address space and global address space for program scope ``const`` variables. 6108Therefore, the kernel machine code does not have to maintain the scalar cache to 6109ensure it is coherent with the vector caches. The scalar and vector caches are 6110invalidated between kernel dispatches by CP since constant address space data 6111may change between kernel dispatch executions. See 6112:ref:`amdgpu-amdhsa-memory-spaces`. 6113 6114The one exception is if scalar writes are used to spill SGPR registers. In this 6115case the AMDGPU backend ensures the memory location used to spill is never 6116accessed by vector memory operations at the same time. If scalar writes are used 6117then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6118return since the locations may be used for vector memory instructions by a 6119future wavefront that uses the same scratch area, or a function call that 6120creates a frame at the same address, respectively. There is no need for a 6121``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6122 6123For kernarg backing memory: 6124 6125* CP invalidates the L1 cache at the start of each kernel dispatch. 6126* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6127 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6128 cache. This also causes it to be treated as non-volatile and so is not 6129 invalidated by ``*_vol``. 6130* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6131 so the L2 cache will be coherent with the CPU and other agents. 6132 6133Scratch backing memory (which is used for the private address space) is accessed 6134with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6135only accessed by a single thread, and is always write-before-read, there is 6136never a need to invalidate these entries from the L1 cache. Hence all cache 6137invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6138 6139The code sequences used to implement the memory model for GFX90A are defined 6140in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6141 6142 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6143 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6144 6145 ============ ============ ============== ========== ================================ 6146 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6147 Ordering Sync Scope Address GFX90A 6148 Space 6149 ============ ============ ============== ========== ================================ 6150 **Non-Atomic** 6151 ------------------------------------------------------------------------------------ 6152 load *none* *none* - global - !volatile & !nontemporal 6153 - generic 6154 - private 1. buffer/global/flat_load 6155 - constant 6156 - !volatile & nontemporal 6157 6158 1. buffer/global/flat_load 6159 glc=1 slc=1 6160 6161 - volatile 6162 6163 1. buffer/global/flat_load 6164 glc=1 scc=1 6165 2. s_waitcnt vmcnt(0) 6166 6167 - Must happen before 6168 any following volatile 6169 global/generic 6170 load/store. 6171 - Ensures that 6172 volatile 6173 operations to 6174 different 6175 addresses will not 6176 be reordered by 6177 hardware. 6178 6179 load *none* *none* - local 1. ds_load 6180 store *none* *none* - global - !volatile & !nontemporal 6181 - generic 6182 - private 1. buffer/global/flat_store 6183 - constant 6184 - !volatile & nontemporal 6185 6186 1. buffer/global/flat_store 6187 glc=1 slc=1 6188 6189 - volatile 6190 6191 1. buffer/global/flat_store 6192 scc=1 6193 2. s_waitcnt vmcnt(0) 6194 6195 - Must happen before 6196 any following volatile 6197 global/generic 6198 load/store. 6199 - Ensures that 6200 volatile 6201 operations to 6202 different 6203 addresses will not 6204 be reordered by 6205 hardware. 6206 6207 store *none* *none* - local 1. ds_store 6208 **Unordered Atomic** 6209 ------------------------------------------------------------------------------------ 6210 load atomic unordered *any* *any* *Same as non-atomic*. 6211 store atomic unordered *any* *any* *Same as non-atomic*. 6212 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6213 **Monotonic Atomic** 6214 ------------------------------------------------------------------------------------ 6215 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6216 - wavefront - generic 6217 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6218 - generic glc=1 6219 6220 - If not TgSplit execution 6221 mode, omit glc=1. 6222 6223 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6224 - wavefront local address space cannot 6225 - workgroup be used.* 6226 6227 1. ds_load 6228 load atomic monotonic - agent - global 1. buffer/global/flat_load 6229 - generic glc=1 6230 load atomic monotonic - system - global 1. buffer/global/flat_load 6231 - generic glc=1 scc=1 6232 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6233 - wavefront - generic 6234 - workgroup 6235 - agent 6236 store atomic monotonic - system - global 1. buffer/global/flat_store 6237 - generic scc=1 6238 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6239 - wavefront local address space cannot 6240 - workgroup be used.* 6241 6242 1. ds_store 6243 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6244 - wavefront - generic 6245 - workgroup 6246 - agent 6247 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6248 - generic scc=1 6249 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6250 - wavefront local address space cannot 6251 - workgroup be used.* 6252 6253 1. ds_atomic 6254 **Acquire Atomic** 6255 ------------------------------------------------------------------------------------ 6256 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6257 - wavefront - local 6258 - generic 6259 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6260 6261 - If not TgSplit execution 6262 mode, omit glc=1. 6263 6264 2. s_waitcnt vmcnt(0) 6265 6266 - If not TgSplit execution 6267 mode, omit. 6268 - Must happen before the 6269 following buffer_wbinvl1_vol. 6270 6271 3. buffer_wbinvl1_vol 6272 6273 - If not TgSplit execution 6274 mode, omit. 6275 - Must happen before 6276 any following 6277 global/generic 6278 load/load 6279 atomic/store/store 6280 atomic/atomicrmw. 6281 - Ensures that 6282 following 6283 loads will not see 6284 stale data. 6285 6286 load atomic acquire - workgroup - local *If TgSplit execution mode, 6287 local address space cannot 6288 be used.* 6289 6290 1. ds_load 6291 2. s_waitcnt lgkmcnt(0) 6292 6293 - If OpenCL, omit. 6294 - Must happen before 6295 any following 6296 global/generic 6297 load/load 6298 atomic/store/store 6299 atomic/atomicrmw. 6300 - Ensures any 6301 following global 6302 data read is no 6303 older than the local load 6304 atomic value being 6305 acquired. 6306 6307 load atomic acquire - workgroup - generic 1. flat_load glc=1 6308 6309 - If not TgSplit execution 6310 mode, omit glc=1. 6311 6312 2. s_waitcnt lgkm/vmcnt(0) 6313 6314 - Use lgkmcnt(0) if not 6315 TgSplit execution mode 6316 and vmcnt(0) if TgSplit 6317 execution mode. 6318 - If OpenCL, omit lgkmcnt(0). 6319 - Must happen before 6320 the following 6321 buffer_wbinvl1_vol and any 6322 following global/generic 6323 load/load 6324 atomic/store/store 6325 atomic/atomicrmw. 6326 - Ensures any 6327 following global 6328 data read is no 6329 older than a local load 6330 atomic value being 6331 acquired. 6332 6333 3. buffer_wbinvl1_vol 6334 6335 - If not TgSplit execution 6336 mode, omit. 6337 - Ensures that 6338 following 6339 loads will not see 6340 stale data. 6341 6342 load atomic acquire - agent - global 1. buffer/global_load 6343 glc=1 6344 2. s_waitcnt vmcnt(0) 6345 6346 - Must happen before 6347 following 6348 buffer_wbinvl1_vol. 6349 - Ensures the load 6350 has completed 6351 before invalidating 6352 the cache. 6353 6354 3. buffer_wbinvl1_vol 6355 6356 - Must happen before 6357 any following 6358 global/generic 6359 load/load 6360 atomic/atomicrmw. 6361 - Ensures that 6362 following 6363 loads will not see 6364 stale global data. 6365 6366 load atomic acquire - system - global 1. buffer/global/flat_load 6367 glc=1 scc=1 6368 2. s_waitcnt vmcnt(0) 6369 6370 - Must happen before 6371 following buffer_invl2 and 6372 buffer_wbinvl1_vol. 6373 - Ensures the load 6374 has completed 6375 before invalidating 6376 the cache. 6377 6378 3. buffer_invl2; 6379 buffer_wbinvl1_vol 6380 6381 - Must happen before 6382 any following 6383 global/generic 6384 load/load 6385 atomic/atomicrmw. 6386 - Ensures that 6387 following 6388 loads will not see 6389 stale MTYPE NC global data. 6390 MTYPE RW and CC memory will 6391 never be stale due to the 6392 memory probes. 6393 6394 load atomic acquire - agent - generic 1. flat_load glc=1 6395 2. s_waitcnt vmcnt(0) & 6396 lgkmcnt(0) 6397 6398 - If TgSplit execution mode, 6399 omit lgkmcnt(0). 6400 - If OpenCL omit 6401 lgkmcnt(0). 6402 - Must happen before 6403 following 6404 buffer_wbinvl1_vol. 6405 - Ensures the flat_load 6406 has completed 6407 before invalidating 6408 the cache. 6409 6410 3. buffer_wbinvl1_vol 6411 6412 - Must happen before 6413 any following 6414 global/generic 6415 load/load 6416 atomic/atomicrmw. 6417 - Ensures that 6418 following loads 6419 will not see stale 6420 global data. 6421 6422 load atomic acquire - system - generic 1. flat_load glc=1 scc=1 6423 2. s_waitcnt vmcnt(0) & 6424 lgkmcnt(0) 6425 6426 - If TgSplit execution mode, 6427 omit lgkmcnt(0). 6428 - If OpenCL omit 6429 lgkmcnt(0). 6430 - Must happen before 6431 following 6432 buffer_invl2 and 6433 buffer_wbinvl1_vol. 6434 - Ensures the flat_load 6435 has completed 6436 before invalidating 6437 the caches. 6438 6439 3. buffer_invl2; 6440 buffer_wbinvl1_vol 6441 6442 - Must happen before 6443 any following 6444 global/generic 6445 load/load 6446 atomic/atomicrmw. 6447 - Ensures that 6448 following 6449 loads will not see 6450 stale MTYPE NC global data. 6451 MTYPE RW and CC memory will 6452 never be stale due to the 6453 memory probes. 6454 6455 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6456 - wavefront - generic 6457 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6458 - wavefront local address space cannot 6459 be used.* 6460 6461 1. ds_atomic 6462 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6463 2. s_waitcnt vmcnt(0) 6464 6465 - If not TgSplit execution 6466 mode, omit. 6467 - Must happen before the 6468 following buffer_wbinvl1_vol. 6469 - Ensures the atomicrmw 6470 has completed 6471 before invalidating 6472 the cache. 6473 6474 3. buffer_wbinvl1_vol 6475 6476 - If not TgSplit execution 6477 mode, omit. 6478 - Must happen before 6479 any following 6480 global/generic 6481 load/load 6482 atomic/atomicrmw. 6483 - Ensures that 6484 following loads 6485 will not see stale 6486 global data. 6487 6488 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6489 local address space cannot 6490 be used.* 6491 6492 1. ds_atomic 6493 2. s_waitcnt lgkmcnt(0) 6494 6495 - If OpenCL, omit. 6496 - Must happen before 6497 any following 6498 global/generic 6499 load/load 6500 atomic/store/store 6501 atomic/atomicrmw. 6502 - Ensures any 6503 following global 6504 data read is no 6505 older than the local 6506 atomicrmw value 6507 being acquired. 6508 6509 atomicrmw acquire - workgroup - generic 1. flat_atomic 6510 2. s_waitcnt lgkm/vmcnt(0) 6511 6512 - Use lgkmcnt(0) if not 6513 TgSplit execution mode 6514 and vmcnt(0) if TgSplit 6515 execution mode. 6516 - If OpenCL, omit lgkmcnt(0). 6517 - Must happen before 6518 the following 6519 buffer_wbinvl1_vol and 6520 any following 6521 global/generic 6522 load/load 6523 atomic/store/store 6524 atomic/atomicrmw. 6525 - Ensures any 6526 following global 6527 data read is no 6528 older than a local 6529 atomicrmw value 6530 being acquired. 6531 6532 3. buffer_wbinvl1_vol 6533 6534 - If not TgSplit execution 6535 mode, omit. 6536 - Ensures that 6537 following 6538 loads will not see 6539 stale data. 6540 6541 atomicrmw acquire - agent - global 1. buffer/global_atomic 6542 2. s_waitcnt vmcnt(0) 6543 6544 - Must happen before 6545 following 6546 buffer_wbinvl1_vol. 6547 - Ensures the 6548 atomicrmw has 6549 completed before 6550 invalidating the 6551 cache. 6552 6553 3. buffer_wbinvl1_vol 6554 6555 - Must happen before 6556 any following 6557 global/generic 6558 load/load 6559 atomic/atomicrmw. 6560 - Ensures that 6561 following loads 6562 will not see stale 6563 global data. 6564 6565 atomicrmw acquire - system - global 1. buffer/global_atomic 6566 scc=1 6567 2. s_waitcnt vmcnt(0) 6568 6569 - Must happen before 6570 following buffer_invl2 and 6571 buffer_wbinvl1_vol. 6572 - Ensures the 6573 atomicrmw has 6574 completed before 6575 invalidating the 6576 caches. 6577 6578 3. buffer_invl2; 6579 buffer_wbinvl1_vol 6580 6581 - Must happen before 6582 any following 6583 global/generic 6584 load/load 6585 atomic/atomicrmw. 6586 - Ensures that 6587 following 6588 loads will not see 6589 stale MTYPE NC global data. 6590 MTYPE RW and CC memory will 6591 never be stale due to the 6592 memory probes. 6593 6594 atomicrmw acquire - agent - generic 1. flat_atomic 6595 2. s_waitcnt vmcnt(0) & 6596 lgkmcnt(0) 6597 6598 - If TgSplit execution mode, 6599 omit lgkmcnt(0). 6600 - If OpenCL, omit 6601 lgkmcnt(0). 6602 - Must happen before 6603 following 6604 buffer_wbinvl1_vol. 6605 - Ensures the 6606 atomicrmw has 6607 completed before 6608 invalidating the 6609 cache. 6610 6611 3. buffer_wbinvl1_vol 6612 6613 - Must happen before 6614 any following 6615 global/generic 6616 load/load 6617 atomic/atomicrmw. 6618 - Ensures that 6619 following loads 6620 will not see stale 6621 global data. 6622 6623 atomicrmw acquire - system - generic 1. flat_atomic scc=1 6624 2. s_waitcnt vmcnt(0) & 6625 lgkmcnt(0) 6626 6627 - If TgSplit execution mode, 6628 omit lgkmcnt(0). 6629 - If OpenCL, omit 6630 lgkmcnt(0). 6631 - Must happen before 6632 following 6633 buffer_invl2 and 6634 buffer_wbinvl1_vol. 6635 - Ensures the 6636 atomicrmw has 6637 completed before 6638 invalidating the 6639 caches. 6640 6641 3. buffer_invl2; 6642 buffer_wbinvl1_vol 6643 6644 - Must happen before 6645 any following 6646 global/generic 6647 load/load 6648 atomic/atomicrmw. 6649 - Ensures that 6650 following 6651 loads will not see 6652 stale MTYPE NC global data. 6653 MTYPE RW and CC memory will 6654 never be stale due to the 6655 memory probes. 6656 6657 fence acquire - singlethread *none* *none* 6658 - wavefront 6659 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6660 6661 - Use lgkmcnt(0) if not 6662 TgSplit execution mode 6663 and vmcnt(0) if TgSplit 6664 execution mode. 6665 - If OpenCL and 6666 address space is 6667 not generic, omit 6668 lgkmcnt(0). 6669 - If OpenCL and 6670 address space is 6671 local, omit 6672 vmcnt(0). 6673 - However, since LLVM 6674 currently has no 6675 address space on 6676 the fence need to 6677 conservatively 6678 always generate. If 6679 fence had an 6680 address space then 6681 set to address 6682 space of OpenCL 6683 fence flag, or to 6684 generic if both 6685 local and global 6686 flags are 6687 specified. 6688 - s_waitcnt vmcnt(0) 6689 must happen after 6690 any preceding 6691 global/generic load 6692 atomic/ 6693 atomicrmw 6694 with an equal or 6695 wider sync scope 6696 and memory ordering 6697 stronger than 6698 unordered (this is 6699 termed the 6700 fence-paired-atomic). 6701 - s_waitcnt lgkmcnt(0) 6702 must happen after 6703 any preceding 6704 local/generic load 6705 atomic/atomicrmw 6706 with an equal or 6707 wider sync scope 6708 and memory ordering 6709 stronger than 6710 unordered (this is 6711 termed the 6712 fence-paired-atomic). 6713 - Must happen before 6714 the following 6715 buffer_wbinvl1_vol and 6716 any following 6717 global/generic 6718 load/load 6719 atomic/store/store 6720 atomic/atomicrmw. 6721 - Ensures any 6722 following global 6723 data read is no 6724 older than the 6725 value read by the 6726 fence-paired-atomic. 6727 6728 3. buffer_wbinvl1_vol 6729 6730 - If not TgSplit execution 6731 mode, omit. 6732 - Ensures that 6733 following 6734 loads will not see 6735 stale data. 6736 6737 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6738 vmcnt(0) 6739 6740 - If TgSplit execution mode, 6741 omit lgkmcnt(0). 6742 - If OpenCL and 6743 address space is 6744 not generic, omit 6745 lgkmcnt(0). 6746 - However, since LLVM 6747 currently has no 6748 address space on 6749 the fence need to 6750 conservatively 6751 always generate 6752 (see comment for 6753 previous fence). 6754 - Could be split into 6755 separate s_waitcnt 6756 vmcnt(0) and 6757 s_waitcnt 6758 lgkmcnt(0) to allow 6759 them to be 6760 independently moved 6761 according to the 6762 following rules. 6763 - s_waitcnt vmcnt(0) 6764 must happen after 6765 any preceding 6766 global/generic load 6767 atomic/atomicrmw 6768 with an equal or 6769 wider sync scope 6770 and memory ordering 6771 stronger than 6772 unordered (this is 6773 termed the 6774 fence-paired-atomic). 6775 - s_waitcnt lgkmcnt(0) 6776 must happen after 6777 any preceding 6778 local/generic load 6779 atomic/atomicrmw 6780 with an equal or 6781 wider sync scope 6782 and memory ordering 6783 stronger than 6784 unordered (this is 6785 termed the 6786 fence-paired-atomic). 6787 - Must happen before 6788 the following 6789 buffer_wbinvl1_vol. 6790 - Ensures that the 6791 fence-paired atomic 6792 has completed 6793 before invalidating 6794 the 6795 cache. Therefore 6796 any following 6797 locations read must 6798 be no older than 6799 the value read by 6800 the 6801 fence-paired-atomic. 6802 6803 2. buffer_wbinvl1_vol 6804 6805 - Must happen before any 6806 following global/generic 6807 load/load 6808 atomic/store/store 6809 atomic/atomicrmw. 6810 - Ensures that 6811 following loads 6812 will not see stale 6813 global data. 6814 6815 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6816 vmcnt(0) 6817 6818 - If TgSplit execution mode, 6819 omit lgkmcnt(0). 6820 - If OpenCL and 6821 address space is 6822 not generic, omit 6823 lgkmcnt(0). 6824 - However, since LLVM 6825 currently has no 6826 address space on 6827 the fence need to 6828 conservatively 6829 always generate 6830 (see comment for 6831 previous fence). 6832 - Could be split into 6833 separate s_waitcnt 6834 vmcnt(0) and 6835 s_waitcnt 6836 lgkmcnt(0) to allow 6837 them to be 6838 independently moved 6839 according to the 6840 following rules. 6841 - s_waitcnt vmcnt(0) 6842 must happen after 6843 any preceding 6844 global/generic load 6845 atomic/atomicrmw 6846 with an equal or 6847 wider sync scope 6848 and memory ordering 6849 stronger than 6850 unordered (this is 6851 termed the 6852 fence-paired-atomic). 6853 - s_waitcnt lgkmcnt(0) 6854 must happen after 6855 any preceding 6856 local/generic load 6857 atomic/atomicrmw 6858 with an equal or 6859 wider sync scope 6860 and memory ordering 6861 stronger than 6862 unordered (this is 6863 termed the 6864 fence-paired-atomic). 6865 - Must happen before 6866 the following buffer_invl2 and 6867 buffer_wbinvl1_vol. 6868 - Ensures that the 6869 fence-paired atomic 6870 has completed 6871 before invalidating 6872 the 6873 cache. Therefore 6874 any following 6875 locations read must 6876 be no older than 6877 the value read by 6878 the 6879 fence-paired-atomic. 6880 6881 2. buffer_invl2; 6882 buffer_wbinvl1_vol 6883 6884 - Must happen before any 6885 following global/generic 6886 load/load 6887 atomic/store/store 6888 atomic/atomicrmw. 6889 - Ensures that 6890 following loads 6891 will not see stale 6892 global data. 6893 6894 **Release Atomic** 6895 ------------------------------------------------------------------------------------ 6896 store atomic release - singlethread - global 1. buffer/global/flat_store 6897 - wavefront - generic 6898 store atomic release - singlethread - local *If TgSplit execution mode, 6899 - wavefront local address space cannot 6900 be used.* 6901 6902 1. ds_store 6903 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6904 - generic 6905 - Use lgkmcnt(0) if not 6906 TgSplit execution mode 6907 and vmcnt(0) if TgSplit 6908 execution mode. 6909 - If OpenCL, omit lgkmcnt(0). 6910 - s_waitcnt vmcnt(0) 6911 must happen after 6912 any preceding 6913 global/generic load/store/ 6914 load atomic/store atomic/ 6915 atomicrmw. 6916 - s_waitcnt lgkmcnt(0) 6917 must happen after 6918 any preceding 6919 local/generic 6920 load/store/load 6921 atomic/store 6922 atomic/atomicrmw. 6923 - Must happen before 6924 the following 6925 store. 6926 - Ensures that all 6927 memory operations 6928 have 6929 completed before 6930 performing the 6931 store that is being 6932 released. 6933 6934 2. buffer/global/flat_store 6935 store atomic release - workgroup - local *If TgSplit execution mode, 6936 local address space cannot 6937 be used.* 6938 6939 1. ds_store 6940 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6941 - generic vmcnt(0) 6942 6943 - If TgSplit execution mode, 6944 omit lgkmcnt(0). 6945 - If OpenCL and 6946 address space is 6947 not generic, omit 6948 lgkmcnt(0). 6949 - Could be split into 6950 separate s_waitcnt 6951 vmcnt(0) and 6952 s_waitcnt 6953 lgkmcnt(0) to allow 6954 them to be 6955 independently moved 6956 according to the 6957 following rules. 6958 - s_waitcnt vmcnt(0) 6959 must happen after 6960 any preceding 6961 global/generic 6962 load/store/load 6963 atomic/store 6964 atomic/atomicrmw. 6965 - s_waitcnt lgkmcnt(0) 6966 must happen after 6967 any preceding 6968 local/generic 6969 load/store/load 6970 atomic/store 6971 atomic/atomicrmw. 6972 - Must happen before 6973 the following 6974 store. 6975 - Ensures that all 6976 memory operations 6977 to memory have 6978 completed before 6979 performing the 6980 store that is being 6981 released. 6982 6983 2. buffer/global/flat_store 6984 store atomic release - system - global 1. buffer_wbl2 6985 - generic 6986 - Must happen before 6987 following s_waitcnt. 6988 - Performs L2 writeback to 6989 ensure previous 6990 global/generic 6991 store/atomicrmw are 6992 visible at system scope. 6993 6994 2. s_waitcnt lgkmcnt(0) & 6995 vmcnt(0) 6996 6997 - If TgSplit execution mode, 6998 omit lgkmcnt(0). 6999 - If OpenCL and 7000 address space is 7001 not generic, omit 7002 lgkmcnt(0). 7003 - Could be split into 7004 separate s_waitcnt 7005 vmcnt(0) and 7006 s_waitcnt 7007 lgkmcnt(0) to allow 7008 them to be 7009 independently moved 7010 according to the 7011 following rules. 7012 - s_waitcnt vmcnt(0) 7013 must happen after any 7014 preceding 7015 global/generic 7016 load/store/load 7017 atomic/store 7018 atomic/atomicrmw. 7019 - s_waitcnt lgkmcnt(0) 7020 must happen after any 7021 preceding 7022 local/generic 7023 load/store/load 7024 atomic/store 7025 atomic/atomicrmw. 7026 - Must happen before 7027 the following 7028 store. 7029 - Ensures that all 7030 memory operations 7031 to memory and the L2 7032 writeback have 7033 completed before 7034 performing the 7035 store that is being 7036 released. 7037 7038 2. buffer/global/flat_store 7039 scc=1 7040 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7041 - wavefront - generic 7042 atomicrmw release - singlethread - local *If TgSplit execution mode, 7043 - wavefront local address space cannot 7044 be used.* 7045 7046 1. ds_atomic 7047 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7048 - generic 7049 - Use lgkmcnt(0) if not 7050 TgSplit execution mode 7051 and vmcnt(0) if TgSplit 7052 execution mode. 7053 - If OpenCL, omit 7054 lgkmcnt(0). 7055 - s_waitcnt vmcnt(0) 7056 must happen after 7057 any preceding 7058 global/generic load/store/ 7059 load atomic/store atomic/ 7060 atomicrmw. 7061 - s_waitcnt lgkmcnt(0) 7062 must happen after 7063 any preceding 7064 local/generic 7065 load/store/load 7066 atomic/store 7067 atomic/atomicrmw. 7068 - Must happen before 7069 the following 7070 atomicrmw. 7071 - Ensures that all 7072 memory operations 7073 have 7074 completed before 7075 performing the 7076 atomicrmw that is 7077 being released. 7078 7079 2. buffer/global/flat_atomic 7080 atomicrmw release - workgroup - local *If TgSplit execution mode, 7081 local address space cannot 7082 be used.* 7083 7084 1. ds_atomic 7085 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7086 - generic vmcnt(0) 7087 7088 - If TgSplit execution mode, 7089 omit lgkmcnt(0). 7090 - If OpenCL, omit 7091 lgkmcnt(0). 7092 - Could be split into 7093 separate s_waitcnt 7094 vmcnt(0) and 7095 s_waitcnt 7096 lgkmcnt(0) to allow 7097 them to be 7098 independently moved 7099 according to the 7100 following rules. 7101 - s_waitcnt vmcnt(0) 7102 must happen after 7103 any preceding 7104 global/generic 7105 load/store/load 7106 atomic/store 7107 atomic/atomicrmw. 7108 - s_waitcnt lgkmcnt(0) 7109 must happen after 7110 any preceding 7111 local/generic 7112 load/store/load 7113 atomic/store 7114 atomic/atomicrmw. 7115 - Must happen before 7116 the following 7117 atomicrmw. 7118 - Ensures that all 7119 memory operations 7120 to global and local 7121 have completed 7122 before performing 7123 the atomicrmw that 7124 is being released. 7125 7126 2. buffer/global/flat_atomic 7127 atomicrmw release - system - global 1. buffer_wbl2 7128 - generic 7129 - Must happen before 7130 following s_waitcnt. 7131 - Performs L2 writeback to 7132 ensure previous 7133 global/generic 7134 store/atomicrmw are 7135 visible at system scope. 7136 7137 2. s_waitcnt lgkmcnt(0) & 7138 vmcnt(0) 7139 7140 - If TgSplit execution mode, 7141 omit lgkmcnt(0). 7142 - If OpenCL, omit 7143 lgkmcnt(0). 7144 - Could be split into 7145 separate s_waitcnt 7146 vmcnt(0) and 7147 s_waitcnt 7148 lgkmcnt(0) to allow 7149 them to be 7150 independently moved 7151 according to the 7152 following rules. 7153 - s_waitcnt vmcnt(0) 7154 must happen after 7155 any preceding 7156 global/generic 7157 load/store/load 7158 atomic/store 7159 atomic/atomicrmw. 7160 - s_waitcnt lgkmcnt(0) 7161 must happen after 7162 any preceding 7163 local/generic 7164 load/store/load 7165 atomic/store 7166 atomic/atomicrmw. 7167 - Must happen before 7168 the following 7169 atomicrmw. 7170 - Ensures that all 7171 memory operations 7172 to memory and the L2 7173 writeback have 7174 completed before 7175 performing the 7176 store that is being 7177 released. 7178 7179 3. buffer/global/flat_atomic 7180 scc=1 7181 fence release - singlethread *none* *none* 7182 - wavefront 7183 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7184 7185 - Use lgkmcnt(0) if not 7186 TgSplit execution mode 7187 and vmcnt(0) if TgSplit 7188 execution mode. 7189 - If OpenCL and 7190 address space is 7191 not generic, omit 7192 lgkmcnt(0). 7193 - If OpenCL and 7194 address space is 7195 local, omit 7196 vmcnt(0). 7197 - However, since LLVM 7198 currently has no 7199 address space on 7200 the fence need to 7201 conservatively 7202 always generate. If 7203 fence had an 7204 address space then 7205 set to address 7206 space of OpenCL 7207 fence flag, or to 7208 generic if both 7209 local and global 7210 flags are 7211 specified. 7212 - s_waitcnt vmcnt(0) 7213 must happen after 7214 any preceding 7215 global/generic 7216 load/store/ 7217 load atomic/store atomic/ 7218 atomicrmw. 7219 - s_waitcnt lgkmcnt(0) 7220 must happen after 7221 any preceding 7222 local/generic 7223 load/load 7224 atomic/store/store 7225 atomic/atomicrmw. 7226 - Must happen before 7227 any following store 7228 atomic/atomicrmw 7229 with an equal or 7230 wider sync scope 7231 and memory ordering 7232 stronger than 7233 unordered (this is 7234 termed the 7235 fence-paired-atomic). 7236 - Ensures that all 7237 memory operations 7238 have 7239 completed before 7240 performing the 7241 following 7242 fence-paired-atomic. 7243 7244 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7245 vmcnt(0) 7246 7247 - If TgSplit execution mode, 7248 omit lgkmcnt(0). 7249 - If OpenCL and 7250 address space is 7251 not generic, omit 7252 lgkmcnt(0). 7253 - If OpenCL and 7254 address space is 7255 local, omit 7256 vmcnt(0). 7257 - However, since LLVM 7258 currently has no 7259 address space on 7260 the fence need to 7261 conservatively 7262 always generate. If 7263 fence had an 7264 address space then 7265 set to address 7266 space of OpenCL 7267 fence flag, or to 7268 generic if both 7269 local and global 7270 flags are 7271 specified. 7272 - Could be split into 7273 separate s_waitcnt 7274 vmcnt(0) and 7275 s_waitcnt 7276 lgkmcnt(0) to allow 7277 them to be 7278 independently moved 7279 according to the 7280 following rules. 7281 - s_waitcnt vmcnt(0) 7282 must happen after 7283 any preceding 7284 global/generic 7285 load/store/load 7286 atomic/store 7287 atomic/atomicrmw. 7288 - s_waitcnt lgkmcnt(0) 7289 must happen after 7290 any preceding 7291 local/generic 7292 load/store/load 7293 atomic/store 7294 atomic/atomicrmw. 7295 - Must happen before 7296 any following store 7297 atomic/atomicrmw 7298 with an equal or 7299 wider sync scope 7300 and memory ordering 7301 stronger than 7302 unordered (this is 7303 termed the 7304 fence-paired-atomic). 7305 - Ensures that all 7306 memory operations 7307 have 7308 completed before 7309 performing the 7310 following 7311 fence-paired-atomic. 7312 7313 fence release - system *none* 1. buffer_wbl2 7314 7315 - If OpenCL and 7316 address space is 7317 local, omit. 7318 - Must happen before 7319 following s_waitcnt. 7320 - Performs L2 writeback to 7321 ensure previous 7322 global/generic 7323 store/atomicrmw are 7324 visible at system scope. 7325 7326 2. s_waitcnt lgkmcnt(0) & 7327 vmcnt(0) 7328 7329 - If TgSplit execution mode, 7330 omit lgkmcnt(0). 7331 - If OpenCL and 7332 address space is 7333 not generic, omit 7334 lgkmcnt(0). 7335 - If OpenCL and 7336 address space is 7337 local, omit 7338 vmcnt(0). 7339 - However, since LLVM 7340 currently has no 7341 address space on 7342 the fence need to 7343 conservatively 7344 always generate. If 7345 fence had an 7346 address space then 7347 set to address 7348 space of OpenCL 7349 fence flag, or to 7350 generic if both 7351 local and global 7352 flags are 7353 specified. 7354 - Could be split into 7355 separate s_waitcnt 7356 vmcnt(0) and 7357 s_waitcnt 7358 lgkmcnt(0) to allow 7359 them to be 7360 independently moved 7361 according to the 7362 following rules. 7363 - s_waitcnt vmcnt(0) 7364 must happen after 7365 any preceding 7366 global/generic 7367 load/store/load 7368 atomic/store 7369 atomic/atomicrmw. 7370 - s_waitcnt lgkmcnt(0) 7371 must happen after 7372 any preceding 7373 local/generic 7374 load/store/load 7375 atomic/store 7376 atomic/atomicrmw. 7377 - Must happen before 7378 any following store 7379 atomic/atomicrmw 7380 with an equal or 7381 wider sync scope 7382 and memory ordering 7383 stronger than 7384 unordered (this is 7385 termed the 7386 fence-paired-atomic). 7387 - Ensures that all 7388 memory operations 7389 have 7390 completed before 7391 performing the 7392 following 7393 fence-paired-atomic. 7394 7395 **Acquire-Release Atomic** 7396 ------------------------------------------------------------------------------------ 7397 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7398 - wavefront - generic 7399 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7400 - wavefront local address space cannot 7401 be used.* 7402 7403 1. ds_atomic 7404 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7405 7406 - Use lgkmcnt(0) if not 7407 TgSplit execution mode 7408 and vmcnt(0) if TgSplit 7409 execution mode. 7410 - If OpenCL, omit 7411 lgkmcnt(0). 7412 - Must happen after 7413 any preceding 7414 local/generic 7415 load/store/load 7416 atomic/store 7417 atomic/atomicrmw. 7418 - s_waitcnt vmcnt(0) 7419 must happen after 7420 any preceding 7421 global/generic load/store/ 7422 load atomic/store atomic/ 7423 atomicrmw. 7424 - s_waitcnt lgkmcnt(0) 7425 must happen after 7426 any preceding 7427 local/generic 7428 load/store/load 7429 atomic/store 7430 atomic/atomicrmw. 7431 - Must happen before 7432 the following 7433 atomicrmw. 7434 - Ensures that all 7435 memory operations 7436 have 7437 completed before 7438 performing the 7439 atomicrmw that is 7440 being released. 7441 7442 2. buffer/global_atomic 7443 3. s_waitcnt vmcnt(0) 7444 7445 - If not TgSplit execution 7446 mode, omit. 7447 - Must happen before 7448 the following 7449 buffer_wbinvl1_vol. 7450 - Ensures any 7451 following global 7452 data read is no 7453 older than the 7454 atomicrmw value 7455 being acquired. 7456 7457 4. buffer_wbinvl1_vol 7458 7459 - If not TgSplit execution 7460 mode, omit. 7461 - Ensures that 7462 following 7463 loads will not see 7464 stale data. 7465 7466 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7467 local address space cannot 7468 be used.* 7469 7470 1. ds_atomic 7471 2. s_waitcnt lgkmcnt(0) 7472 7473 - If OpenCL, omit. 7474 - Must happen before 7475 any following 7476 global/generic 7477 load/load 7478 atomic/store/store 7479 atomic/atomicrmw. 7480 - Ensures any 7481 following global 7482 data read is no 7483 older than the local load 7484 atomic value being 7485 acquired. 7486 7487 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7488 7489 - Use lgkmcnt(0) if not 7490 TgSplit execution mode 7491 and vmcnt(0) if TgSplit 7492 execution mode. 7493 - If OpenCL, omit 7494 lgkmcnt(0). 7495 - s_waitcnt vmcnt(0) 7496 must happen after 7497 any preceding 7498 global/generic load/store/ 7499 load atomic/store atomic/ 7500 atomicrmw. 7501 - s_waitcnt lgkmcnt(0) 7502 must happen after 7503 any preceding 7504 local/generic 7505 load/store/load 7506 atomic/store 7507 atomic/atomicrmw. 7508 - Must happen before 7509 the following 7510 atomicrmw. 7511 - Ensures that all 7512 memory operations 7513 have 7514 completed before 7515 performing the 7516 atomicrmw that is 7517 being released. 7518 7519 2. flat_atomic 7520 3. s_waitcnt lgkmcnt(0) & 7521 vmcnt(0) 7522 7523 - If not TgSplit execution 7524 mode, omit vmcnt(0). 7525 - If OpenCL, omit 7526 lgkmcnt(0). 7527 - Must happen before 7528 the following 7529 buffer_wbinvl1_vol and 7530 any following 7531 global/generic 7532 load/load 7533 atomic/store/store 7534 atomic/atomicrmw. 7535 - Ensures any 7536 following global 7537 data read is no 7538 older than a local load 7539 atomic value being 7540 acquired. 7541 7542 3. buffer_wbinvl1_vol 7543 7544 - If not TgSplit execution 7545 mode, omit. 7546 - Ensures that 7547 following 7548 loads will not see 7549 stale data. 7550 7551 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7552 vmcnt(0) 7553 7554 - If TgSplit execution mode, 7555 omit lgkmcnt(0). 7556 - If OpenCL, omit 7557 lgkmcnt(0). 7558 - Could be split into 7559 separate s_waitcnt 7560 vmcnt(0) and 7561 s_waitcnt 7562 lgkmcnt(0) to allow 7563 them to be 7564 independently moved 7565 according to the 7566 following rules. 7567 - s_waitcnt vmcnt(0) 7568 must happen after 7569 any preceding 7570 global/generic 7571 load/store/load 7572 atomic/store 7573 atomic/atomicrmw. 7574 - s_waitcnt lgkmcnt(0) 7575 must happen after 7576 any preceding 7577 local/generic 7578 load/store/load 7579 atomic/store 7580 atomic/atomicrmw. 7581 - Must happen before 7582 the following 7583 atomicrmw. 7584 - Ensures that all 7585 memory operations 7586 to global have 7587 completed before 7588 performing the 7589 atomicrmw that is 7590 being released. 7591 7592 2. buffer/global_atomic 7593 3. s_waitcnt vmcnt(0) 7594 7595 - Must happen before 7596 following 7597 buffer_wbinvl1_vol. 7598 - Ensures the 7599 atomicrmw has 7600 completed before 7601 invalidating the 7602 cache. 7603 7604 4. buffer_wbinvl1_vol 7605 7606 - Must happen before 7607 any following 7608 global/generic 7609 load/load 7610 atomic/atomicrmw. 7611 - Ensures that 7612 following loads 7613 will not see stale 7614 global data. 7615 7616 atomicrmw acq_rel - system - global 1. buffer_wbl2 7617 7618 - Must happen before 7619 following s_waitcnt. 7620 - Performs L2 writeback to 7621 ensure previous 7622 global/generic 7623 store/atomicrmw are 7624 visible at system scope. 7625 7626 2. s_waitcnt lgkmcnt(0) & 7627 vmcnt(0) 7628 7629 - If TgSplit execution mode, 7630 omit lgkmcnt(0). 7631 - If OpenCL, omit 7632 lgkmcnt(0). 7633 - Could be split into 7634 separate s_waitcnt 7635 vmcnt(0) and 7636 s_waitcnt 7637 lgkmcnt(0) to allow 7638 them to be 7639 independently moved 7640 according to the 7641 following rules. 7642 - s_waitcnt vmcnt(0) 7643 must happen after 7644 any preceding 7645 global/generic 7646 load/store/load 7647 atomic/store 7648 atomic/atomicrmw. 7649 - s_waitcnt lgkmcnt(0) 7650 must happen after 7651 any preceding 7652 local/generic 7653 load/store/load 7654 atomic/store 7655 atomic/atomicrmw. 7656 - Must happen before 7657 the following 7658 atomicrmw. 7659 - Ensures that all 7660 memory operations 7661 to global and L2 writeback 7662 have completed before 7663 performing the 7664 atomicrmw that is 7665 being released. 7666 7667 3. buffer/global_atomic 7668 scc=1 7669 4. s_waitcnt vmcnt(0) 7670 7671 - Must happen before 7672 following buffer_invl2 and 7673 buffer_wbinvl1_vol. 7674 - Ensures the 7675 atomicrmw has 7676 completed before 7677 invalidating the 7678 caches. 7679 7680 5. buffer_invl2; 7681 buffer_wbinvl1_vol 7682 7683 - Must happen before 7684 any following 7685 global/generic 7686 load/load 7687 atomic/atomicrmw. 7688 - Ensures that 7689 following loads 7690 will not see stale 7691 MTYPE NC global data. 7692 MTYPE RW and CC memory will 7693 never be stale due to the 7694 memory probes. 7695 7696 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7697 vmcnt(0) 7698 7699 - If TgSplit execution mode, 7700 omit lgkmcnt(0). 7701 - If OpenCL, omit 7702 lgkmcnt(0). 7703 - Could be split into 7704 separate s_waitcnt 7705 vmcnt(0) and 7706 s_waitcnt 7707 lgkmcnt(0) to allow 7708 them to be 7709 independently moved 7710 according to the 7711 following rules. 7712 - s_waitcnt vmcnt(0) 7713 must happen after 7714 any preceding 7715 global/generic 7716 load/store/load 7717 atomic/store 7718 atomic/atomicrmw. 7719 - s_waitcnt lgkmcnt(0) 7720 must happen after 7721 any preceding 7722 local/generic 7723 load/store/load 7724 atomic/store 7725 atomic/atomicrmw. 7726 - Must happen before 7727 the following 7728 atomicrmw. 7729 - Ensures that all 7730 memory operations 7731 to global have 7732 completed before 7733 performing the 7734 atomicrmw that is 7735 being released. 7736 7737 2. flat_atomic 7738 3. s_waitcnt vmcnt(0) & 7739 lgkmcnt(0) 7740 7741 - If TgSplit execution mode, 7742 omit lgkmcnt(0). 7743 - If OpenCL, omit 7744 lgkmcnt(0). 7745 - Must happen before 7746 following 7747 buffer_wbinvl1_vol. 7748 - Ensures the 7749 atomicrmw has 7750 completed before 7751 invalidating the 7752 cache. 7753 7754 4. buffer_wbinvl1_vol 7755 7756 - Must happen before 7757 any following 7758 global/generic 7759 load/load 7760 atomic/atomicrmw. 7761 - Ensures that 7762 following loads 7763 will not see stale 7764 global data. 7765 7766 atomicrmw acq_rel - system - generic 1. buffer_wbl2 7767 7768 - Must happen before 7769 following s_waitcnt. 7770 - Performs L2 writeback to 7771 ensure previous 7772 global/generic 7773 store/atomicrmw are 7774 visible at system scope. 7775 7776 2. s_waitcnt lgkmcnt(0) & 7777 vmcnt(0) 7778 7779 - If TgSplit execution mode, 7780 omit lgkmcnt(0). 7781 - If OpenCL, omit 7782 lgkmcnt(0). 7783 - Could be split into 7784 separate s_waitcnt 7785 vmcnt(0) and 7786 s_waitcnt 7787 lgkmcnt(0) to allow 7788 them to be 7789 independently moved 7790 according to the 7791 following rules. 7792 - s_waitcnt vmcnt(0) 7793 must happen after 7794 any preceding 7795 global/generic 7796 load/store/load 7797 atomic/store 7798 atomic/atomicrmw. 7799 - s_waitcnt lgkmcnt(0) 7800 must happen after 7801 any preceding 7802 local/generic 7803 load/store/load 7804 atomic/store 7805 atomic/atomicrmw. 7806 - Must happen before 7807 the following 7808 atomicrmw. 7809 - Ensures that all 7810 memory operations 7811 to global and L2 writeback 7812 have completed before 7813 performing the 7814 atomicrmw that is 7815 being released. 7816 7817 3. flat_atomic scc=1 7818 4. s_waitcnt vmcnt(0) & 7819 lgkmcnt(0) 7820 7821 - If TgSplit execution mode, 7822 omit lgkmcnt(0). 7823 - If OpenCL, omit 7824 lgkmcnt(0). 7825 - Must happen before 7826 following buffer_invl2 and 7827 buffer_wbinvl1_vol. 7828 - Ensures the 7829 atomicrmw has 7830 completed before 7831 invalidating the 7832 caches. 7833 7834 5. buffer_invl2; 7835 buffer_wbinvl1_vol 7836 7837 - Must happen before 7838 any following 7839 global/generic 7840 load/load 7841 atomic/atomicrmw. 7842 - Ensures that 7843 following loads 7844 will not see stale 7845 MTYPE NC global data. 7846 MTYPE RW and CC memory will 7847 never be stale due to the 7848 memory probes. 7849 7850 fence acq_rel - singlethread *none* *none* 7851 - wavefront 7852 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7853 7854 - Use lgkmcnt(0) if not 7855 TgSplit execution mode 7856 and vmcnt(0) if TgSplit 7857 execution mode. 7858 - If OpenCL and 7859 address space is 7860 not generic, omit 7861 lgkmcnt(0). 7862 - If OpenCL and 7863 address space is 7864 local, omit 7865 vmcnt(0). 7866 - However, 7867 since LLVM 7868 currently has no 7869 address space on 7870 the fence need to 7871 conservatively 7872 always generate 7873 (see comment for 7874 previous fence). 7875 - s_waitcnt vmcnt(0) 7876 must happen after 7877 any preceding 7878 global/generic 7879 load/store/ 7880 load atomic/store atomic/ 7881 atomicrmw. 7882 - s_waitcnt lgkmcnt(0) 7883 must happen after 7884 any preceding 7885 local/generic 7886 load/load 7887 atomic/store/store 7888 atomic/atomicrmw. 7889 - Must happen before 7890 any following 7891 global/generic 7892 load/load 7893 atomic/store/store 7894 atomic/atomicrmw. 7895 - Ensures that all 7896 memory operations 7897 have 7898 completed before 7899 performing any 7900 following global 7901 memory operations. 7902 - Ensures that the 7903 preceding 7904 local/generic load 7905 atomic/atomicrmw 7906 with an equal or 7907 wider sync scope 7908 and memory ordering 7909 stronger than 7910 unordered (this is 7911 termed the 7912 acquire-fence-paired-atomic) 7913 has completed 7914 before following 7915 global memory 7916 operations. This 7917 satisfies the 7918 requirements of 7919 acquire. 7920 - Ensures that all 7921 previous memory 7922 operations have 7923 completed before a 7924 following 7925 local/generic store 7926 atomic/atomicrmw 7927 with an equal or 7928 wider sync scope 7929 and memory ordering 7930 stronger than 7931 unordered (this is 7932 termed the 7933 release-fence-paired-atomic). 7934 This satisfies the 7935 requirements of 7936 release. 7937 - Must happen before 7938 the following 7939 buffer_wbinvl1_vol. 7940 - Ensures that the 7941 acquire-fence-paired 7942 atomic has completed 7943 before invalidating 7944 the 7945 cache. Therefore 7946 any following 7947 locations read must 7948 be no older than 7949 the value read by 7950 the 7951 acquire-fence-paired-atomic. 7952 7953 3. buffer_wbinvl1_vol 7954 7955 - If not TgSplit execution 7956 mode, omit. 7957 - Ensures that 7958 following 7959 loads will not see 7960 stale data. 7961 7962 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7963 vmcnt(0) 7964 7965 - If TgSplit execution mode, 7966 omit lgkmcnt(0). 7967 - If OpenCL and 7968 address space is 7969 not generic, omit 7970 lgkmcnt(0). 7971 - However, since LLVM 7972 currently has no 7973 address space on 7974 the fence need to 7975 conservatively 7976 always generate 7977 (see comment for 7978 previous fence). 7979 - Could be split into 7980 separate s_waitcnt 7981 vmcnt(0) and 7982 s_waitcnt 7983 lgkmcnt(0) to allow 7984 them to be 7985 independently moved 7986 according to the 7987 following rules. 7988 - s_waitcnt vmcnt(0) 7989 must happen after 7990 any preceding 7991 global/generic 7992 load/store/load 7993 atomic/store 7994 atomic/atomicrmw. 7995 - s_waitcnt lgkmcnt(0) 7996 must happen after 7997 any preceding 7998 local/generic 7999 load/store/load 8000 atomic/store 8001 atomic/atomicrmw. 8002 - Must happen before 8003 the following 8004 buffer_wbinvl1_vol. 8005 - Ensures that the 8006 preceding 8007 global/local/generic 8008 load 8009 atomic/atomicrmw 8010 with an equal or 8011 wider sync scope 8012 and memory ordering 8013 stronger than 8014 unordered (this is 8015 termed the 8016 acquire-fence-paired-atomic) 8017 has completed 8018 before invalidating 8019 the cache. This 8020 satisfies the 8021 requirements of 8022 acquire. 8023 - Ensures that all 8024 previous memory 8025 operations have 8026 completed before a 8027 following 8028 global/local/generic 8029 store 8030 atomic/atomicrmw 8031 with an equal or 8032 wider sync scope 8033 and memory ordering 8034 stronger than 8035 unordered (this is 8036 termed the 8037 release-fence-paired-atomic). 8038 This satisfies the 8039 requirements of 8040 release. 8041 8042 2. buffer_wbinvl1_vol 8043 8044 - Must happen before 8045 any following 8046 global/generic 8047 load/load 8048 atomic/store/store 8049 atomic/atomicrmw. 8050 - Ensures that 8051 following loads 8052 will not see stale 8053 global data. This 8054 satisfies the 8055 requirements of 8056 acquire. 8057 8058 fence acq_rel - system *none* 1. buffer_wbl2 8059 8060 - If OpenCL and 8061 address space is 8062 local, omit. 8063 - Must happen before 8064 following s_waitcnt. 8065 - Performs L2 writeback to 8066 ensure previous 8067 global/generic 8068 store/atomicrmw are 8069 visible at system scope. 8070 8071 2. s_waitcnt lgkmcnt(0) & 8072 vmcnt(0) 8073 8074 - If TgSplit execution mode, 8075 omit lgkmcnt(0). 8076 - If OpenCL and 8077 address space is 8078 not generic, omit 8079 lgkmcnt(0). 8080 - However, since LLVM 8081 currently has no 8082 address space on 8083 the fence need to 8084 conservatively 8085 always generate 8086 (see comment for 8087 previous fence). 8088 - Could be split into 8089 separate s_waitcnt 8090 vmcnt(0) and 8091 s_waitcnt 8092 lgkmcnt(0) to allow 8093 them to be 8094 independently moved 8095 according to the 8096 following rules. 8097 - s_waitcnt vmcnt(0) 8098 must happen after 8099 any preceding 8100 global/generic 8101 load/store/load 8102 atomic/store 8103 atomic/atomicrmw. 8104 - s_waitcnt lgkmcnt(0) 8105 must happen after 8106 any preceding 8107 local/generic 8108 load/store/load 8109 atomic/store 8110 atomic/atomicrmw. 8111 - Must happen before 8112 the following buffer_invl2 and 8113 buffer_wbinvl1_vol. 8114 - Ensures that the 8115 preceding 8116 global/local/generic 8117 load 8118 atomic/atomicrmw 8119 with an equal or 8120 wider sync scope 8121 and memory ordering 8122 stronger than 8123 unordered (this is 8124 termed the 8125 acquire-fence-paired-atomic) 8126 has completed 8127 before invalidating 8128 the cache. This 8129 satisfies the 8130 requirements of 8131 acquire. 8132 - Ensures that all 8133 previous memory 8134 operations have 8135 completed before a 8136 following 8137 global/local/generic 8138 store 8139 atomic/atomicrmw 8140 with an equal or 8141 wider sync scope 8142 and memory ordering 8143 stronger than 8144 unordered (this is 8145 termed the 8146 release-fence-paired-atomic). 8147 This satisfies the 8148 requirements of 8149 release. 8150 8151 3. buffer_invl2; 8152 buffer_wbinvl1_vol 8153 8154 - Must happen before 8155 any following 8156 global/generic 8157 load/load 8158 atomic/store/store 8159 atomic/atomicrmw. 8160 - Ensures that 8161 following loads 8162 will not see stale 8163 MTYPE NC global data. 8164 MTYPE RW and CC memory will 8165 never be stale due to the 8166 memory probes. 8167 8168 **Sequential Consistent Atomic** 8169 ------------------------------------------------------------------------------------ 8170 load atomic seq_cst - singlethread - global *Same as corresponding 8171 - wavefront - local load atomic acquire, 8172 - generic except must generated 8173 all instructions even 8174 for OpenCL.* 8175 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8176 - generic 8177 - Use lgkmcnt(0) if not 8178 TgSplit execution mode 8179 and vmcnt(0) if TgSplit 8180 execution mode. 8181 - s_waitcnt lgkmcnt(0) must 8182 happen after 8183 preceding 8184 local/generic load 8185 atomic/store 8186 atomic/atomicrmw 8187 with memory 8188 ordering of seq_cst 8189 and with equal or 8190 wider sync scope. 8191 (Note that seq_cst 8192 fences have their 8193 own s_waitcnt 8194 lgkmcnt(0) and so do 8195 not need to be 8196 considered.) 8197 - s_waitcnt vmcnt(0) 8198 must happen after 8199 preceding 8200 global/generic load 8201 atomic/store 8202 atomic/atomicrmw 8203 with memory 8204 ordering of seq_cst 8205 and with equal or 8206 wider sync scope. 8207 (Note that seq_cst 8208 fences have their 8209 own s_waitcnt 8210 vmcnt(0) and so do 8211 not need to be 8212 considered.) 8213 - Ensures any 8214 preceding 8215 sequential 8216 consistent global/local 8217 memory instructions 8218 have completed 8219 before executing 8220 this sequentially 8221 consistent 8222 instruction. This 8223 prevents reordering 8224 a seq_cst store 8225 followed by a 8226 seq_cst load. (Note 8227 that seq_cst is 8228 stronger than 8229 acquire/release as 8230 the reordering of 8231 load acquire 8232 followed by a store 8233 release is 8234 prevented by the 8235 s_waitcnt of 8236 the release, but 8237 there is nothing 8238 preventing a store 8239 release followed by 8240 load acquire from 8241 completing out of 8242 order. The s_waitcnt 8243 could be placed after 8244 seq_store or before 8245 the seq_load. We 8246 choose the load to 8247 make the s_waitcnt be 8248 as late as possible 8249 so that the store 8250 may have already 8251 completed.) 8252 8253 2. *Following 8254 instructions same as 8255 corresponding load 8256 atomic acquire, 8257 except must generated 8258 all instructions even 8259 for OpenCL.* 8260 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8261 local address space cannot 8262 be used.* 8263 8264 *Same as corresponding 8265 load atomic acquire, 8266 except must generated 8267 all instructions even 8268 for OpenCL.* 8269 8270 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8271 - system - generic vmcnt(0) 8272 8273 - If TgSplit execution mode, 8274 omit lgkmcnt(0). 8275 - Could be split into 8276 separate s_waitcnt 8277 vmcnt(0) 8278 and s_waitcnt 8279 lgkmcnt(0) to allow 8280 them to be 8281 independently moved 8282 according to the 8283 following rules. 8284 - s_waitcnt lgkmcnt(0) 8285 must happen after 8286 preceding 8287 global/generic load 8288 atomic/store 8289 atomic/atomicrmw 8290 with memory 8291 ordering of seq_cst 8292 and with equal or 8293 wider sync scope. 8294 (Note that seq_cst 8295 fences have their 8296 own s_waitcnt 8297 lgkmcnt(0) and so do 8298 not need to be 8299 considered.) 8300 - s_waitcnt vmcnt(0) 8301 must happen after 8302 preceding 8303 global/generic load 8304 atomic/store 8305 atomic/atomicrmw 8306 with memory 8307 ordering of seq_cst 8308 and with equal or 8309 wider sync scope. 8310 (Note that seq_cst 8311 fences have their 8312 own s_waitcnt 8313 vmcnt(0) and so do 8314 not need to be 8315 considered.) 8316 - Ensures any 8317 preceding 8318 sequential 8319 consistent global 8320 memory instructions 8321 have completed 8322 before executing 8323 this sequentially 8324 consistent 8325 instruction. This 8326 prevents reordering 8327 a seq_cst store 8328 followed by a 8329 seq_cst load. (Note 8330 that seq_cst is 8331 stronger than 8332 acquire/release as 8333 the reordering of 8334 load acquire 8335 followed by a store 8336 release is 8337 prevented by the 8338 s_waitcnt of 8339 the release, but 8340 there is nothing 8341 preventing a store 8342 release followed by 8343 load acquire from 8344 completing out of 8345 order. The s_waitcnt 8346 could be placed after 8347 seq_store or before 8348 the seq_load. We 8349 choose the load to 8350 make the s_waitcnt be 8351 as late as possible 8352 so that the store 8353 may have already 8354 completed.) 8355 8356 2. *Following 8357 instructions same as 8358 corresponding load 8359 atomic acquire, 8360 except must generated 8361 all instructions even 8362 for OpenCL.* 8363 store atomic seq_cst - singlethread - global *Same as corresponding 8364 - wavefront - local store atomic release, 8365 - workgroup - generic except must generated 8366 - agent all instructions even 8367 - system for OpenCL.* 8368 atomicrmw seq_cst - singlethread - global *Same as corresponding 8369 - wavefront - local atomicrmw acq_rel, 8370 - workgroup - generic except must generated 8371 - agent all instructions even 8372 - system for OpenCL.* 8373 fence seq_cst - singlethread *none* *Same as corresponding 8374 - wavefront fence acq_rel, 8375 - workgroup except must generated 8376 - agent all instructions even 8377 - system for OpenCL.* 8378 ============ ============ ============== ========== ================================ 8379 8380.. _amdgpu-amdhsa-memory-model-gfx10: 8381 8382Memory Model GFX10 8383++++++++++++++++++ 8384 8385For GFX10: 8386 8387* Each agent has multiple shader arrays (SA). 8388* Each SA has multiple work-group processors (WGP). 8389* Each WGP has multiple compute units (CU). 8390* Each CU has multiple SIMDs that execute wavefronts. 8391* The wavefronts for a single work-group are executed in the same 8392 WGP. In CU wavefront execution mode the wavefronts may be executed by 8393 different SIMDs in the same CU. In WGP wavefront execution mode the 8394 wavefronts may be executed by different SIMDs in different CUs in the same 8395 WGP. 8396* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8397 executing on it. 8398* All LDS operations of a WGP are performed as wavefront wide operations in a 8399 global order and involve no caching. Completion is reported to a wavefront in 8400 execution order. 8401* The LDS memory has multiple request queues shared by the SIMDs of a 8402 WGP. Therefore, the LDS operations performed by different wavefronts of a 8403 work-group can be reordered relative to each other, which can result in 8404 reordering the visibility of vector memory operations with respect to LDS 8405 operations of other wavefronts in the same work-group. A ``s_waitcnt 8406 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8407 vector memory operations between wavefronts of a work-group, but not between 8408 operations performed by the same wavefront. 8409* The vector memory operations are performed as wavefront wide operations. 8410 Completion of load/store/sample operations are reported to a wavefront in 8411 execution order of other load/store/sample operations performed by that 8412 wavefront. 8413* The vector memory operations access a vector L0 cache. There is a single L0 8414 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8415 special action is required for coherence between the lanes of a single 8416 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8417 wavefronts executing in the same work-group as they may be executing on SIMDs 8418 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8419 required for coherence between wavefronts executing in different work-groups 8420 as they may be executing on different WGPs. 8421* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8422 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8423 operations are used in a restricted way so do not impact the memory model. See 8424 :ref:`amdgpu-amdhsa-memory-spaces`. 8425* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8426 the same SA. Therefore, no special action is required for coherence between 8427 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8428 required for coherence between wavefronts executing in different work-groups 8429 as they may be executing on different SAs that access different L1s. 8430* The L1 caches have independent quadrants to service disjoint ranges of virtual 8431 addresses. 8432* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8433 vector and scalar memory operations performed by different wavefronts, whether 8434 executing in the same or different work-groups (which may be executing on 8435 different CUs accessing different L0s), can be reordered relative to each 8436 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8437 synchronization between vector memory operations of different wavefronts. It 8438 ensures a previous vector memory operation has completed before executing a 8439 subsequent vector memory or LDS operation and so can be used to meet the 8440 requirements of acquire, release and sequential consistency. 8441* The L1 caches use an L2 cache shared by all SAs on the same agent. 8442* The L2 cache has independent channels to service disjoint ranges of virtual 8443 addresses. 8444* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8445 quadrant has a separate request queue per L2 channel. Therefore, the vector 8446 and scalar memory operations performed by wavefronts executing in different 8447 work-groups (which may be executing on different SAs) of an agent can be 8448 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8449 required to ensure synchronization between vector memory operations of 8450 different SAs. It ensures a previous vector memory operation has completed 8451 before executing a subsequent vector memory and so can be used to meet the 8452 requirements of acquire, release and sequential consistency. 8453* The L2 cache can be kept coherent with other agents on some targets, or ranges 8454 of virtual addresses can be set up to bypass it to ensure system coherence. 8455 8456Scalar memory operations are only used to access memory that is proven to not 8457change during the execution of the kernel dispatch. This includes constant 8458address space and global address space for program scope ``const`` variables. 8459Therefore, the kernel machine code does not have to maintain the scalar cache to 8460ensure it is coherent with the vector caches. The scalar and vector caches are 8461invalidated between kernel dispatches by CP since constant address space data 8462may change between kernel dispatch executions. See 8463:ref:`amdgpu-amdhsa-memory-spaces`. 8464 8465The one exception is if scalar writes are used to spill SGPR registers. In this 8466case the AMDGPU backend ensures the memory location used to spill is never 8467accessed by vector memory operations at the same time. If scalar writes are used 8468then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8469return since the locations may be used for vector memory instructions by a 8470future wavefront that uses the same scratch area, or a function call that 8471creates a frame at the same address, respectively. There is no need for a 8472``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8473 8474For kernarg backing memory: 8475 8476* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8477* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8478 needing to invalidate the L2 cache. 8479* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8480 so the L2 cache will be coherent with the CPU and other agents. 8481 8482Scratch backing memory (which is used for the private address space) is accessed 8483with MTYPE NC (non-coherent). Since the private address space is only accessed 8484by a single thread, and is always write-before-read, there is never a need to 8485invalidate these entries from the L0 or L1 caches. 8486 8487Wavefronts are executed in native mode with in-order reporting of loads and 8488sample instructions. In this mode vmcnt reports completion of load, atomic with 8489return and sample instructions in order, and the vscnt reports the completion of 8490store and atomic without return in order. See ``MEM_ORDERED`` field in 8491:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8492 8493Wavefronts can be executed in WGP or CU wavefront execution mode: 8494 8495* In WGP wavefront execution mode the wavefronts of a work-group are executed 8496 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8497 CU L0 caches is required for work-group synchronization. Also accesses to L1 8498 at work-group scope need to be explicitly ordered as the accesses from 8499 different CUs are not ordered. 8500* In CU wavefront execution mode the wavefronts of a work-group are executed on 8501 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8502 the work-group access the same L0 which in turn ensures L1 accesses are 8503 ordered and so do not require explicit management of the caches for 8504 work-group synchronization. 8505 8506See ``WGP_MODE`` field in 8507:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8508:ref:`amdgpu-target-features`. 8509 8510The code sequences used to implement the memory model for GFX10 are defined in 8511table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8512 8513 .. table:: AMDHSA Memory Model Code Sequences GFX10 8514 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8515 8516 ============ ============ ============== ========== ================================ 8517 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8518 Ordering Sync Scope Address GFX10 8519 Space 8520 ============ ============ ============== ========== ================================ 8521 **Non-Atomic** 8522 ------------------------------------------------------------------------------------ 8523 load *none* *none* - global - !volatile & !nontemporal 8524 - generic 8525 - private 1. buffer/global/flat_load 8526 - constant 8527 - !volatile & nontemporal 8528 8529 1. buffer/global/flat_load 8530 slc=1 8531 8532 - volatile 8533 8534 1. buffer/global/flat_load 8535 glc=1 dlc=1 8536 2. s_waitcnt vmcnt(0) 8537 8538 - Must happen before 8539 any following volatile 8540 global/generic 8541 load/store. 8542 - Ensures that 8543 volatile 8544 operations to 8545 different 8546 addresses will not 8547 be reordered by 8548 hardware. 8549 8550 load *none* *none* - local 1. ds_load 8551 store *none* *none* - global - !volatile & !nontemporal 8552 - generic 8553 - private 1. buffer/global/flat_store 8554 - constant 8555 - !volatile & nontemporal 8556 8557 1. buffer/global/flat_store 8558 slc=1 8559 8560 - volatile 8561 8562 1. buffer/global/flat_store 8563 2. s_waitcnt vscnt(0) 8564 8565 - Must happen before 8566 any following volatile 8567 global/generic 8568 load/store. 8569 - Ensures that 8570 volatile 8571 operations to 8572 different 8573 addresses will not 8574 be reordered by 8575 hardware. 8576 8577 store *none* *none* - local 1. ds_store 8578 **Unordered Atomic** 8579 ------------------------------------------------------------------------------------ 8580 load atomic unordered *any* *any* *Same as non-atomic*. 8581 store atomic unordered *any* *any* *Same as non-atomic*. 8582 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8583 **Monotonic Atomic** 8584 ------------------------------------------------------------------------------------ 8585 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8586 - wavefront - generic 8587 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8588 - generic glc=1 8589 8590 - If CU wavefront execution 8591 mode, omit glc=1. 8592 8593 load atomic monotonic - singlethread - local 1. ds_load 8594 - wavefront 8595 - workgroup 8596 load atomic monotonic - agent - global 1. buffer/global/flat_load 8597 - system - generic glc=1 dlc=1 8598 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8599 - wavefront - generic 8600 - workgroup 8601 - agent 8602 - system 8603 store atomic monotonic - singlethread - local 1. ds_store 8604 - wavefront 8605 - workgroup 8606 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8607 - wavefront - generic 8608 - workgroup 8609 - agent 8610 - system 8611 atomicrmw monotonic - singlethread - local 1. ds_atomic 8612 - wavefront 8613 - workgroup 8614 **Acquire Atomic** 8615 ------------------------------------------------------------------------------------ 8616 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8617 - wavefront - local 8618 - generic 8619 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8620 8621 - If CU wavefront execution 8622 mode, omit glc=1. 8623 8624 2. s_waitcnt vmcnt(0) 8625 8626 - If CU wavefront execution 8627 mode, omit. 8628 - Must happen before 8629 the following buffer_gl0_inv 8630 and before any following 8631 global/generic 8632 load/load 8633 atomic/store/store 8634 atomic/atomicrmw. 8635 8636 3. buffer_gl0_inv 8637 8638 - If CU wavefront execution 8639 mode, omit. 8640 - Ensures that 8641 following 8642 loads will not see 8643 stale data. 8644 8645 load atomic acquire - workgroup - local 1. ds_load 8646 2. s_waitcnt lgkmcnt(0) 8647 8648 - If OpenCL, omit. 8649 - Must happen before 8650 the following buffer_gl0_inv 8651 and before any following 8652 global/generic load/load 8653 atomic/store/store 8654 atomic/atomicrmw. 8655 - Ensures any 8656 following global 8657 data read is no 8658 older than the local load 8659 atomic value being 8660 acquired. 8661 8662 3. buffer_gl0_inv 8663 8664 - If CU wavefront execution 8665 mode, omit. 8666 - If OpenCL, omit. 8667 - Ensures that 8668 following 8669 loads will not see 8670 stale data. 8671 8672 load atomic acquire - workgroup - generic 1. flat_load glc=1 8673 8674 - If CU wavefront execution 8675 mode, omit glc=1. 8676 8677 2. s_waitcnt lgkmcnt(0) & 8678 vmcnt(0) 8679 8680 - If CU wavefront execution 8681 mode, omit vmcnt(0). 8682 - If OpenCL, omit 8683 lgkmcnt(0). 8684 - Must happen before 8685 the following 8686 buffer_gl0_inv and any 8687 following global/generic 8688 load/load 8689 atomic/store/store 8690 atomic/atomicrmw. 8691 - Ensures any 8692 following global 8693 data read is no 8694 older than a local load 8695 atomic value being 8696 acquired. 8697 8698 3. buffer_gl0_inv 8699 8700 - If CU wavefront execution 8701 mode, omit. 8702 - Ensures that 8703 following 8704 loads will not see 8705 stale data. 8706 8707 load atomic acquire - agent - global 1. buffer/global_load 8708 - system glc=1 dlc=1 8709 2. s_waitcnt vmcnt(0) 8710 8711 - Must happen before 8712 following 8713 buffer_gl*_inv. 8714 - Ensures the load 8715 has completed 8716 before invalidating 8717 the caches. 8718 8719 3. buffer_gl0_inv; 8720 buffer_gl1_inv 8721 8722 - Must happen before 8723 any following 8724 global/generic 8725 load/load 8726 atomic/atomicrmw. 8727 - Ensures that 8728 following 8729 loads will not see 8730 stale global data. 8731 8732 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8733 - system 2. s_waitcnt vmcnt(0) & 8734 lgkmcnt(0) 8735 8736 - If OpenCL omit 8737 lgkmcnt(0). 8738 - Must happen before 8739 following 8740 buffer_gl*_invl. 8741 - Ensures the flat_load 8742 has completed 8743 before invalidating 8744 the caches. 8745 8746 3. buffer_gl0_inv; 8747 buffer_gl1_inv 8748 8749 - Must happen before 8750 any following 8751 global/generic 8752 load/load 8753 atomic/atomicrmw. 8754 - Ensures that 8755 following loads 8756 will not see stale 8757 global data. 8758 8759 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8760 - wavefront - local 8761 - generic 8762 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8763 2. s_waitcnt vm/vscnt(0) 8764 8765 - If CU wavefront execution 8766 mode, omit. 8767 - Use vmcnt(0) if atomic with 8768 return and vscnt(0) if 8769 atomic with no-return. 8770 - Must happen before 8771 the following buffer_gl0_inv 8772 and before any following 8773 global/generic 8774 load/load 8775 atomic/store/store 8776 atomic/atomicrmw. 8777 8778 3. buffer_gl0_inv 8779 8780 - If CU wavefront execution 8781 mode, omit. 8782 - Ensures that 8783 following 8784 loads will not see 8785 stale data. 8786 8787 atomicrmw acquire - workgroup - local 1. ds_atomic 8788 2. s_waitcnt lgkmcnt(0) 8789 8790 - If OpenCL, omit. 8791 - Must happen before 8792 the following 8793 buffer_gl0_inv. 8794 - Ensures any 8795 following global 8796 data read is no 8797 older than the local 8798 atomicrmw value 8799 being acquired. 8800 8801 3. buffer_gl0_inv 8802 8803 - If OpenCL omit. 8804 - Ensures that 8805 following 8806 loads will not see 8807 stale data. 8808 8809 atomicrmw acquire - workgroup - generic 1. flat_atomic 8810 2. s_waitcnt lgkmcnt(0) & 8811 vm/vscnt(0) 8812 8813 - If CU wavefront execution 8814 mode, omit vm/vscnt(0). 8815 - If OpenCL, omit lgkmcnt(0). 8816 - Use vmcnt(0) if atomic with 8817 return and vscnt(0) if 8818 atomic with no-return. 8819 - Must happen before 8820 the following 8821 buffer_gl0_inv. 8822 - Ensures any 8823 following global 8824 data read is no 8825 older than a local 8826 atomicrmw value 8827 being acquired. 8828 8829 3. buffer_gl0_inv 8830 8831 - If CU wavefront execution 8832 mode, omit. 8833 - Ensures that 8834 following 8835 loads will not see 8836 stale data. 8837 8838 atomicrmw acquire - agent - global 1. buffer/global_atomic 8839 - system 2. s_waitcnt vm/vscnt(0) 8840 8841 - Use vmcnt(0) if atomic with 8842 return and vscnt(0) if 8843 atomic with no-return. 8844 - Must happen before 8845 following 8846 buffer_gl*_inv. 8847 - Ensures the 8848 atomicrmw has 8849 completed before 8850 invalidating the 8851 caches. 8852 8853 3. buffer_gl0_inv; 8854 buffer_gl1_inv 8855 8856 - Must happen before 8857 any following 8858 global/generic 8859 load/load 8860 atomic/atomicrmw. 8861 - Ensures that 8862 following loads 8863 will not see stale 8864 global data. 8865 8866 atomicrmw acquire - agent - generic 1. flat_atomic 8867 - system 2. s_waitcnt vm/vscnt(0) & 8868 lgkmcnt(0) 8869 8870 - If OpenCL, omit 8871 lgkmcnt(0). 8872 - Use vmcnt(0) if atomic with 8873 return and vscnt(0) if 8874 atomic with no-return. 8875 - Must happen before 8876 following 8877 buffer_gl*_inv. 8878 - Ensures the 8879 atomicrmw has 8880 completed before 8881 invalidating the 8882 caches. 8883 8884 3. buffer_gl0_inv; 8885 buffer_gl1_inv 8886 8887 - Must happen before 8888 any following 8889 global/generic 8890 load/load 8891 atomic/atomicrmw. 8892 - Ensures that 8893 following loads 8894 will not see stale 8895 global data. 8896 8897 fence acquire - singlethread *none* *none* 8898 - wavefront 8899 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 8900 vmcnt(0) & vscnt(0) 8901 8902 - If CU wavefront execution 8903 mode, omit vmcnt(0) and 8904 vscnt(0). 8905 - If OpenCL and 8906 address space is 8907 not generic, omit 8908 lgkmcnt(0). 8909 - If OpenCL and 8910 address space is 8911 local, omit 8912 vmcnt(0) and vscnt(0). 8913 - However, since LLVM 8914 currently has no 8915 address space on 8916 the fence need to 8917 conservatively 8918 always generate. If 8919 fence had an 8920 address space then 8921 set to address 8922 space of OpenCL 8923 fence flag, or to 8924 generic if both 8925 local and global 8926 flags are 8927 specified. 8928 - Could be split into 8929 separate s_waitcnt 8930 vmcnt(0), s_waitcnt 8931 vscnt(0) and s_waitcnt 8932 lgkmcnt(0) to allow 8933 them to be 8934 independently moved 8935 according to the 8936 following rules. 8937 - s_waitcnt vmcnt(0) 8938 must happen after 8939 any preceding 8940 global/generic load 8941 atomic/ 8942 atomicrmw-with-return-value 8943 with an equal or 8944 wider sync scope 8945 and memory ordering 8946 stronger than 8947 unordered (this is 8948 termed the 8949 fence-paired-atomic). 8950 - s_waitcnt vscnt(0) 8951 must happen after 8952 any preceding 8953 global/generic 8954 atomicrmw-no-return-value 8955 with an equal or 8956 wider sync scope 8957 and memory ordering 8958 stronger than 8959 unordered (this is 8960 termed the 8961 fence-paired-atomic). 8962 - s_waitcnt lgkmcnt(0) 8963 must happen after 8964 any preceding 8965 local/generic load 8966 atomic/atomicrmw 8967 with an equal or 8968 wider sync scope 8969 and memory ordering 8970 stronger than 8971 unordered (this is 8972 termed the 8973 fence-paired-atomic). 8974 - Must happen before 8975 the following 8976 buffer_gl0_inv. 8977 - Ensures that the 8978 fence-paired atomic 8979 has completed 8980 before invalidating 8981 the 8982 cache. Therefore 8983 any following 8984 locations read must 8985 be no older than 8986 the value read by 8987 the 8988 fence-paired-atomic. 8989 8990 3. buffer_gl0_inv 8991 8992 - If CU wavefront execution 8993 mode, omit. 8994 - Ensures that 8995 following 8996 loads will not see 8997 stale data. 8998 8999 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9000 - system vmcnt(0) & vscnt(0) 9001 9002 - If OpenCL and 9003 address space is 9004 not generic, omit 9005 lgkmcnt(0). 9006 - If OpenCL and 9007 address space is 9008 local, omit 9009 vmcnt(0) and vscnt(0). 9010 - However, since LLVM 9011 currently has no 9012 address space on 9013 the fence need to 9014 conservatively 9015 always generate 9016 (see comment for 9017 previous fence). 9018 - Could be split into 9019 separate s_waitcnt 9020 vmcnt(0), s_waitcnt 9021 vscnt(0) and s_waitcnt 9022 lgkmcnt(0) to allow 9023 them to be 9024 independently moved 9025 according to the 9026 following rules. 9027 - s_waitcnt vmcnt(0) 9028 must happen after 9029 any preceding 9030 global/generic load 9031 atomic/ 9032 atomicrmw-with-return-value 9033 with an equal or 9034 wider sync scope 9035 and memory ordering 9036 stronger than 9037 unordered (this is 9038 termed the 9039 fence-paired-atomic). 9040 - s_waitcnt vscnt(0) 9041 must happen after 9042 any preceding 9043 global/generic 9044 atomicrmw-no-return-value 9045 with an equal or 9046 wider sync scope 9047 and memory ordering 9048 stronger than 9049 unordered (this is 9050 termed the 9051 fence-paired-atomic). 9052 - s_waitcnt lgkmcnt(0) 9053 must happen after 9054 any preceding 9055 local/generic load 9056 atomic/atomicrmw 9057 with an equal or 9058 wider sync scope 9059 and memory ordering 9060 stronger than 9061 unordered (this is 9062 termed the 9063 fence-paired-atomic). 9064 - Must happen before 9065 the following 9066 buffer_gl*_inv. 9067 - Ensures that the 9068 fence-paired atomic 9069 has completed 9070 before invalidating 9071 the 9072 caches. Therefore 9073 any following 9074 locations read must 9075 be no older than 9076 the value read by 9077 the 9078 fence-paired-atomic. 9079 9080 2. buffer_gl0_inv; 9081 buffer_gl1_inv 9082 9083 - Must happen before any 9084 following global/generic 9085 load/load 9086 atomic/store/store 9087 atomic/atomicrmw. 9088 - Ensures that 9089 following loads 9090 will not see stale 9091 global data. 9092 9093 **Release Atomic** 9094 ------------------------------------------------------------------------------------ 9095 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9096 - wavefront - local 9097 - generic 9098 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9099 - generic vmcnt(0) & vscnt(0) 9100 9101 - If CU wavefront execution 9102 mode, omit vmcnt(0) and 9103 vscnt(0). 9104 - If OpenCL, omit 9105 lgkmcnt(0). 9106 - Could be split into 9107 separate s_waitcnt 9108 vmcnt(0), s_waitcnt 9109 vscnt(0) and s_waitcnt 9110 lgkmcnt(0) to allow 9111 them to be 9112 independently moved 9113 according to the 9114 following rules. 9115 - s_waitcnt vmcnt(0) 9116 must happen after 9117 any preceding 9118 global/generic load/load 9119 atomic/ 9120 atomicrmw-with-return-value. 9121 - s_waitcnt vscnt(0) 9122 must happen after 9123 any preceding 9124 global/generic 9125 store/store 9126 atomic/ 9127 atomicrmw-no-return-value. 9128 - s_waitcnt lgkmcnt(0) 9129 must happen after 9130 any preceding 9131 local/generic 9132 load/store/load 9133 atomic/store 9134 atomic/atomicrmw. 9135 - Must happen before 9136 the following 9137 store. 9138 - Ensures that all 9139 memory operations 9140 have 9141 completed before 9142 performing the 9143 store that is being 9144 released. 9145 9146 2. buffer/global/flat_store 9147 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9148 9149 - If CU wavefront execution 9150 mode, omit. 9151 - If OpenCL, omit. 9152 - Could be split into 9153 separate s_waitcnt 9154 vmcnt(0) and s_waitcnt 9155 vscnt(0) to allow 9156 them to be 9157 independently moved 9158 according to the 9159 following rules. 9160 - s_waitcnt vmcnt(0) 9161 must happen after 9162 any preceding 9163 global/generic load/load 9164 atomic/ 9165 atomicrmw-with-return-value. 9166 - s_waitcnt vscnt(0) 9167 must happen after 9168 any preceding 9169 global/generic 9170 store/store atomic/ 9171 atomicrmw-no-return-value. 9172 - Must happen before 9173 the following 9174 store. 9175 - Ensures that all 9176 global memory 9177 operations have 9178 completed before 9179 performing the 9180 store that is being 9181 released. 9182 9183 2. ds_store 9184 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9185 - system - generic vmcnt(0) & vscnt(0) 9186 9187 - If OpenCL and 9188 address space is 9189 not generic, omit 9190 lgkmcnt(0). 9191 - Could be split into 9192 separate s_waitcnt 9193 vmcnt(0), s_waitcnt vscnt(0) 9194 and s_waitcnt 9195 lgkmcnt(0) to allow 9196 them to be 9197 independently moved 9198 according to the 9199 following rules. 9200 - s_waitcnt vmcnt(0) 9201 must happen after 9202 any preceding 9203 global/generic 9204 load/load 9205 atomic/ 9206 atomicrmw-with-return-value. 9207 - s_waitcnt vscnt(0) 9208 must happen after 9209 any preceding 9210 global/generic 9211 store/store atomic/ 9212 atomicrmw-no-return-value. 9213 - s_waitcnt lgkmcnt(0) 9214 must happen after 9215 any preceding 9216 local/generic 9217 load/store/load 9218 atomic/store 9219 atomic/atomicrmw. 9220 - Must happen before 9221 the following 9222 store. 9223 - Ensures that all 9224 memory operations 9225 have 9226 completed before 9227 performing the 9228 store that is being 9229 released. 9230 9231 2. buffer/global/flat_store 9232 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9233 - wavefront - local 9234 - generic 9235 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9236 - generic vmcnt(0) & vscnt(0) 9237 9238 - If CU wavefront execution 9239 mode, omit vmcnt(0) and 9240 vscnt(0). 9241 - If OpenCL, omit lgkmcnt(0). 9242 - Could be split into 9243 separate s_waitcnt 9244 vmcnt(0), s_waitcnt 9245 vscnt(0) and s_waitcnt 9246 lgkmcnt(0) to allow 9247 them to be 9248 independently moved 9249 according to the 9250 following rules. 9251 - s_waitcnt vmcnt(0) 9252 must happen after 9253 any preceding 9254 global/generic load/load 9255 atomic/ 9256 atomicrmw-with-return-value. 9257 - s_waitcnt vscnt(0) 9258 must happen after 9259 any preceding 9260 global/generic 9261 store/store 9262 atomic/ 9263 atomicrmw-no-return-value. 9264 - s_waitcnt lgkmcnt(0) 9265 must happen after 9266 any preceding 9267 local/generic 9268 load/store/load 9269 atomic/store 9270 atomic/atomicrmw. 9271 - Must happen before 9272 the following 9273 atomicrmw. 9274 - Ensures that all 9275 memory operations 9276 have 9277 completed before 9278 performing the 9279 atomicrmw that is 9280 being released. 9281 9282 2. buffer/global/flat_atomic 9283 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9284 9285 - If CU wavefront execution 9286 mode, omit. 9287 - If OpenCL, omit. 9288 - Could be split into 9289 separate s_waitcnt 9290 vmcnt(0) and s_waitcnt 9291 vscnt(0) to allow 9292 them to be 9293 independently moved 9294 according to the 9295 following rules. 9296 - s_waitcnt vmcnt(0) 9297 must happen after 9298 any preceding 9299 global/generic load/load 9300 atomic/ 9301 atomicrmw-with-return-value. 9302 - s_waitcnt vscnt(0) 9303 must happen after 9304 any preceding 9305 global/generic 9306 store/store atomic/ 9307 atomicrmw-no-return-value. 9308 - Must happen before 9309 the following 9310 store. 9311 - Ensures that all 9312 global memory 9313 operations have 9314 completed before 9315 performing the 9316 store that is being 9317 released. 9318 9319 2. ds_atomic 9320 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9321 - system - generic vmcnt(0) & vscnt(0) 9322 9323 - If OpenCL, omit 9324 lgkmcnt(0). 9325 - Could be split into 9326 separate s_waitcnt 9327 vmcnt(0), s_waitcnt 9328 vscnt(0) and s_waitcnt 9329 lgkmcnt(0) to allow 9330 them to be 9331 independently moved 9332 according to the 9333 following rules. 9334 - s_waitcnt vmcnt(0) 9335 must happen after 9336 any preceding 9337 global/generic 9338 load/load atomic/ 9339 atomicrmw-with-return-value. 9340 - s_waitcnt vscnt(0) 9341 must happen after 9342 any preceding 9343 global/generic 9344 store/store atomic/ 9345 atomicrmw-no-return-value. 9346 - s_waitcnt lgkmcnt(0) 9347 must happen after 9348 any preceding 9349 local/generic 9350 load/store/load 9351 atomic/store 9352 atomic/atomicrmw. 9353 - Must happen before 9354 the following 9355 atomicrmw. 9356 - Ensures that all 9357 memory operations 9358 to global and local 9359 have completed 9360 before performing 9361 the atomicrmw that 9362 is being released. 9363 9364 2. buffer/global/flat_atomic 9365 fence release - singlethread *none* *none* 9366 - wavefront 9367 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9368 vmcnt(0) & vscnt(0) 9369 9370 - If CU wavefront execution 9371 mode, omit vmcnt(0) and 9372 vscnt(0). 9373 - If OpenCL and 9374 address space is 9375 not generic, omit 9376 lgkmcnt(0). 9377 - If OpenCL and 9378 address space is 9379 local, omit 9380 vmcnt(0) and vscnt(0). 9381 - However, since LLVM 9382 currently has no 9383 address space on 9384 the fence need to 9385 conservatively 9386 always generate. If 9387 fence had an 9388 address space then 9389 set to address 9390 space of OpenCL 9391 fence flag, or to 9392 generic if both 9393 local and global 9394 flags are 9395 specified. 9396 - Could be split into 9397 separate s_waitcnt 9398 vmcnt(0), s_waitcnt 9399 vscnt(0) and s_waitcnt 9400 lgkmcnt(0) to allow 9401 them to be 9402 independently moved 9403 according to the 9404 following rules. 9405 - s_waitcnt vmcnt(0) 9406 must happen after 9407 any preceding 9408 global/generic 9409 load/load 9410 atomic/ 9411 atomicrmw-with-return-value. 9412 - s_waitcnt vscnt(0) 9413 must happen after 9414 any preceding 9415 global/generic 9416 store/store atomic/ 9417 atomicrmw-no-return-value. 9418 - s_waitcnt lgkmcnt(0) 9419 must happen after 9420 any preceding 9421 local/generic 9422 load/store/load 9423 atomic/store atomic/ 9424 atomicrmw. 9425 - Must happen before 9426 any following store 9427 atomic/atomicrmw 9428 with an equal or 9429 wider sync scope 9430 and memory ordering 9431 stronger than 9432 unordered (this is 9433 termed the 9434 fence-paired-atomic). 9435 - Ensures that all 9436 memory operations 9437 have 9438 completed before 9439 performing the 9440 following 9441 fence-paired-atomic. 9442 9443 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9444 - system vmcnt(0) & vscnt(0) 9445 9446 - If OpenCL and 9447 address space is 9448 not generic, omit 9449 lgkmcnt(0). 9450 - If OpenCL and 9451 address space is 9452 local, omit 9453 vmcnt(0) and vscnt(0). 9454 - However, since LLVM 9455 currently has no 9456 address space on 9457 the fence need to 9458 conservatively 9459 always generate. If 9460 fence had an 9461 address space then 9462 set to address 9463 space of OpenCL 9464 fence flag, or to 9465 generic if both 9466 local and global 9467 flags are 9468 specified. 9469 - Could be split into 9470 separate s_waitcnt 9471 vmcnt(0), s_waitcnt 9472 vscnt(0) and s_waitcnt 9473 lgkmcnt(0) to allow 9474 them to be 9475 independently moved 9476 according to the 9477 following rules. 9478 - s_waitcnt vmcnt(0) 9479 must happen after 9480 any preceding 9481 global/generic 9482 load/load atomic/ 9483 atomicrmw-with-return-value. 9484 - s_waitcnt vscnt(0) 9485 must happen after 9486 any preceding 9487 global/generic 9488 store/store atomic/ 9489 atomicrmw-no-return-value. 9490 - s_waitcnt lgkmcnt(0) 9491 must happen after 9492 any preceding 9493 local/generic 9494 load/store/load 9495 atomic/store 9496 atomic/atomicrmw. 9497 - Must happen before 9498 any following store 9499 atomic/atomicrmw 9500 with an equal or 9501 wider sync scope 9502 and memory ordering 9503 stronger than 9504 unordered (this is 9505 termed the 9506 fence-paired-atomic). 9507 - Ensures that all 9508 memory operations 9509 have 9510 completed before 9511 performing the 9512 following 9513 fence-paired-atomic. 9514 9515 **Acquire-Release Atomic** 9516 ------------------------------------------------------------------------------------ 9517 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9518 - wavefront - local 9519 - generic 9520 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9521 vmcnt(0) & vscnt(0) 9522 9523 - If CU wavefront execution 9524 mode, omit vmcnt(0) and 9525 vscnt(0). 9526 - If OpenCL, omit 9527 lgkmcnt(0). 9528 - Must happen after 9529 any preceding 9530 local/generic 9531 load/store/load 9532 atomic/store 9533 atomic/atomicrmw. 9534 - Could be split into 9535 separate s_waitcnt 9536 vmcnt(0), s_waitcnt 9537 vscnt(0), and s_waitcnt 9538 lgkmcnt(0) to allow 9539 them to be 9540 independently moved 9541 according to the 9542 following rules. 9543 - s_waitcnt vmcnt(0) 9544 must happen after 9545 any preceding 9546 global/generic load/load 9547 atomic/ 9548 atomicrmw-with-return-value. 9549 - s_waitcnt vscnt(0) 9550 must happen after 9551 any preceding 9552 global/generic 9553 store/store 9554 atomic/ 9555 atomicrmw-no-return-value. 9556 - s_waitcnt lgkmcnt(0) 9557 must happen after 9558 any preceding 9559 local/generic 9560 load/store/load 9561 atomic/store 9562 atomic/atomicrmw. 9563 - Must happen before 9564 the following 9565 atomicrmw. 9566 - Ensures that all 9567 memory operations 9568 have 9569 completed before 9570 performing the 9571 atomicrmw that is 9572 being released. 9573 9574 2. buffer/global_atomic 9575 3. s_waitcnt vm/vscnt(0) 9576 9577 - If CU wavefront execution 9578 mode, omit. 9579 - Use vmcnt(0) if atomic with 9580 return and vscnt(0) if 9581 atomic with no-return. 9582 - Must happen before 9583 the following 9584 buffer_gl0_inv. 9585 - Ensures any 9586 following global 9587 data read is no 9588 older than the 9589 atomicrmw value 9590 being acquired. 9591 9592 4. buffer_gl0_inv 9593 9594 - If CU wavefront execution 9595 mode, omit. 9596 - Ensures that 9597 following 9598 loads will not see 9599 stale data. 9600 9601 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9602 9603 - If CU wavefront execution 9604 mode, omit. 9605 - If OpenCL, omit. 9606 - Could be split into 9607 separate s_waitcnt 9608 vmcnt(0) and s_waitcnt 9609 vscnt(0) to allow 9610 them to be 9611 independently moved 9612 according to the 9613 following rules. 9614 - s_waitcnt vmcnt(0) 9615 must happen after 9616 any preceding 9617 global/generic load/load 9618 atomic/ 9619 atomicrmw-with-return-value. 9620 - s_waitcnt vscnt(0) 9621 must happen after 9622 any preceding 9623 global/generic 9624 store/store atomic/ 9625 atomicrmw-no-return-value. 9626 - Must happen before 9627 the following 9628 store. 9629 - Ensures that all 9630 global memory 9631 operations have 9632 completed before 9633 performing the 9634 store that is being 9635 released. 9636 9637 2. ds_atomic 9638 3. s_waitcnt lgkmcnt(0) 9639 9640 - If OpenCL, omit. 9641 - Must happen before 9642 the following 9643 buffer_gl0_inv. 9644 - Ensures any 9645 following global 9646 data read is no 9647 older than the local load 9648 atomic value being 9649 acquired. 9650 9651 4. buffer_gl0_inv 9652 9653 - If CU wavefront execution 9654 mode, omit. 9655 - If OpenCL omit. 9656 - Ensures that 9657 following 9658 loads will not see 9659 stale data. 9660 9661 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9662 vmcnt(0) & vscnt(0) 9663 9664 - If CU wavefront execution 9665 mode, omit vmcnt(0) and 9666 vscnt(0). 9667 - If OpenCL, omit lgkmcnt(0). 9668 - Could be split into 9669 separate s_waitcnt 9670 vmcnt(0), s_waitcnt 9671 vscnt(0) and s_waitcnt 9672 lgkmcnt(0) to allow 9673 them to be 9674 independently moved 9675 according to the 9676 following rules. 9677 - s_waitcnt vmcnt(0) 9678 must happen after 9679 any preceding 9680 global/generic load/load 9681 atomic/ 9682 atomicrmw-with-return-value. 9683 - s_waitcnt vscnt(0) 9684 must happen after 9685 any preceding 9686 global/generic 9687 store/store 9688 atomic/ 9689 atomicrmw-no-return-value. 9690 - s_waitcnt lgkmcnt(0) 9691 must happen after 9692 any preceding 9693 local/generic 9694 load/store/load 9695 atomic/store 9696 atomic/atomicrmw. 9697 - Must happen before 9698 the following 9699 atomicrmw. 9700 - Ensures that all 9701 memory operations 9702 have 9703 completed before 9704 performing the 9705 atomicrmw that is 9706 being released. 9707 9708 2. flat_atomic 9709 3. s_waitcnt lgkmcnt(0) & 9710 vmcnt(0) & vscnt(0) 9711 9712 - If CU wavefront execution 9713 mode, omit vmcnt(0) and 9714 vscnt(0). 9715 - If OpenCL, omit lgkmcnt(0). 9716 - Must happen before 9717 the following 9718 buffer_gl0_inv. 9719 - Ensures any 9720 following global 9721 data read is no 9722 older than the load 9723 atomic value being 9724 acquired. 9725 9726 3. buffer_gl0_inv 9727 9728 - If CU wavefront execution 9729 mode, omit. 9730 - Ensures that 9731 following 9732 loads will not see 9733 stale data. 9734 9735 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9736 - system vmcnt(0) & vscnt(0) 9737 9738 - If OpenCL, omit 9739 lgkmcnt(0). 9740 - Could be split into 9741 separate s_waitcnt 9742 vmcnt(0), s_waitcnt 9743 vscnt(0) and s_waitcnt 9744 lgkmcnt(0) to allow 9745 them to be 9746 independently moved 9747 according to the 9748 following rules. 9749 - s_waitcnt vmcnt(0) 9750 must happen after 9751 any preceding 9752 global/generic 9753 load/load atomic/ 9754 atomicrmw-with-return-value. 9755 - s_waitcnt vscnt(0) 9756 must happen after 9757 any preceding 9758 global/generic 9759 store/store atomic/ 9760 atomicrmw-no-return-value. 9761 - s_waitcnt lgkmcnt(0) 9762 must happen after 9763 any preceding 9764 local/generic 9765 load/store/load 9766 atomic/store 9767 atomic/atomicrmw. 9768 - Must happen before 9769 the following 9770 atomicrmw. 9771 - Ensures that all 9772 memory operations 9773 to global have 9774 completed before 9775 performing the 9776 atomicrmw that is 9777 being released. 9778 9779 2. buffer/global_atomic 9780 3. s_waitcnt vm/vscnt(0) 9781 9782 - Use vmcnt(0) if atomic with 9783 return and vscnt(0) if 9784 atomic with no-return. 9785 - Must happen before 9786 following 9787 buffer_gl*_inv. 9788 - Ensures the 9789 atomicrmw has 9790 completed before 9791 invalidating the 9792 caches. 9793 9794 4. buffer_gl0_inv; 9795 buffer_gl1_inv 9796 9797 - Must happen before 9798 any following 9799 global/generic 9800 load/load 9801 atomic/atomicrmw. 9802 - Ensures that 9803 following loads 9804 will not see stale 9805 global data. 9806 9807 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9808 - system vmcnt(0) & vscnt(0) 9809 9810 - If OpenCL, omit 9811 lgkmcnt(0). 9812 - Could be split into 9813 separate s_waitcnt 9814 vmcnt(0), s_waitcnt 9815 vscnt(0), and s_waitcnt 9816 lgkmcnt(0) to allow 9817 them to be 9818 independently moved 9819 according to the 9820 following rules. 9821 - s_waitcnt vmcnt(0) 9822 must happen after 9823 any preceding 9824 global/generic 9825 load/load atomic 9826 atomicrmw-with-return-value. 9827 - s_waitcnt vscnt(0) 9828 must happen after 9829 any preceding 9830 global/generic 9831 store/store atomic/ 9832 atomicrmw-no-return-value. 9833 - s_waitcnt lgkmcnt(0) 9834 must happen after 9835 any preceding 9836 local/generic 9837 load/store/load 9838 atomic/store 9839 atomic/atomicrmw. 9840 - Must happen before 9841 the following 9842 atomicrmw. 9843 - Ensures that all 9844 memory operations 9845 have 9846 completed before 9847 performing the 9848 atomicrmw that is 9849 being released. 9850 9851 2. flat_atomic 9852 3. s_waitcnt vm/vscnt(0) & 9853 lgkmcnt(0) 9854 9855 - If OpenCL, omit 9856 lgkmcnt(0). 9857 - Use vmcnt(0) if atomic with 9858 return and vscnt(0) if 9859 atomic with no-return. 9860 - Must happen before 9861 following 9862 buffer_gl*_inv. 9863 - Ensures the 9864 atomicrmw has 9865 completed before 9866 invalidating the 9867 caches. 9868 9869 4. buffer_gl0_inv; 9870 buffer_gl1_inv 9871 9872 - Must happen before 9873 any following 9874 global/generic 9875 load/load 9876 atomic/atomicrmw. 9877 - Ensures that 9878 following loads 9879 will not see stale 9880 global data. 9881 9882 fence acq_rel - singlethread *none* *none* 9883 - wavefront 9884 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9885 vmcnt(0) & vscnt(0) 9886 9887 - If CU wavefront execution 9888 mode, omit vmcnt(0) and 9889 vscnt(0). 9890 - If OpenCL and 9891 address space is 9892 not generic, omit 9893 lgkmcnt(0). 9894 - If OpenCL and 9895 address space is 9896 local, omit 9897 vmcnt(0) and vscnt(0). 9898 - However, 9899 since LLVM 9900 currently has no 9901 address space on 9902 the fence need to 9903 conservatively 9904 always generate 9905 (see comment for 9906 previous fence). 9907 - Could be split into 9908 separate s_waitcnt 9909 vmcnt(0), s_waitcnt 9910 vscnt(0) and s_waitcnt 9911 lgkmcnt(0) to allow 9912 them to be 9913 independently moved 9914 according to the 9915 following rules. 9916 - s_waitcnt vmcnt(0) 9917 must happen after 9918 any preceding 9919 global/generic 9920 load/load 9921 atomic/ 9922 atomicrmw-with-return-value. 9923 - s_waitcnt vscnt(0) 9924 must happen after 9925 any preceding 9926 global/generic 9927 store/store atomic/ 9928 atomicrmw-no-return-value. 9929 - s_waitcnt lgkmcnt(0) 9930 must happen after 9931 any preceding 9932 local/generic 9933 load/store/load 9934 atomic/store atomic/ 9935 atomicrmw. 9936 - Must happen before 9937 any following 9938 global/generic 9939 load/load 9940 atomic/store/store 9941 atomic/atomicrmw. 9942 - Ensures that all 9943 memory operations 9944 have 9945 completed before 9946 performing any 9947 following global 9948 memory operations. 9949 - Ensures that the 9950 preceding 9951 local/generic load 9952 atomic/atomicrmw 9953 with an equal or 9954 wider sync scope 9955 and memory ordering 9956 stronger than 9957 unordered (this is 9958 termed the 9959 acquire-fence-paired-atomic) 9960 has completed 9961 before following 9962 global memory 9963 operations. This 9964 satisfies the 9965 requirements of 9966 acquire. 9967 - Ensures that all 9968 previous memory 9969 operations have 9970 completed before a 9971 following 9972 local/generic store 9973 atomic/atomicrmw 9974 with an equal or 9975 wider sync scope 9976 and memory ordering 9977 stronger than 9978 unordered (this is 9979 termed the 9980 release-fence-paired-atomic). 9981 This satisfies the 9982 requirements of 9983 release. 9984 - Must happen before 9985 the following 9986 buffer_gl0_inv. 9987 - Ensures that the 9988 acquire-fence-paired 9989 atomic has completed 9990 before invalidating 9991 the 9992 cache. Therefore 9993 any following 9994 locations read must 9995 be no older than 9996 the value read by 9997 the 9998 acquire-fence-paired-atomic. 9999 10000 3. buffer_gl0_inv 10001 10002 - If CU wavefront execution 10003 mode, omit. 10004 - Ensures that 10005 following 10006 loads will not see 10007 stale data. 10008 10009 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 10010 - system vmcnt(0) & vscnt(0) 10011 10012 - If OpenCL and 10013 address space is 10014 not generic, omit 10015 lgkmcnt(0). 10016 - If OpenCL and 10017 address space is 10018 local, omit 10019 vmcnt(0) and vscnt(0). 10020 - However, since LLVM 10021 currently has no 10022 address space on 10023 the fence need to 10024 conservatively 10025 always generate 10026 (see comment for 10027 previous fence). 10028 - Could be split into 10029 separate s_waitcnt 10030 vmcnt(0), s_waitcnt 10031 vscnt(0) and s_waitcnt 10032 lgkmcnt(0) to allow 10033 them to be 10034 independently moved 10035 according to the 10036 following rules. 10037 - s_waitcnt vmcnt(0) 10038 must happen after 10039 any preceding 10040 global/generic 10041 load/load 10042 atomic/ 10043 atomicrmw-with-return-value. 10044 - s_waitcnt vscnt(0) 10045 must happen after 10046 any preceding 10047 global/generic 10048 store/store atomic/ 10049 atomicrmw-no-return-value. 10050 - s_waitcnt lgkmcnt(0) 10051 must happen after 10052 any preceding 10053 local/generic 10054 load/store/load 10055 atomic/store 10056 atomic/atomicrmw. 10057 - Must happen before 10058 the following 10059 buffer_gl*_inv. 10060 - Ensures that the 10061 preceding 10062 global/local/generic 10063 load 10064 atomic/atomicrmw 10065 with an equal or 10066 wider sync scope 10067 and memory ordering 10068 stronger than 10069 unordered (this is 10070 termed the 10071 acquire-fence-paired-atomic) 10072 has completed 10073 before invalidating 10074 the caches. This 10075 satisfies the 10076 requirements of 10077 acquire. 10078 - Ensures that all 10079 previous memory 10080 operations have 10081 completed before a 10082 following 10083 global/local/generic 10084 store 10085 atomic/atomicrmw 10086 with an equal or 10087 wider sync scope 10088 and memory ordering 10089 stronger than 10090 unordered (this is 10091 termed the 10092 release-fence-paired-atomic). 10093 This satisfies the 10094 requirements of 10095 release. 10096 10097 2. buffer_gl0_inv; 10098 buffer_gl1_inv 10099 10100 - Must happen before 10101 any following 10102 global/generic 10103 load/load 10104 atomic/store/store 10105 atomic/atomicrmw. 10106 - Ensures that 10107 following loads 10108 will not see stale 10109 global data. This 10110 satisfies the 10111 requirements of 10112 acquire. 10113 10114 **Sequential Consistent Atomic** 10115 ------------------------------------------------------------------------------------ 10116 load atomic seq_cst - singlethread - global *Same as corresponding 10117 - wavefront - local load atomic acquire, 10118 - generic except must generated 10119 all instructions even 10120 for OpenCL.* 10121 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10122 - generic vmcnt(0) & vscnt(0) 10123 10124 - If CU wavefront execution 10125 mode, omit vmcnt(0) and 10126 vscnt(0). 10127 - Could be split into 10128 separate s_waitcnt 10129 vmcnt(0), s_waitcnt 10130 vscnt(0), and s_waitcnt 10131 lgkmcnt(0) to allow 10132 them to be 10133 independently moved 10134 according to the 10135 following rules. 10136 - s_waitcnt lgkmcnt(0) must 10137 happen after 10138 preceding 10139 local/generic load 10140 atomic/store 10141 atomic/atomicrmw 10142 with memory 10143 ordering of seq_cst 10144 and with equal or 10145 wider sync scope. 10146 (Note that seq_cst 10147 fences have their 10148 own s_waitcnt 10149 lgkmcnt(0) and so do 10150 not need to be 10151 considered.) 10152 - s_waitcnt vmcnt(0) 10153 must happen after 10154 preceding 10155 global/generic load 10156 atomic/ 10157 atomicrmw-with-return-value 10158 with memory 10159 ordering of seq_cst 10160 and with equal or 10161 wider sync scope. 10162 (Note that seq_cst 10163 fences have their 10164 own s_waitcnt 10165 vmcnt(0) and so do 10166 not need to be 10167 considered.) 10168 - s_waitcnt vscnt(0) 10169 Must happen after 10170 preceding 10171 global/generic store 10172 atomic/ 10173 atomicrmw-no-return-value 10174 with memory 10175 ordering of seq_cst 10176 and with equal or 10177 wider sync scope. 10178 (Note that seq_cst 10179 fences have their 10180 own s_waitcnt 10181 vscnt(0) and so do 10182 not need to be 10183 considered.) 10184 - Ensures any 10185 preceding 10186 sequential 10187 consistent global/local 10188 memory instructions 10189 have completed 10190 before executing 10191 this sequentially 10192 consistent 10193 instruction. This 10194 prevents reordering 10195 a seq_cst store 10196 followed by a 10197 seq_cst load. (Note 10198 that seq_cst is 10199 stronger than 10200 acquire/release as 10201 the reordering of 10202 load acquire 10203 followed by a store 10204 release is 10205 prevented by the 10206 s_waitcnt of 10207 the release, but 10208 there is nothing 10209 preventing a store 10210 release followed by 10211 load acquire from 10212 completing out of 10213 order. The s_waitcnt 10214 could be placed after 10215 seq_store or before 10216 the seq_load. We 10217 choose the load to 10218 make the s_waitcnt be 10219 as late as possible 10220 so that the store 10221 may have already 10222 completed.) 10223 10224 2. *Following 10225 instructions same as 10226 corresponding load 10227 atomic acquire, 10228 except must generated 10229 all instructions even 10230 for OpenCL.* 10231 load atomic seq_cst - workgroup - local 10232 10233 1. s_waitcnt vmcnt(0) & vscnt(0) 10234 10235 - If CU wavefront execution 10236 mode, omit. 10237 - Could be split into 10238 separate s_waitcnt 10239 vmcnt(0) and s_waitcnt 10240 vscnt(0) to allow 10241 them to be 10242 independently moved 10243 according to the 10244 following rules. 10245 - s_waitcnt vmcnt(0) 10246 Must happen after 10247 preceding 10248 global/generic load 10249 atomic/ 10250 atomicrmw-with-return-value 10251 with memory 10252 ordering of seq_cst 10253 and with equal or 10254 wider sync scope. 10255 (Note that seq_cst 10256 fences have their 10257 own s_waitcnt 10258 vmcnt(0) and so do 10259 not need to be 10260 considered.) 10261 - s_waitcnt vscnt(0) 10262 Must happen after 10263 preceding 10264 global/generic store 10265 atomic/ 10266 atomicrmw-no-return-value 10267 with memory 10268 ordering of seq_cst 10269 and with equal or 10270 wider sync scope. 10271 (Note that seq_cst 10272 fences have their 10273 own s_waitcnt 10274 vscnt(0) and so do 10275 not need to be 10276 considered.) 10277 - Ensures any 10278 preceding 10279 sequential 10280 consistent global 10281 memory instructions 10282 have completed 10283 before executing 10284 this sequentially 10285 consistent 10286 instruction. This 10287 prevents reordering 10288 a seq_cst store 10289 followed by a 10290 seq_cst load. (Note 10291 that seq_cst is 10292 stronger than 10293 acquire/release as 10294 the reordering of 10295 load acquire 10296 followed by a store 10297 release is 10298 prevented by the 10299 s_waitcnt of 10300 the release, but 10301 there is nothing 10302 preventing a store 10303 release followed by 10304 load acquire from 10305 completing out of 10306 order. The s_waitcnt 10307 could be placed after 10308 seq_store or before 10309 the seq_load. We 10310 choose the load to 10311 make the s_waitcnt be 10312 as late as possible 10313 so that the store 10314 may have already 10315 completed.) 10316 10317 2. *Following 10318 instructions same as 10319 corresponding load 10320 atomic acquire, 10321 except must generated 10322 all instructions even 10323 for OpenCL.* 10324 10325 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10326 - system - generic vmcnt(0) & vscnt(0) 10327 10328 - Could be split into 10329 separate s_waitcnt 10330 vmcnt(0), s_waitcnt 10331 vscnt(0) and s_waitcnt 10332 lgkmcnt(0) to allow 10333 them to be 10334 independently moved 10335 according to the 10336 following rules. 10337 - s_waitcnt lgkmcnt(0) 10338 must happen after 10339 preceding 10340 local load 10341 atomic/store 10342 atomic/atomicrmw 10343 with memory 10344 ordering of seq_cst 10345 and with equal or 10346 wider sync scope. 10347 (Note that seq_cst 10348 fences have their 10349 own s_waitcnt 10350 lgkmcnt(0) and so do 10351 not need to be 10352 considered.) 10353 - s_waitcnt vmcnt(0) 10354 must happen after 10355 preceding 10356 global/generic load 10357 atomic/ 10358 atomicrmw-with-return-value 10359 with memory 10360 ordering of seq_cst 10361 and with equal or 10362 wider sync scope. 10363 (Note that seq_cst 10364 fences have their 10365 own s_waitcnt 10366 vmcnt(0) and so do 10367 not need to be 10368 considered.) 10369 - s_waitcnt vscnt(0) 10370 Must happen after 10371 preceding 10372 global/generic store 10373 atomic/ 10374 atomicrmw-no-return-value 10375 with memory 10376 ordering of seq_cst 10377 and with equal or 10378 wider sync scope. 10379 (Note that seq_cst 10380 fences have their 10381 own s_waitcnt 10382 vscnt(0) and so do 10383 not need to be 10384 considered.) 10385 - Ensures any 10386 preceding 10387 sequential 10388 consistent global 10389 memory instructions 10390 have completed 10391 before executing 10392 this sequentially 10393 consistent 10394 instruction. This 10395 prevents reordering 10396 a seq_cst store 10397 followed by a 10398 seq_cst load. (Note 10399 that seq_cst is 10400 stronger than 10401 acquire/release as 10402 the reordering of 10403 load acquire 10404 followed by a store 10405 release is 10406 prevented by the 10407 s_waitcnt of 10408 the release, but 10409 there is nothing 10410 preventing a store 10411 release followed by 10412 load acquire from 10413 completing out of 10414 order. The s_waitcnt 10415 could be placed after 10416 seq_store or before 10417 the seq_load. We 10418 choose the load to 10419 make the s_waitcnt be 10420 as late as possible 10421 so that the store 10422 may have already 10423 completed.) 10424 10425 2. *Following 10426 instructions same as 10427 corresponding load 10428 atomic acquire, 10429 except must generated 10430 all instructions even 10431 for OpenCL.* 10432 store atomic seq_cst - singlethread - global *Same as corresponding 10433 - wavefront - local store atomic release, 10434 - workgroup - generic except must generated 10435 - agent all instructions even 10436 - system for OpenCL.* 10437 atomicrmw seq_cst - singlethread - global *Same as corresponding 10438 - wavefront - local atomicrmw acq_rel, 10439 - workgroup - generic except must generated 10440 - agent all instructions even 10441 - system for OpenCL.* 10442 fence seq_cst - singlethread *none* *Same as corresponding 10443 - wavefront fence acq_rel, 10444 - workgroup except must generated 10445 - agent all instructions even 10446 - system for OpenCL.* 10447 ============ ============ ============== ========== ================================ 10448 10449Trap Handler ABI 10450~~~~~~~~~~~~~~~~ 10451 10452For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10453runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10454supports the ``s_trap`` instruction. For usage see: 10455 10456- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10457- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10458- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10459 10460 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10461 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10462 10463 =================== =============== =============== ======================================= 10464 Usage Code Sequence Trap Handler Description 10465 Inputs 10466 =================== =============== =============== ======================================= 10467 reserved ``s_trap 0x00`` Reserved by hardware. 10468 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10469 ``queue_ptr`` intrinsic (not implemented). 10470 ``VGPR0``: 10471 ``arg`` 10472 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10473 ``queue_ptr`` the trap instruction. The associated 10474 queue is signalled to put it into the 10475 error state. When the queue is put in 10476 the error state, the waves executing 10477 dispatches on the queue will be 10478 terminated. 10479 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10480 as a no-operation. The trap handler 10481 is entered and immediately returns to 10482 continue execution of the wavefront. 10483 - If the debugger is enabled, causes 10484 the debug trap to be reported by the 10485 debugger and the wavefront is put in 10486 the halt state with the PC at the 10487 instruction. The debugger must 10488 increment the PC and resume the wave. 10489 reserved ``s_trap 0x04`` Reserved. 10490 reserved ``s_trap 0x05`` Reserved. 10491 reserved ``s_trap 0x06`` Reserved. 10492 reserved ``s_trap 0x07`` Reserved. 10493 reserved ``s_trap 0x08`` Reserved. 10494 reserved ``s_trap 0xfe`` Reserved. 10495 reserved ``s_trap 0xff`` Reserved. 10496 =================== =============== =============== ======================================= 10497 10498.. 10499 10500 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10501 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10502 10503 =================== =============== =============== ======================================= 10504 Usage Code Sequence Trap Handler Description 10505 Inputs 10506 =================== =============== =============== ======================================= 10507 reserved ``s_trap 0x00`` Reserved by hardware. 10508 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10509 breakpoints. Causes wave to be halted 10510 with the PC at the trap instruction. 10511 The debugger is responsible to resume 10512 the wave, including the instruction 10513 that the breakpoint overwrote. 10514 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10515 ``queue_ptr`` the trap instruction. The associated 10516 queue is signalled to put it into the 10517 error state. When the queue is put in 10518 the error state, the waves executing 10519 dispatches on the queue will be 10520 terminated. 10521 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10522 as a no-operation. The trap handler 10523 is entered and immediately returns to 10524 continue execution of the wavefront. 10525 - If the debugger is enabled, causes 10526 the debug trap to be reported by the 10527 debugger and the wavefront is put in 10528 the halt state with the PC at the 10529 instruction. The debugger must 10530 increment the PC and resume the wave. 10531 reserved ``s_trap 0x04`` Reserved. 10532 reserved ``s_trap 0x05`` Reserved. 10533 reserved ``s_trap 0x06`` Reserved. 10534 reserved ``s_trap 0x07`` Reserved. 10535 reserved ``s_trap 0x08`` Reserved. 10536 reserved ``s_trap 0xfe`` Reserved. 10537 reserved ``s_trap 0xff`` Reserved. 10538 =================== =============== =============== ======================================= 10539 10540.. 10541 10542 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10543 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10544 10545 =================== =============== ================ ================= ======================================= 10546 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10547 =================== =============== ================ ================= ======================================= 10548 reserved ``s_trap 0x00`` Reserved by hardware. 10549 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10550 breakpoints. Causes wave to be halted 10551 with the PC at the trap instruction. 10552 The debugger is responsible to resume 10553 the wave, including the instruction 10554 that the breakpoint overwrote. 10555 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10556 ``queue_ptr`` the trap instruction. The associated 10557 queue is signalled to put it into the 10558 error state. When the queue is put in 10559 the error state, the waves executing 10560 dispatches on the queue will be 10561 terminated. 10562 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10563 as a no-operation. The trap handler 10564 is entered and immediately returns to 10565 continue execution of the wavefront. 10566 - If the debugger is enabled, causes 10567 the debug trap to be reported by the 10568 debugger and the wavefront is put in 10569 the halt state with the PC at the 10570 instruction. The debugger must 10571 increment the PC and resume the wave. 10572 reserved ``s_trap 0x04`` Reserved. 10573 reserved ``s_trap 0x05`` Reserved. 10574 reserved ``s_trap 0x06`` Reserved. 10575 reserved ``s_trap 0x07`` Reserved. 10576 reserved ``s_trap 0x08`` Reserved. 10577 reserved ``s_trap 0xfe`` Reserved. 10578 reserved ``s_trap 0xff`` Reserved. 10579 =================== =============== ================ ================= ======================================= 10580 10581.. _amdgpu-amdhsa-function-call-convention: 10582 10583Call Convention 10584~~~~~~~~~~~~~~~ 10585 10586.. note:: 10587 10588 This section is currently incomplete and has inaccuracies. It is WIP that will 10589 be updated as information is determined. 10590 10591See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10592addresses. Unswizzled addresses are normal linear addresses. 10593 10594.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10595 10596Kernel Functions 10597++++++++++++++++ 10598 10599This section describes the call convention ABI for the outer kernel function. 10600 10601See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10602convention. 10603 10604The following is not part of the AMDGPU kernel calling convention but describes 10605how the AMDGPU implements function calls: 10606 106071. Clang decides the kernarg layout to match the *HSA Programmer's Language 10608 Reference* [HSA]_. 10609 10610 - All structs are passed directly. 10611 - Lambda values are passed *TBA*. 10612 10613 .. TODO:: 10614 10615 - Does this really follow HSA rules? Or are structs >16 bytes passed 10616 by-value struct? 10617 - What is ABI for lambda values? 10618 106194. The kernel performs certain setup in its prolog, as described in 10620 :ref:`amdgpu-amdhsa-kernel-prolog`. 10621 10622.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10623 10624Non-Kernel Functions 10625++++++++++++++++++++ 10626 10627This section describes the call convention ABI for functions other than the 10628outer kernel function. 10629 10630If a kernel has function calls then scratch is always allocated and used for 10631the call stack which grows from low address to high address using the swizzled 10632scratch address space. 10633 10634On entry to a function: 10635 106361. SGPR0-3 contain a V# with the following properties (see 10637 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10638 10639 * Base address pointing to the beginning of the wavefront scratch backing 10640 memory. 10641 * Swizzled with dword element size and stride of wavefront size elements. 10642 106432. The FLAT_SCRATCH register pair is setup. See 10644 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 106453. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10646 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 106474. The EXEC register is set to the lanes active on entry to the function. 106485. MODE register: *TBD* 106496. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10650 below. 106517. SGPR30-31 return address (RA). The code address that the function must 10652 return to when it completes. The value is undefined if the function is *no 10653 return*. 106548. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10655 offset relative to the beginning of the wavefront scratch backing memory. 10656 10657 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10658 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10659 manner. 10660 10661 The unswizzled SP value can be converted into the swizzled SP value by: 10662 10663 | swizzled SP = unswizzled SP / wavefront size 10664 10665 This may be used to obtain the private address space address of stack 10666 objects and to convert this address to a flat address by adding the flat 10667 scratch aperture base address. 10668 10669 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10670 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10671 10672 .. note:: 10673 10674 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10675 OpenCL language which has the largest base type defined as 16 bytes. 10676 10677 On entry, the swizzled SP value is the address of the first function 10678 argument passed on the stack. Other stack passed arguments are positive 10679 offsets from the entry swizzled SP value. 10680 10681 The function may use positive offsets beyond the last stack passed argument 10682 for stack allocated local variables and register spill slots. If necessary, 10683 the function may align these to greater alignment than 16 bytes. After these 10684 the function may dynamically allocate space for such things as runtime sized 10685 ``alloca`` local allocations. 10686 10687 If the function calls another function, it will place any stack allocated 10688 arguments after the last local allocation and adjust SGPR32 to the address 10689 after the last local allocation. 10690 106919. All other registers are unspecified. 1069210. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10693 to the function. 10694 10695On exit from a function: 10696 106971. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10698 described below. Any registers used are considered clobbered registers. 106992. The following registers are preserved and have the same value as on entry: 10700 10701 * FLAT_SCRATCH 10702 * EXEC 10703 * GFX6-GFX8: M0 10704 * All SGPR registers except the clobbered registers of SGPR4-31. 10705 * VGPR40-47 10706 * VGPR56-63 10707 * VGPR72-79 10708 * VGPR88-95 10709 * VGPR104-111 10710 * VGPR120-127 10711 * VGPR136-143 10712 * VGPR152-159 10713 * VGPR168-175 10714 * VGPR184-191 10715 * VGPR200-207 10716 * VGPR216-223 10717 * VGPR232-239 10718 * VGPR248-255 10719 10720 .. note:: 10721 10722 Except the argument registers, the VGPRs clobbered and the preserved 10723 registers are intermixed at regular intervals in order to keep a 10724 similar ratio independent of the number of allocated VGPRs. 10725 10726 * Lanes of all VGPRs that are inactive at the call site. 10727 10728 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10729 optimization may mark some of clobbered SGPR and VGPR registers as 10730 preserved if it can be determined that the called function does not change 10731 their value. 10732 107332. The PC is set to the RA provided on entry. 107343. MODE register: *TBD*. 107354. All other registers are clobbered. 107365. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10737 function is available to the caller. 10738 10739.. TODO:: 10740 10741 - On gfx908 are all ACC registers clobbered? 10742 10743 - How are function results returned? The address of structured types is passed 10744 by reference, but what about other types? 10745 10746The function input arguments are made up of the formal arguments explicitly 10747declared by the source language function plus the implicit input arguments used 10748by the implementation. 10749 10750The source language input arguments are: 10751 107521. Any source language implicit ``this`` or ``self`` argument comes first as a 10753 pointer type. 107542. Followed by the function formal arguments in left to right source order. 10755 10756The source language result arguments are: 10757 107581. The function result argument. 10759 10760The source language input or result struct type arguments that are less than or 10761equal to 16 bytes, are decomposed recursively into their base type fields, and 10762each field is passed as if a separate argument. For input arguments, if the 10763called function requires the struct to be in memory, for example because its 10764address is taken, then the function body is responsible for allocating a stack 10765location and copying the field arguments into it. Clang terms this *direct 10766struct*. 10767 10768The source language input struct type arguments that are greater than 16 bytes, 10769are passed by reference. The caller is responsible for allocating a stack 10770location to make a copy of the struct value and pass the address as the input 10771argument. The called function is responsible to perform the dereference when 10772accessing the input argument. Clang terms this *by-value struct*. 10773 10774A source language result struct type argument that is greater than 16 bytes, is 10775returned by reference. The caller is responsible for allocating a stack location 10776to hold the result value and passes the address as the last input argument 10777(before the implicit input arguments). In this case there are no result 10778arguments. The called function is responsible to perform the dereference when 10779storing the result value. Clang terms this *structured return (sret)*. 10780 10781*TODO: correct the ``sret`` definition.* 10782 10783.. TODO:: 10784 10785 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10786 pass as non-decomposed struct as stack argument? Or something else? Is the 10787 memory location in the caller stack frame, or a stack memory argument and so 10788 no address is passed as the caller can directly write to the argument stack 10789 location? But then the stack location is still live after return. If an 10790 argument stack location is it the first stack argument or the last one? 10791 10792Lambda argument types are treated as struct types with an implementation defined 10793set of fields. 10794 10795.. TODO:: 10796 10797 Need to specify the ABI for lambda types for AMDGPU. 10798 10799For AMDGPU backend all source language arguments (including the decomposed 10800struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10801they are passed in SGPRs. 10802 10803The AMDGPU backend walks the function call graph from the leaves to determine 10804which implicit input arguments are used, propagating to each caller of the 10805function. The used implicit arguments are appended to the function arguments 10806after the source language arguments in the following order: 10807 10808.. TODO:: 10809 10810 Is recursion or external functions supported? 10811 108121. Work-Item ID (1 VGPR) 10813 10814 The X, Y and Z work-item ID are packed into a single VGRP with the following 10815 layout. Only fields actually used by the function are set. The other bits 10816 are undefined. 10817 10818 The values come from the initial kernel execution state. See 10819 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10820 10821 .. table:: Work-item implicit argument layout 10822 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10823 10824 ======= ======= ============== 10825 Bits Size Field Name 10826 ======= ======= ============== 10827 9:0 10 bits X Work-Item ID 10828 19:10 10 bits Y Work-Item ID 10829 29:20 10 bits Z Work-Item ID 10830 31:30 2 bits Unused 10831 ======= ======= ============== 10832 108332. Dispatch Ptr (2 SGPRs) 10834 10835 The value comes from the initial kernel execution state. See 10836 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10837 108383. Queue Ptr (2 SGPRs) 10839 10840 The value comes from the initial kernel execution state. See 10841 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10842 108434. Kernarg Segment Ptr (2 SGPRs) 10844 10845 The value comes from the initial kernel execution state. See 10846 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10847 108485. Dispatch id (2 SGPRs) 10849 10850 The value comes from the initial kernel execution state. See 10851 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10852 108536. Work-Group ID X (1 SGPR) 10854 10855 The value comes from the initial kernel execution state. See 10856 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10857 108587. Work-Group ID Y (1 SGPR) 10859 10860 The value comes from the initial kernel execution state. See 10861 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10862 108638. Work-Group ID Z (1 SGPR) 10864 10865 The value comes from the initial kernel execution state. See 10866 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10867 108689. Implicit Argument Ptr (2 SGPRs) 10869 10870 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10871 global address space pointer to the first kernarg implicit argument. 10872 10873The input and result arguments are assigned in order in the following manner: 10874 10875.. note:: 10876 10877 There are likely some errors and omissions in the following description that 10878 need correction. 10879 10880 .. TODO:: 10881 10882 Check the Clang source code to decipher how function arguments and return 10883 results are handled. Also see the AMDGPU specific values used. 10884 10885* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 10886 VGPR31. 10887 10888 If there are more arguments than will fit in these registers, the remaining 10889 arguments are allocated on the stack in order on naturally aligned 10890 addresses. 10891 10892 .. TODO:: 10893 10894 How are overly aligned structures allocated on the stack? 10895 10896* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 10897 SGPR29. 10898 10899 If there are more arguments than will fit in these registers, the remaining 10900 arguments are allocated on the stack in order on naturally aligned 10901 addresses. 10902 10903Note that decomposed struct type arguments may have some fields passed in 10904registers and some in memory. 10905 10906.. TODO:: 10907 10908 So, a struct which can pass some fields as decomposed register arguments, will 10909 pass the rest as decomposed stack elements? But an argument that will not start 10910 in registers will not be decomposed and will be passed as a non-decomposed 10911 stack value? 10912 10913The following is not part of the AMDGPU function calling convention but 10914describes how the AMDGPU implements function calls: 10915 109161. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 10917 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 10918 are used, or for the reasons defined in ``SIFrameLowering``. 109192. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 10920 to access the incoming stack arguments in the function. The BP is needed 10921 only when the function requires the runtime stack alignment. 10922 109233. Allocating SGPR arguments on the stack are not supported. 10924 109254. No CFI is currently generated. See 10926 :ref:`amdgpu-dwarf-call-frame-information`. 10927 10928 .. note:: 10929 10930 CFI will be generated that defines the CFA as the unswizzled address 10931 relative to the wave scratch base in the unswizzled private address space 10932 of the lowest address stack allocated local variable. 10933 10934 ``DW_AT_frame_base`` will be defined as the swizzled address in the 10935 swizzled private address space by dividing the CFA by the wavefront size 10936 (since CFA is always at least dword aligned which matches the scratch 10937 swizzle element size). 10938 10939 If no dynamic stack alignment was performed, the stack allocated arguments 10940 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 10941 local variables and register spill slots are accessed as positive offsets 10942 relative to ``DW_AT_frame_base``. 10943 109445. Function argument passing is implemented by copying the input physical 10945 registers to virtual registers on entry. The register allocator can spill if 10946 necessary. These are copied back to physical registers at call sites. The 10947 net effect is that each function call can have these values in entirely 10948 distinct locations. The IPRA can help avoid shuffling argument registers. 109496. Call sites are implemented by setting up the arguments at positive offsets 10950 from SP. Then SP is incremented to account for the known frame size before 10951 the call and decremented after the call. 10952 10953 .. note:: 10954 10955 The CFI will reflect the changed calculation needed to compute the CFA 10956 from SP. 10957 109587. 4 byte spill slots are used in the stack frame. One slot is allocated for an 10959 emergency spill slot. Buffer instructions are used for stack accesses and 10960 not the ``flat_scratch`` instruction. 10961 10962 .. TODO:: 10963 10964 Explain when the emergency spill slot is used. 10965 10966.. TODO:: 10967 10968 Possible broken issues: 10969 10970 - Stack arguments must be aligned to required alignment. 10971 - Stack is aligned to max(16, max formal argument alignment) 10972 - Direct argument < 64 bits should check register budget. 10973 - Register budget calculation should respect ``inreg`` for SGPR. 10974 - SGPR overflow is not handled. 10975 - struct with 1 member unpeeling is not checking size of member. 10976 - ``sret`` is after ``this`` pointer. 10977 - Caller is not implementing stack realignment: need an extra pointer. 10978 - Should say AMDGPU passes FP rather than SP. 10979 - Should CFI define CFA as address of locals or arguments. Difference is 10980 apparent when have implemented dynamic alignment. 10981 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 10982 highest address of stack frame and use negative offset for locals. Would 10983 allow SP to be the same as FP and could support signal-handler-like as now 10984 have a real SP for the top of the stack. 10985 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 10986 arguments? 10987 10988AMDPAL 10989------ 10990 10991This section provides code conventions used when the target triple OS is 10992``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 10993from the application/runtime to each invocation of a hardware shader. These 10994parameters include both generic, application-controlled parameters called 10995*user data* as well as system-generated parameters that are a product of the 10996draw or dispatch execution. 10997 10998User Data 10999~~~~~~~~~ 11000 11001Each hardware stage has a set of 32-bit *user data registers* which can be 11002written from a command buffer and then loaded into SGPRs when waves are launched 11003via a subsequent dispatch or draw operation. This is the way most arguments are 11004passed from the application/runtime to a hardware shader. 11005 11006Compute User Data 11007~~~~~~~~~~~~~~~~~ 11008 11009Compute shader user data mappings are simpler than graphics shaders and have a 11010fixed mapping. 11011 11012Note that there are always 10 available *user data entries* in registers - 11013entries beyond that limit must be fetched from memory (via the spill table 11014pointer) by the shader. 11015 11016 .. table:: PAL Compute Shader User Data Registers 11017 :name: pal-compute-user-data-registers 11018 11019 ============= ================================ 11020 User Register Description 11021 ============= ================================ 11022 0 Global Internal Table (32-bit pointer) 11023 1 Per-Shader Internal Table (32-bit pointer) 11024 2 - 11 Application-Controlled User Data (10 32-bit values) 11025 12 Spill Table (32-bit pointer) 11026 13 - 14 Thread Group Count (64-bit pointer) 11027 15 GDS Range 11028 ============= ================================ 11029 11030Graphics User Data 11031~~~~~~~~~~~~~~~~~~ 11032 11033Graphics pipelines support a much more flexible user data mapping: 11034 11035 .. table:: PAL Graphics Shader User Data Registers 11036 :name: pal-graphics-user-data-registers 11037 11038 ============= ================================ 11039 User Register Description 11040 ============= ================================ 11041 0 Global Internal Table (32-bit pointer) 11042 + Per-Shader Internal Table (32-bit pointer) 11043 + 1-15 Application Controlled User Data 11044 (1-15 Contiguous 32-bit Values in Registers) 11045 + Spill Table (32-bit pointer) 11046 + Draw Index (First Stage Only) 11047 + Vertex Offset (First Stage Only) 11048 + Instance Offset (First Stage Only) 11049 ============= ================================ 11050 11051 The placement of the global internal table remains fixed in the first *user 11052 data SGPR register*. Otherwise all parameters are optional, and can be mapped 11053 to any desired *user data SGPR register*, with the following restrictions: 11054 11055 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 11056 active hardware stage in a graphics pipeline (i.e. where the API vertex 11057 shader runs). 11058 11059 * Application-controlled user data must be mapped into a contiguous range of 11060 user data registers. 11061 11062 * The application-controlled user data range supports compaction remapping, so 11063 only *entries* that are actually consumed by the shader must be assigned to 11064 corresponding *registers*. Note that in order to support an efficient runtime 11065 implementation, the remapping must pack *registers* in the same order as 11066 *entries*, with unused *entries* removed. 11067 11068.. _pal_global_internal_table: 11069 11070Global Internal Table 11071~~~~~~~~~~~~~~~~~~~~~ 11072 11073The global internal table is a table of *shader resource descriptors* (SRDs) 11074that define how certain engine-wide, runtime-managed resources should be 11075accessed from a shader. The majority of these resources have HW-defined formats, 11076and it is up to the compiler to write/read data as required by the target 11077hardware. 11078 11079The following table illustrates the required format: 11080 11081 .. table:: PAL Global Internal Table 11082 :name: pal-git-table 11083 11084 ============= ================================ 11085 Offset Description 11086 ============= ================================ 11087 0-3 Graphics Scratch SRD 11088 4-7 Compute Scratch SRD 11089 8-11 ES/GS Ring Output SRD 11090 12-15 ES/GS Ring Input SRD 11091 16-19 GS/VS Ring Output #0 11092 20-23 GS/VS Ring Output #1 11093 24-27 GS/VS Ring Output #2 11094 28-31 GS/VS Ring Output #3 11095 32-35 GS/VS Ring Input SRD 11096 36-39 Tessellation Factor Buffer SRD 11097 40-43 Off-Chip LDS Buffer SRD 11098 44-47 Off-Chip Param Cache Buffer SRD 11099 48-51 Sample Position Buffer SRD 11100 52 vaRange::ShadowDescriptorTable High Bits 11101 ============= ================================ 11102 11103 The pointer to the global internal table passed to the shader as user data 11104 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 11105 the top 32 bits of the pipeline, so the shader may use the program 11106 counter's top 32 bits. 11107 11108.. _pal_call-convention: 11109 11110Call Convention 11111~~~~~~~~~~~~~~~ 11112 11113For graphics use cases, the calling convention is `amdgpu_gfx`. 11114 11115.. note:: 11116 11117 `amdgpu_gfx` Function calls are currently in development and are 11118 subject to major changes. 11119 11120This calling convention shares most properties with calling non-kernel 11121functions (see 11122:ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions`). 11123Differences are: 11124 11125 - Currently there are none, differences will be listed here 11126 11127Unspecified OS 11128-------------- 11129 11130This section provides code conventions used when the target triple OS is 11131empty (see :ref:`amdgpu-target-triples`). 11132 11133Trap Handler ABI 11134~~~~~~~~~~~~~~~~ 11135 11136For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11137not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11138instructions are handled as follows: 11139 11140 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11141 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11142 11143 =============== =============== =========================================== 11144 Usage Code Sequence Description 11145 =============== =============== =========================================== 11146 llvm.trap s_endpgm Causes wavefront to be terminated. 11147 llvm.debugtrap *none* Compiler warning given that there is no 11148 trap handler installed. 11149 =============== =============== =========================================== 11150 11151Source Languages 11152================ 11153 11154.. _amdgpu-opencl: 11155 11156OpenCL 11157------ 11158 11159When the language is OpenCL the following differences occur: 11160 111611. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 111622. The AMDGPU backend appends additional arguments to the kernel's explicit 11163 arguments for the AMDHSA OS (see 11164 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 111653. Additional metadata is generated 11166 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11167 11168 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11169 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11170 11171 ======== ==== ========= =========================================== 11172 Position Byte Byte Description 11173 Size Alignment 11174 ======== ==== ========= =========================================== 11175 1 8 8 OpenCL Global Offset X 11176 2 8 8 OpenCL Global Offset Y 11177 3 8 8 OpenCL Global Offset Z 11178 4 8 8 OpenCL address of printf buffer 11179 5 8 8 OpenCL address of virtual queue used by 11180 enqueue_kernel. 11181 6 8 8 OpenCL address of AqlWrap struct used by 11182 enqueue_kernel. 11183 7 8 8 Pointer argument used for Multi-gird 11184 synchronization. 11185 ======== ==== ========= =========================================== 11186 11187.. _amdgpu-hcc: 11188 11189HCC 11190--- 11191 11192When the language is HCC the following differences occur: 11193 111941. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11195 11196.. _amdgpu-assembler: 11197 11198Assembler 11199--------- 11200 11201AMDGPU backend has LLVM-MC based assembler which is currently in development. 11202It supports AMDGCN GFX6-GFX10. 11203 11204This section describes general syntax for instructions and operands. 11205 11206Instructions 11207~~~~~~~~~~~~ 11208 11209An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11210 11211 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11212 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11213 11214:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11215:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11216 11217The order of operands and modifiers is fixed. 11218Most modifiers are optional and may be omitted. 11219 11220Links to detailed instruction syntax description may be found in the following 11221table. Note that features under development are not included 11222in this description. 11223 11224 =================================== ======================================= 11225 Core ISA ISA Extensions 11226 =================================== ======================================= 11227 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11228 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11229 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11230 11231 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11232 11233 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11234 11235 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11236 11237 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11238 11239 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11240 11241 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11242 11243 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11244 =================================== ======================================= 11245 11246For more information about instructions, their semantics and supported 11247combinations of operands, refer to one of instruction set architecture manuals 11248[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_, 11249[AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11250 11251Operands 11252~~~~~~~~ 11253 11254Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11255 11256Modifiers 11257~~~~~~~~~ 11258 11259Detailed description of modifiers may be found 11260:doc:`here<AMDGPUModifierSyntax>`. 11261 11262Instruction Examples 11263~~~~~~~~~~~~~~~~~~~~ 11264 11265DS 11266++ 11267 11268.. code-block:: nasm 11269 11270 ds_add_u32 v2, v4 offset:16 11271 ds_write_src2_b64 v2 offset0:4 offset1:8 11272 ds_cmpst_f32 v2, v4, v6 11273 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11274 11275For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11276Manual. 11277 11278FLAT 11279++++ 11280 11281.. code-block:: nasm 11282 11283 flat_load_dword v1, v[3:4] 11284 flat_store_dwordx3 v[3:4], v[5:7] 11285 flat_atomic_swap v1, v[3:4], v5 glc 11286 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11287 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11288 11289For full list of supported instructions, refer to "FLAT instructions" in ISA 11290Manual. 11291 11292MUBUF 11293+++++ 11294 11295.. code-block:: nasm 11296 11297 buffer_load_dword v1, off, s[4:7], s1 11298 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11299 buffer_store_format_xy v[1:2], off, s[4:7], s1 11300 buffer_wbinvl1 11301 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11302 11303For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11304Manual. 11305 11306SMRD/SMEM 11307+++++++++ 11308 11309.. code-block:: nasm 11310 11311 s_load_dword s1, s[2:3], 0xfc 11312 s_load_dwordx8 s[8:15], s[2:3], s4 11313 s_load_dwordx16 s[88:103], s[2:3], s4 11314 s_dcache_inv_vol 11315 s_memtime s[4:5] 11316 11317For full list of supported instructions, refer to "Scalar Memory Operations" in 11318ISA Manual. 11319 11320SOP1 11321++++ 11322 11323.. code-block:: nasm 11324 11325 s_mov_b32 s1, s2 11326 s_mov_b64 s[0:1], 0x80000000 11327 s_cmov_b32 s1, 200 11328 s_wqm_b64 s[2:3], s[4:5] 11329 s_bcnt0_i32_b64 s1, s[2:3] 11330 s_swappc_b64 s[2:3], s[4:5] 11331 s_cbranch_join s[4:5] 11332 11333For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11334Manual. 11335 11336SOP2 11337++++ 11338 11339.. code-block:: nasm 11340 11341 s_add_u32 s1, s2, s3 11342 s_and_b64 s[2:3], s[4:5], s[6:7] 11343 s_cselect_b32 s1, s2, s3 11344 s_andn2_b32 s2, s4, s6 11345 s_lshr_b64 s[2:3], s[4:5], s6 11346 s_ashr_i32 s2, s4, s6 11347 s_bfm_b64 s[2:3], s4, s6 11348 s_bfe_i64 s[2:3], s[4:5], s6 11349 s_cbranch_g_fork s[4:5], s[6:7] 11350 11351For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11352Manual. 11353 11354SOPC 11355++++ 11356 11357.. code-block:: nasm 11358 11359 s_cmp_eq_i32 s1, s2 11360 s_bitcmp1_b32 s1, s2 11361 s_bitcmp0_b64 s[2:3], s4 11362 s_setvskip s3, s5 11363 11364For full list of supported instructions, refer to "SOPC Instructions" in ISA 11365Manual. 11366 11367SOPP 11368++++ 11369 11370.. code-block:: nasm 11371 11372 s_barrier 11373 s_nop 2 11374 s_endpgm 11375 s_waitcnt 0 ; Wait for all counters to be 0 11376 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11377 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11378 s_sethalt 9 11379 s_sleep 10 11380 s_sendmsg 0x1 11381 s_sendmsg sendmsg(MSG_INTERRUPT) 11382 s_trap 1 11383 11384For full list of supported instructions, refer to "SOPP Instructions" in ISA 11385Manual. 11386 11387Unless otherwise mentioned, little verification is performed on the operands 11388of SOPP Instructions, so it is up to the programmer to be familiar with the 11389range or acceptable values. 11390 11391VALU 11392++++ 11393 11394For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11395the assembler will automatically use optimal encoding based on its operands. To 11396force specific encoding, one can add a suffix to the opcode of the instruction: 11397 11398* _e32 for 32-bit VOP1/VOP2/VOPC 11399* _e64 for 64-bit VOP3 11400* _dpp for VOP_DPP 11401* _sdwa for VOP_SDWA 11402 11403VOP1/VOP2/VOP3/VOPC examples: 11404 11405.. code-block:: nasm 11406 11407 v_mov_b32 v1, v2 11408 v_mov_b32_e32 v1, v2 11409 v_nop 11410 v_cvt_f64_i32_e32 v[1:2], v2 11411 v_floor_f32_e32 v1, v2 11412 v_bfrev_b32_e32 v1, v2 11413 v_add_f32_e32 v1, v2, v3 11414 v_mul_i32_i24_e64 v1, v2, 3 11415 v_mul_i32_i24_e32 v1, -3, v3 11416 v_mul_i32_i24_e32 v1, -100, v3 11417 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11418 v_max_f16_e32 v1, v2, v3 11419 11420VOP_DPP examples: 11421 11422.. code-block:: nasm 11423 11424 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11425 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11426 v_mov_b32 v0, v0 wave_shl:1 11427 v_mov_b32 v0, v0 row_mirror 11428 v_mov_b32 v0, v0 row_bcast:31 11429 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11430 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11431 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11432 11433VOP_SDWA examples: 11434 11435.. code-block:: nasm 11436 11437 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11438 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11439 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11440 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11441 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11442 11443For full list of supported instructions, refer to "Vector ALU instructions". 11444 11445.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11446 11447Code Object V2 Predefined Symbols 11448~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11449 11450.. warning:: 11451 Code object V2 is not the default code object version emitted by 11452 this version of LLVM. 11453 11454The AMDGPU assembler defines and updates some symbols automatically. These 11455symbols do not affect code generation. 11456 11457.option.machine_version_major 11458+++++++++++++++++++++++++++++ 11459 11460Set to the GFX major generation number of the target being assembled for. For 11461example, when assembling for a "GFX9" target this will be set to the integer 11462value "9". The possible GFX major generation numbers are presented in 11463:ref:`amdgpu-processors`. 11464 11465.option.machine_version_minor 11466+++++++++++++++++++++++++++++ 11467 11468Set to the GFX minor generation number of the target being assembled for. For 11469example, when assembling for a "GFX810" target this will be set to the integer 11470value "1". The possible GFX minor generation numbers are presented in 11471:ref:`amdgpu-processors`. 11472 11473.option.machine_version_stepping 11474++++++++++++++++++++++++++++++++ 11475 11476Set to the GFX stepping generation number of the target being assembled for. 11477For example, when assembling for a "GFX704" target this will be set to the 11478integer value "4". The possible GFX stepping generation numbers are presented 11479in :ref:`amdgpu-processors`. 11480 11481.kernel.vgpr_count 11482++++++++++++++++++ 11483 11484Set to zero each time a 11485:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11486encountered. At each instruction, if the current value of this symbol is less 11487than or equal to the maximum VGPR number explicitly referenced within that 11488instruction then the symbol value is updated to equal that VGPR number plus 11489one. 11490 11491.kernel.sgpr_count 11492++++++++++++++++++ 11493 11494Set to zero each time a 11495:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11496encountered. At each instruction, if the current value of this symbol is less 11497than or equal to the maximum VGPR number explicitly referenced within that 11498instruction then the symbol value is updated to equal that SGPR number plus 11499one. 11500 11501.. _amdgpu-amdhsa-assembler-directives-v2: 11502 11503Code Object V2 Directives 11504~~~~~~~~~~~~~~~~~~~~~~~~~ 11505 11506.. warning:: 11507 Code object V2 is not the default code object version emitted by 11508 this version of LLVM. 11509 11510AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11511one can specify them with assembler directives. 11512 11513.hsa_code_object_version major, minor 11514+++++++++++++++++++++++++++++++++++++ 11515 11516*major* and *minor* are integers that specify the version of the HSA code 11517object that will be generated by the assembler. 11518 11519.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11520+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11521 11522 11523*major*, *minor*, and *stepping* are all integers that describe the instruction 11524set architecture (ISA) version of the assembly program. 11525 11526*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11527"AMD" and *arch* should always be equal to "AMDGPU". 11528 11529By default, the assembler will derive the ISA version, *vendor*, and *arch* 11530from the value of the -mcpu option that is passed to the assembler. 11531 11532.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11533 11534.amdgpu_hsa_kernel (name) 11535+++++++++++++++++++++++++ 11536 11537This directives specifies that the symbol with given name is a kernel entry 11538point (label) and the object should contain corresponding symbol of type 11539STT_AMDGPU_HSA_KERNEL. 11540 11541.amd_kernel_code_t 11542++++++++++++++++++ 11543 11544This directive marks the beginning of a list of key / value pairs that are used 11545to specify the amd_kernel_code_t object that will be emitted by the assembler. 11546The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11547amd_kernel_code_t values that are unspecified a default value will be used. The 11548default value for all keys is 0, with the following exceptions: 11549 11550- *amd_code_version_major* defaults to 1. 11551- *amd_kernel_code_version_minor* defaults to 2. 11552- *amd_machine_kind* defaults to 1. 11553- *amd_machine_version_major*, *machine_version_minor*, and 11554 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11555 that is passed to the assembler. 11556- *kernel_code_entry_byte_offset* defaults to 256. 11557- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11558 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11559 Note that wavefront size is specified as a power of two, so a value of **n** 11560 means a size of 2^ **n**. 11561- *call_convention* defaults to -1. 11562- *kernarg_segment_alignment*, *group_segment_alignment*, and 11563 *private_segment_alignment* default to 4. Note that alignments are specified 11564 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11565- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11566 GFX90A onwards. 11567- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11568 GFX10 onwards. 11569- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11570 11571The *.amd_kernel_code_t* directive must be placed immediately after the 11572function label and before any instructions. 11573 11574For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11575comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11576 11577.. _amdgpu-amdhsa-assembler-example-v2: 11578 11579Code Object V2 Example Source Code 11580~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11581 11582.. warning:: 11583 Code Object V2 is not the default code object version emitted by 11584 this version of LLVM. 11585 11586Here is an example of a minimal assembly source file, defining one HSA kernel: 11587 11588.. code:: 11589 :number-lines: 11590 11591 .hsa_code_object_version 1,0 11592 .hsa_code_object_isa 11593 11594 .hsatext 11595 .globl hello_world 11596 .p2align 8 11597 .amdgpu_hsa_kernel hello_world 11598 11599 hello_world: 11600 11601 .amd_kernel_code_t 11602 enable_sgpr_kernarg_segment_ptr = 1 11603 is_ptr64 = 1 11604 compute_pgm_rsrc1_vgprs = 0 11605 compute_pgm_rsrc1_sgprs = 0 11606 compute_pgm_rsrc2_user_sgpr = 2 11607 compute_pgm_rsrc1_wgp_mode = 0 11608 compute_pgm_rsrc1_mem_ordered = 0 11609 compute_pgm_rsrc1_fwd_progress = 1 11610 .end_amd_kernel_code_t 11611 11612 s_load_dwordx2 s[0:1], s[0:1] 0x0 11613 v_mov_b32 v0, 3.14159 11614 s_waitcnt lgkmcnt(0) 11615 v_mov_b32 v1, s0 11616 v_mov_b32 v2, s1 11617 flat_store_dword v[1:2], v0 11618 s_endpgm 11619 .Lfunc_end0: 11620 .size hello_world, .Lfunc_end0-hello_world 11621 11622.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11623 11624Code Object V3 to V4 Predefined Symbols 11625~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11626 11627The AMDGPU assembler defines and updates some symbols automatically. These 11628symbols do not affect code generation. 11629 11630.amdgcn.gfx_generation_number 11631+++++++++++++++++++++++++++++ 11632 11633Set to the GFX major generation number of the target being assembled for. For 11634example, when assembling for a "GFX9" target this will be set to the integer 11635value "9". The possible GFX major generation numbers are presented in 11636:ref:`amdgpu-processors`. 11637 11638.amdgcn.gfx_generation_minor 11639++++++++++++++++++++++++++++ 11640 11641Set to the GFX minor generation number of the target being assembled for. For 11642example, when assembling for a "GFX810" target this will be set to the integer 11643value "1". The possible GFX minor generation numbers are presented in 11644:ref:`amdgpu-processors`. 11645 11646.amdgcn.gfx_generation_stepping 11647+++++++++++++++++++++++++++++++ 11648 11649Set to the GFX stepping generation number of the target being assembled for. 11650For example, when assembling for a "GFX704" target this will be set to the 11651integer value "4". The possible GFX stepping generation numbers are presented 11652in :ref:`amdgpu-processors`. 11653 11654.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 11655 11656.amdgcn.next_free_vgpr 11657++++++++++++++++++++++ 11658 11659Set to zero before assembly begins. At each instruction, if the current value 11660of this symbol is less than or equal to the maximum VGPR number explicitly 11661referenced within that instruction then the symbol value is updated to equal 11662that VGPR number plus one. 11663 11664May be used to set the `.amdhsa_next_free_vgpr` directive in 11665:ref:`amdhsa-kernel-directives-table`. 11666 11667May be set at any time, e.g. manually set to zero at the start of each kernel. 11668 11669.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 11670 11671.amdgcn.next_free_sgpr 11672++++++++++++++++++++++ 11673 11674Set to zero before assembly begins. At each instruction, if the current value 11675of this symbol is less than or equal the maximum SGPR number explicitly 11676referenced within that instruction then the symbol value is updated to equal 11677that SGPR number plus one. 11678 11679May be used to set the `.amdhsa_next_free_spgr` directive in 11680:ref:`amdhsa-kernel-directives-table`. 11681 11682May be set at any time, e.g. manually set to zero at the start of each kernel. 11683 11684.. _amdgpu-amdhsa-assembler-directives-v3-v4: 11685 11686Code Object V3 to V4 Directives 11687~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11688 11689Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 11690architecture processors, and are not OS-specific. Directives which begin with 11691``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 11692``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 11693:ref:`amdgpu-processors`. 11694 11695.. _amdgpu-assembler-directive-amdgcn-target: 11696 11697.amdgcn_target <target-triple> "-" <target-id> 11698++++++++++++++++++++++++++++++++++++++++++++++ 11699 11700Optional directive which declares the ``<target-triple>-<target-id>`` supported 11701by the containing assembler source file. Used by the assembler to validate 11702command-line options such as ``-triple``, ``-mcpu``, and 11703``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 11704:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 11705 11706.. note:: 11707 11708 The target ID syntax used for code object V2 to V3 for this directive differs 11709 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 11710 11711.amdhsa_kernel <name> 11712+++++++++++++++++++++ 11713 11714Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 11715``<name>.kd``, in the current location of the current section. Only valid when 11716the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 11717instruction to execute, and does not need to be previously defined. 11718 11719Marks the beginning of a list of directives used to generate the bytes of a 11720kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 11721Directives which may appear in this list are described in 11722:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 11723be valid for the target being assembled for, and cannot be repeated. Directives 11724support the range of values specified by the field they reference in 11725:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 11726assumed to have its default value, unless it is marked as "Required", in which 11727case it is an error to omit the directive. This list of directives is 11728terminated by an ``.end_amdhsa_kernel`` directive. 11729 11730 .. table:: AMDHSA Kernel Assembler Directives 11731 :name: amdhsa-kernel-directives-table 11732 11733 ======================================================== =================== ============ =================== 11734 Directive Default Supported On Description 11735 ======================================================== =================== ============ =================== 11736 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 11737 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11738 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 11739 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11740 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 11741 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11742 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 11743 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11744 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 11745 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11746 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 11747 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11748 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 11749 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11750 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 11751 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11752 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 11753 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11754 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 11755 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11756 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 11757 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11758 Specific 11759 (wavefrontsize64) 11760 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 11761 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11762 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 11763 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11764 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 11765 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11766 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 11767 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11768 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 11769 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11770 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 11771 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11772 Possible values are defined in 11773 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 11774 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 11775 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 11776 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11777 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 11778 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11779 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11780 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 11781 Used to calculate ACCUM_OFFSET in 11782 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11783 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 11784 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11785 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11786 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 11787 scratch memory. Used to calculate 11788 GRANULATED_WAVEFRONT_SGPR_COUNT in 11789 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11790 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 11791 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11792 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11793 (xnack) 11794 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 11795 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11796 Possible values are defined in 11797 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11798 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 11799 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11800 Possible values are defined in 11801 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11802 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 11803 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11804 Possible values are defined in 11805 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11806 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 11807 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11808 Possible values are defined in 11809 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11810 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 11811 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11812 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 11813 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11814 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 11815 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11816 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 11817 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11818 Specific 11819 (tgsplit) 11820 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 11821 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11822 Specific 11823 (cumode) 11824 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 11825 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11826 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 11827 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11828 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 11829 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11830 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 11831 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11832 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 11833 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11834 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 11835 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11836 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 11837 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11838 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 11839 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11840 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 11841 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11842 ======================================================== =================== ============ =================== 11843 11844.amdgpu_metadata 11845++++++++++++++++ 11846 11847Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 11848note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 11849 11850The contents must be in the [YAML]_ markup format, with the same structure and 11851semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 11852:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 11853 11854This directive is terminated by an ``.end_amdgpu_metadata`` directive. 11855 11856.. _amdgpu-amdhsa-assembler-example-v3-v4: 11857 11858Code Object V3 to V4 Example Source Code 11859~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11860 11861Here is an example of a minimal assembly source file, defining one HSA kernel: 11862 11863.. code:: 11864 :number-lines: 11865 11866 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 11867 11868 .text 11869 .globl hello_world 11870 .p2align 8 11871 .type hello_world,@function 11872 hello_world: 11873 s_load_dwordx2 s[0:1], s[0:1] 0x0 11874 v_mov_b32 v0, 3.14159 11875 s_waitcnt lgkmcnt(0) 11876 v_mov_b32 v1, s0 11877 v_mov_b32 v2, s1 11878 flat_store_dword v[1:2], v0 11879 s_endpgm 11880 .Lfunc_end0: 11881 .size hello_world, .Lfunc_end0-hello_world 11882 11883 .rodata 11884 .p2align 6 11885 .amdhsa_kernel hello_world 11886 .amdhsa_user_sgpr_kernarg_segment_ptr 1 11887 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11888 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11889 .end_amdhsa_kernel 11890 11891 .amdgpu_metadata 11892 --- 11893 amdhsa.version: 11894 - 1 11895 - 0 11896 amdhsa.kernels: 11897 - .name: hello_world 11898 .symbol: hello_world.kd 11899 .kernarg_segment_size: 48 11900 .group_segment_fixed_size: 0 11901 .private_segment_fixed_size: 0 11902 .kernarg_segment_align: 4 11903 .wavefront_size: 64 11904 .sgpr_count: 2 11905 .vgpr_count: 3 11906 .max_flat_workgroup_size: 256 11907 ... 11908 .end_amdgpu_metadata 11909 11910If an assembly source file contains multiple kernels and/or functions, the 11911:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 11912:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 11913the ``.set <symbol>, <expression>`` directive. For example, in the case of two 11914kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 11915to group the function with the kernel that calls it and reset the symbols 11916between the two connected components: 11917 11918.. code:: 11919 :number-lines: 11920 11921 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 11922 11923 // gpr tracking symbols are implicitly set to zero 11924 11925 .text 11926 .globl kern0 11927 .p2align 8 11928 .type kern0,@function 11929 kern0: 11930 // ... 11931 s_endpgm 11932 .Lkern0_end: 11933 .size kern0, .Lkern0_end-kern0 11934 11935 .rodata 11936 .p2align 6 11937 .amdhsa_kernel kern0 11938 // ... 11939 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11940 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11941 .end_amdhsa_kernel 11942 11943 // reset symbols to begin tracking usage in func1 and kern1 11944 .set .amdgcn.next_free_vgpr, 0 11945 .set .amdgcn.next_free_sgpr, 0 11946 11947 .text 11948 .hidden func1 11949 .global func1 11950 .p2align 2 11951 .type func1,@function 11952 func1: 11953 // ... 11954 s_setpc_b64 s[30:31] 11955 .Lfunc1_end: 11956 .size func1, .Lfunc1_end-func1 11957 11958 .globl kern1 11959 .p2align 8 11960 .type kern1,@function 11961 kern1: 11962 // ... 11963 s_getpc_b64 s[4:5] 11964 s_add_u32 s4, s4, func1@rel32@lo+4 11965 s_addc_u32 s5, s5, func1@rel32@lo+4 11966 s_swappc_b64 s[30:31], s[4:5] 11967 // ... 11968 s_endpgm 11969 .Lkern1_end: 11970 .size kern1, .Lkern1_end-kern1 11971 11972 .rodata 11973 .p2align 6 11974 .amdhsa_kernel kern1 11975 // ... 11976 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11977 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11978 .end_amdhsa_kernel 11979 11980These symbols cannot identify connected components in order to automatically 11981track the usage for each kernel. However, in some cases careful organization of 11982the kernels and functions in the source file means there is minimal additional 11983effort required to accurately calculate GPR usage. 11984 11985Additional Documentation 11986======================== 11987 11988.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 11989.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 11990.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 11991.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 11992.. [AMD-GCN-GFX10-RDNA1] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 11993.. [AMD-GCN-GFX10-RDNA2] `AMD "RDNA 2" Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 11994.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 11995.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 11996.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 11997.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 11998.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 11999.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12000.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12001.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12002.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12003.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12004.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12005.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12006.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12007.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12008.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12009.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12010