1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 42to specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as: 73 74 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 75 loader on Linux. See *AMD ROCm Platform Release Notes* 76 [AMD-ROCm-Release-Notes]_ for supported hardware and 77 software. 78 - AMD's PAL runtime using the *pal-amdhsa* loader on 79 Windows. 80 81 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 82 runtime using the *pal-amdpal* loader on Windows and Linux 83 Pro. 84 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 85 3D runtime using the *mesa-mesa3d* loader on Linux. 86 ============== ============================================================ 87 88 .. table:: AMDGPU Environments 89 :name: amdgpu-environment-table 90 91 ============ ============================================================== 92 Environment Description 93 ============ ============================================================== 94 *<empty>* Default. 95 ============ ============================================================== 96 97.. _amdgpu-processors: 98 99Processors 100---------- 101 102Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 103specify the AMDGPU processor together with optional target features. See 104:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 105specific information. 106 107Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 108 109* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 110 111 112 .. table:: AMDGPU Processors 113 :name: amdgpu-processor-table 114 115 =========== =============== ============ ===== ================= =============== =============== ====================== 116 Processor Alternative Target dGPU/ Target Target OS Support Example 117 Processor Triple APU Features Properties *(see* Products 118 Architecture Supported `amdgpu-os`_ 119 *and 120 corresponding 121 runtime release 122 notes for 123 current 124 information and 125 level of 126 support)* 127 =========== =============== ============ ===== ================= =============== =============== ====================== 128 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 129 ----------------------------------------------------------------------------------------------------------------------- 130 ``r600`` ``r600`` dGPU - Does not 131 support 132 generic 133 address 134 space 135 ``r630`` ``r600`` dGPU - Does not 136 support 137 generic 138 address 139 space 140 ``rs880`` ``r600`` dGPU - Does not 141 support 142 generic 143 address 144 space 145 ``rv670`` ``r600`` dGPU - Does not 146 support 147 generic 148 address 149 space 150 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 151 ----------------------------------------------------------------------------------------------------------------------- 152 ``rv710`` ``r600`` dGPU - Does not 153 support 154 generic 155 address 156 space 157 ``rv730`` ``r600`` dGPU - Does not 158 support 159 generic 160 address 161 space 162 ``rv770`` ``r600`` dGPU - Does not 163 support 164 generic 165 address 166 space 167 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 168 ----------------------------------------------------------------------------------------------------------------------- 169 ``cedar`` ``r600`` dGPU - Does not 170 support 171 generic 172 address 173 space 174 ``cypress`` ``r600`` dGPU - Does not 175 support 176 generic 177 address 178 space 179 ``juniper`` ``r600`` dGPU - Does not 180 support 181 generic 182 address 183 space 184 ``redwood`` ``r600`` dGPU - Does not 185 support 186 generic 187 address 188 space 189 ``sumo`` ``r600`` dGPU - Does not 190 support 191 generic 192 address 193 space 194 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 195 ----------------------------------------------------------------------------------------------------------------------- 196 ``barts`` ``r600`` dGPU - Does not 197 support 198 generic 199 address 200 space 201 ``caicos`` ``r600`` dGPU - Does not 202 support 203 generic 204 address 205 space 206 ``cayman`` ``r600`` dGPU - Does not 207 support 208 generic 209 address 210 space 211 ``turks`` ``r600`` dGPU - Does not 212 support 213 generic 214 address 215 space 216 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 217 ----------------------------------------------------------------------------------------------------------------------- 218 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 219 support 220 generic 221 address 222 space 223 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 224 - ``verde`` support 225 generic 226 address 227 space 228 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 229 - ``oland`` support 230 generic 231 address 232 space 233 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 234 ----------------------------------------------------------------------------------------------------------------------- 235 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 236 flat - *pal-amdhsa* - A6 Pro-7050B 237 scratch - *pal-amdpal* - A8-7100 238 - A8 Pro-7150B 239 - A10-7300 240 - A10 Pro-7350B 241 - FX-7500 242 - A8-7200P 243 - A10-7400P 244 - FX-7600P 245 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 246 flat - *pal-amdhsa* - FirePro W9100 247 scratch - *pal-amdpal* - FirePro S9150 248 - FirePro S9170 249 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 250 flat - *pal-amdhsa* - Radeon R9 290x 251 scratch - *pal-amdpal* - Radeon R390 252 - Radeon R390x 253 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 254 - ``mullins`` flat - *pal-amdpal* - E1-2200 255 scratch - E1-2500 256 - E2-3000 257 - E2-3800 258 - A4-5000 259 - A4-5100 260 - A6-5200 261 - A4 Pro-3340B 262 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 263 flat - *pal-amdpal* - Radeon HD 8770 264 scratch - R7 260 265 - R7 260X 266 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 267 flat - *pal-amdpal* 268 scratch .. TODO:: 269 270 Add product 271 names. 272 273 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 274 ----------------------------------------------------------------------------------------------------------------------- 275 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 276 flat - *pal-amdhsa* - Pro A6-8500B 277 scratch - *pal-amdpal* - A8-8600P 278 - Pro A8-8600B 279 - FX-8800P 280 - Pro A12-8800B 281 - A10-8700P 282 - Pro A10-8700B 283 - A10-8780P 284 - A10-9600P 285 - A10-9630P 286 - A12-9700P 287 - A12-9730P 288 - FX-9800P 289 - FX-9830P 290 - E2-9010 291 - A6-9210 292 - A9-9410 293 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 294 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 295 scratch - *pal-amdpal* - Radeon R9 385 296 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 297 - *pal-amdhsa* - Radeon R9 Fury 298 - *pal-amdpal* - Radeon R9 FuryX 299 - Radeon Pro Duo 300 - FirePro S9300x2 301 - Radeon Instinct MI8 302 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 303 flat - *pal-amdhsa* - Radeon RX 480 304 scratch - *pal-amdpal* - Radeon Instinct MI6 305 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 306 flat - *pal-amdhsa* 307 scratch - *pal-amdpal* 308 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 309 flat - *pal-amdhsa* - FirePro S7100 310 scratch - *pal-amdpal* - FirePro W7100 311 - Mobile FirePro 312 M7170 313 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 314 flat - *pal-amdhsa* 315 scratch - *pal-amdpal* .. TODO:: 316 317 Add product 318 names. 319 320 **GCN GFX9 (Vega)** [AMD-GCN-GFX9]_ 321 ----------------------------------------------------------------------------------------------------------------------- 322 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 323 flat - *pal-amdhsa* Frontier Edition 324 scratch - *pal-amdpal* - Radeon RX Vega 56 325 - Radeon RX Vega 64 326 - Radeon RX Vega 64 327 Liquid 328 - Radeon Instinct MI25 329 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 330 flat - *pal-amdhsa* - Ryzen 5 2400G 331 scratch - *pal-amdpal* 332 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 333 - *pal-amdhsa* 334 - *pal-amdpal* .. TODO:: 335 336 Add product 337 names. 338 339 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 340 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 341 scratch - *pal-amdpal* - Radeon VII 342 - Radeon Pro VII 343 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* *TBA* 344 - xnack - Absolute 345 flat .. TODO:: 346 scratch 347 Add product 348 names. 349 350 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 351 flat 352 scratch .. TODO:: 353 354 Add product 355 names. 356 357 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 358 - tgsplit flat 359 - xnack scratch .. TODO:: 360 - Packed 361 work-item Add product 362 IDs names. 363 364 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 365 flat - Ryzen 7 4700GE 366 scratch - Ryzen 5 4600G 367 - Ryzen 5 4600GE 368 - Ryzen 3 4300G 369 - Ryzen 3 4300GE 370 - Ryzen Pro 4000G 371 - Ryzen 7 Pro 4700G 372 - Ryzen 7 Pro 4750GE 373 - Ryzen 5 Pro 4650G 374 - Ryzen 5 Pro 4650GE 375 - Ryzen 3 Pro 4350G 376 - Ryzen 3 Pro 4350GE 377 378 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 379 ----------------------------------------------------------------------------------------------------------------------- 380 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 381 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 382 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 383 - Radeon Pro 5600M 384 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* *TBA* 385 - wavefrontsize64 - Absolute - *pal-amdhsa* 386 - xnack flat - *pal-amdpal* 387 scratch .. TODO:: 388 389 Add product 390 names. 391 392 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 393 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 394 - xnack scratch - *pal-amdpal* 395 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 396 ----------------------------------------------------------------------------------------------------------------------- 397 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 398 - wavefrontsize64 flat - *pal-amdhsa* 399 scratch - *pal-amdpal* .. TODO:: 400 401 Add product 402 names. 403 404 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 405 - wavefrontsize64 flat - *pal-amdhsa* 406 scratch - *pal-amdpal* .. TODO:: 407 408 Add product 409 names. 410 411 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 412 - wavefrontsize64 flat - *pal-amdhsa* 413 scratch - *pal-amdpal* .. TODO:: 414 415 Add product 416 names. 417 418 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 419 - wavefrontsize64 flat 420 scratch .. TODO:: 421 422 Add product 423 names. 424 425 =========== =============== ============ ===== ================= =============== =============== ====================== 426 427.. _amdgpu-target-features: 428 429Target Features 430--------------- 431 432Target features control how code is generated to support certain 433processor specific features. Not all target features are supported by 434all processors. The runtime must ensure that the features supported by 435the device used to execute the code match the features enabled when 436generating the code. A mismatch of features may result in incorrect 437execution, or a reduction in performance. 438 439The target features supported by each processor is listed in 440:ref:`amdgpu-processor-table`. 441 442Target features are controlled by exactly one of the following Clang 443options: 444 445``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 446 447 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 448 optional components of the target ID. If omitted, the target feature has the 449 ``any`` value. See :ref:`amdgpu-target-id`. 450 451``-m[no-]<target-feature>`` 452 453 Target features not specified by the target ID are specified using a 454 separate option. These target features can have an ``on`` or ``off`` 455 value. ``on`` is specified by omitting the ``no-`` prefix, and 456 ``off`` is specified by including the ``no-`` prefix. The default 457 if not specified is ``off``. 458 459For example: 460 461``-mcpu=gfx908:xnack+`` 462 Enable the ``xnack`` feature. 463``-mcpu=gfx908:xnack-`` 464 Disable the ``xnack`` feature. 465``-mcumode`` 466 Enable the ``cumode`` feature. 467``-mno-cumode`` 468 Disable the ``cumode`` feature. 469 470 .. table:: AMDGPU Target Features 471 :name: amdgpu-target-features-table 472 473 =============== ============================ ================================================== 474 Target Feature Clang Option to Control Description 475 Name 476 =============== ============================ ================================================== 477 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 478 when generating code for kernels. When disabled 479 native WGP wavefront execution mode is used, 480 when enabled CU wavefront execution mode is used 481 (see :ref:`amdgpu-amdhsa-memory-model`). 482 483 sramecc - ``-mcpu`` If specified, generate code that can only be 484 - ``--offload-arch`` loaded and executed in a process that has a 485 matching setting for SRAMECC. 486 487 If not specified, generate code that can be 488 loaded and executed in a process with either 489 setting of SRAMECC. 490 491 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 492 work-groups are launched in threadgroup split mode. 493 When enabled the waves of a work-group may be 494 launched in different CUs. 495 496 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 497 generating code for kernels. When disabled 498 native wavefront size 32 is used, when enabled 499 wavefront size 64 is used. 500 501 xnack - ``-mcpu`` If specified, generate code that can only be 502 - ``--offload-arch`` loaded and executed in a process that has a 503 matching setting for XNACK replay. 504 505 If not specified, generate code that can be 506 loaded and executed in a process with either 507 setting of XNACK replay. 508 509 This is used for demand paging and page 510 migration. If XNACK replay is enabled in 511 the device, then if a page fault occurs 512 the code may execute incorrectly if the 513 ``xnack`` feature is not enabled. Executing 514 code that has the feature enabled on a 515 device that does not have XNACK replay 516 enabled will execute correctly but may 517 be less performant than code with the 518 feature disabled. 519 =============== ============================ ================================================== 520 521.. _amdgpu-target-id: 522 523Target ID 524--------- 525 526AMDGPU supports target IDs. See `Clang Offload Bundler 527<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 528description. The AMDGPU target specific information is: 529 530**processor** 531 Is a AMDGPU processor or alternative processor name specified in 532 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 533 the primary processor and alternative processor names. The canonical form 534 target ID only allow the primary processor name. 535 536**target-feature** 537 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 538 is supported by the processor. The target features supported by each processor 539 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 540 a target ID are marked as being controlled by ``-mcpu`` and 541 ``--offload-arch``. Each target feature must appear at most once in a target 542 ID. The non-canonical form target ID allows the target features to be 543 specified in any order. The canonical form target ID requires the target 544 features to be specified in alphabetic order. 545 546.. _amdgpu-target-id-v2-v3: 547 548Code Object V2 to V3 Target ID 549~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 550 551The target ID syntax for code object V2 to V3 is the same as defined in `Clang 552Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 553when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 554directive and the bundle entry ID. In those cases it has the following BNF 555syntax: 556 557.. code:: 558 559 <target-id> ::== <processor> ( "+" <target-feature> )* 560 561Where a target feature is omitted if *Off* and present if *On* or *Any*. 562 563.. note:: 564 565 The code object V2 to V3 cannot represent *Any* and treats it the same as 566 *On*. 567 568.. _amdgpu-embedding-bundled-objects: 569 570Embedding Bundled Code Objects 571------------------------------ 572 573AMDGPU supports the HIP and OpenMP languages that perform code object embedding 574as described in `Clang Offload Bundler 575<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 576 577.. note:: 578 579 The target ID syntax used for code object V2 to V3 for a bundle entry ID 580 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 581 582.. _amdgpu-address-spaces: 583 584Address Spaces 585-------------- 586 587The AMDGPU architecture supports a number of memory address spaces. The address 588space names use the OpenCL standard names, with some additions. 589 590The AMDGPU address spaces correspond to target architecture specific LLVM 591address space numbers used in LLVM IR. 592 593The AMDGPU address spaces are described in 594:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 595supported for the ``amdgcn`` target. 596 597 .. table:: AMDGPU Address Spaces 598 :name: amdgpu-address-spaces-table 599 600 ================================= =============== =========== ================ ======= ============================ 601 .. 64-Bit Process Address Space 602 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 603 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 604 Space Number Name Name Size 605 ================================= =============== =========== ================ ======= ============================ 606 Generic 0 flat flat 64 0x0000000000000000 607 Global 1 global global 64 0x0000000000000000 608 Region 2 N/A GDS 32 *not implemented for AMDHSA* 609 Local 3 group LDS 32 0xFFFFFFFF 610 Constant 4 constant *same as global* 64 0x0000000000000000 611 Private 5 private scratch 32 0xFFFFFFFF 612 Constant 32-bit 6 *TODO* 0x00000000 613 Buffer Fat Pointer (experimental) 7 *TODO* 614 ================================= =============== =========== ================ ======= ============================ 615 616**Generic** 617 The generic address space is supported unless the *Target Properties* column 618 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 619 space*. 620 621 The generic address space uses the hardware flat address support for two fixed 622 ranges of virtual addresses (the private and local apertures), that are 623 outside the range of addressable global memory, to map from a flat address to 624 a private or local address. This uses FLAT instructions that can take a flat 625 address and access global, private (scratch), and group (LDS) memory depending 626 on if the address is within one of the aperture ranges. 627 628 Flat access to scratch requires hardware aperture setup and setup in the 629 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 630 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 631 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 632 633 To convert between a private or group address space address (termed a segment 634 address) and a flat address the base address of the corresponding aperture 635 can be used. For GFX7-GFX8 these are available in the 636 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 637 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 638 GFX9-GFX10 the aperture base addresses are directly available as inline 639 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 640 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 641 aligned to 2^32 which makes it easier to convert from flat to segment or 642 segment to flat. 643 644 A global address space address has the same value when used as a flat address 645 so no conversion is needed. 646 647**Global and Constant** 648 The global and constant address spaces both use global virtual addresses, 649 which are the same virtual address space used by the CPU. However, some 650 virtual addresses may only be accessible to the CPU, some only accessible 651 by the GPU, and some by both. 652 653 Using the constant address space indicates that the data will not change 654 during the execution of the kernel. This allows scalar read instructions to 655 be used. As the constant address space could only be modified on the host 656 side, a generic pointer loaded from the constant address space is safe to be 657 assumed as a global pointer since only the device global memory is visible 658 and managed on the host side. The vector and scalar L1 caches are invalidated 659 of volatile data before each kernel dispatch execution to allow constant 660 memory to change values between kernel dispatches. 661 662**Region** 663 The region address space uses the hardware Global Data Store (GDS). All 664 wavefronts executing on the same device will access the same memory for any 665 given region address. However, the same region address accessed by wavefronts 666 executing on different devices will access different memory. It is higher 667 performance than global memory. It is allocated by the runtime. The data 668 store (DS) instructions can be used to access it. 669 670**Local** 671 The local address space uses the hardware Local Data Store (LDS) which is 672 automatically allocated when the hardware creates the wavefronts of a 673 work-group, and freed when all the wavefronts of a work-group have 674 terminated. All wavefronts belonging to the same work-group will access the 675 same memory for any given local address. However, the same local address 676 accessed by wavefronts belonging to different work-groups will access 677 different memory. It is higher performance than global memory. The data store 678 (DS) instructions can be used to access it. 679 680**Private** 681 The private address space uses the hardware scratch memory support which 682 automatically allocates memory when it creates a wavefront and frees it when 683 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 684 given private address will be different to the memory accessed by another lane 685 of the same or different wavefront for the same private address. 686 687 If a kernel dispatch uses scratch, then the hardware allocates memory from a 688 pool of backing memory allocated by the runtime for each wavefront. The lanes 689 of the wavefront access this using dword (4 byte) interleaving. The mapping 690 used from private address to backing memory address is: 691 692 ``wavefront-scratch-base + 693 ((private-address / 4) * wavefront-size * 4) + 694 (wavefront-lane-id * 4) + (private-address % 4)`` 695 696 If each lane of a wavefront accesses the same private address, the 697 interleaving results in adjacent dwords being accessed and hence requires 698 fewer cache lines to be fetched. 699 700 There are different ways that the wavefront scratch base address is 701 determined by a wavefront (see 702 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 703 704 Scratch memory can be accessed in an interleaved manner using buffer 705 instructions with the scratch buffer descriptor and per wavefront scratch 706 offset, by the scratch instructions, or by flat instructions. Multi-dword 707 access is not supported except by flat and scratch instructions in 708 GFX9-GFX10. 709 710**Constant 32-bit** 711 *TODO* 712 713**Buffer Fat Pointer** 714 The buffer fat pointer is an experimental address space that is currently 715 unsupported in the backend. It exposes a non-integral pointer that is in 716 the future intended to support the modelling of 128-bit buffer descriptors 717 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 718 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 719 model the buffer descriptors used heavily in graphics workloads targeting 720 the backend. 721 722.. _amdgpu-memory-scopes: 723 724Memory Scopes 725------------- 726 727This section provides LLVM memory synchronization scopes supported by the AMDGPU 728backend memory model when the target triple OS is ``amdhsa`` (see 729:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 730 731The memory model supported is based on the HSA memory model [HSA]_ which is 732based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 733relation is transitive over the synchronizes-with relation independent of scope 734and synchronizes-with allows the memory scope instances to be inclusive (see 735table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 736 737This is different to the OpenCL [OpenCL]_ memory model which does not have scope 738inclusion and requires the memory scopes to exactly match. However, this 739is conservatively correct for OpenCL. 740 741 .. table:: AMDHSA LLVM Sync Scopes 742 :name: amdgpu-amdhsa-llvm-sync-scopes-table 743 744 ======================= =================================================== 745 LLVM Sync Scope Description 746 ======================= =================================================== 747 *none* The default: ``system``. 748 749 Synchronizes with, and participates in modification 750 and seq_cst total orderings with, other operations 751 (except image operations) for all address spaces 752 (except private, or generic that accesses private) 753 provided the other operation's sync scope is: 754 755 - ``system``. 756 - ``agent`` and executed by a thread on the same 757 agent. 758 - ``workgroup`` and executed by a thread in the 759 same work-group. 760 - ``wavefront`` and executed by a thread in the 761 same wavefront. 762 763 ``agent`` Synchronizes with, and participates in modification 764 and seq_cst total orderings with, other operations 765 (except image operations) for all address spaces 766 (except private, or generic that accesses private) 767 provided the other operation's sync scope is: 768 769 - ``system`` or ``agent`` and executed by a thread 770 on the same agent. 771 - ``workgroup`` and executed by a thread in the 772 same work-group. 773 - ``wavefront`` and executed by a thread in the 774 same wavefront. 775 776 ``workgroup`` Synchronizes with, and participates in modification 777 and seq_cst total orderings with, other operations 778 (except image operations) for all address spaces 779 (except private, or generic that accesses private) 780 provided the other operation's sync scope is: 781 782 - ``system``, ``agent`` or ``workgroup`` and 783 executed by a thread in the same work-group. 784 - ``wavefront`` and executed by a thread in the 785 same wavefront. 786 787 ``wavefront`` Synchronizes with, and participates in modification 788 and seq_cst total orderings with, other operations 789 (except image operations) for all address spaces 790 (except private, or generic that accesses private) 791 provided the other operation's sync scope is: 792 793 - ``system``, ``agent``, ``workgroup`` or 794 ``wavefront`` and executed by a thread in the 795 same wavefront. 796 797 ``singlethread`` Only synchronizes with and participates in 798 modification and seq_cst total orderings with, 799 other operations (except image operations) running 800 in the same thread for all address spaces (for 801 example, in signal handlers). 802 803 ``one-as`` Same as ``system`` but only synchronizes with other 804 operations within the same address space. 805 806 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 807 operations within the same address space. 808 809 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 810 other operations within the same address space. 811 812 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 813 other operations within the same address space. 814 815 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 816 other operations within the same address space. 817 ======================= =================================================== 818 819LLVM IR Intrinsics 820------------------ 821 822The AMDGPU backend implements the following LLVM IR intrinsics. 823 824*This section is WIP.* 825 826.. TODO:: 827 828 List AMDGPU intrinsics. 829 830LLVM IR Attributes 831------------------ 832 833The AMDGPU backend supports the following LLVM IR attributes. 834 835 .. table:: AMDGPU LLVM IR Attributes 836 :name: amdgpu-llvm-ir-attributes-table 837 838 ======================================= ========================================================== 839 LLVM Attribute Description 840 ======================================= ========================================================== 841 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 842 will be specified when the kernel is dispatched. Generated 843 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 844 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 845 argument block size for the implicit arguments. This 846 varies by OS and language (for OpenCL see 847 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 848 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 849 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 850 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 851 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 852 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 853 execution unit. Generated by the ``amdgpu_waves_per_eu`` 854 CLANG attribute [CLANG-ATTR]_. 855 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 856 mode register to be set on entry. Overrides the default for 857 the calling convention. 858 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 859 the mode register to be set on entry. Overrides the default 860 for the calling convention. 861 ======================================= ========================================================== 862 863.. _amdgpu-elf-code-object: 864 865ELF Code Object 866=============== 867 868The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 869can be linked by ``lld`` to produce a standard ELF shared code object which can 870be loaded and executed on an AMDGPU target. 871 872.. _amdgpu-elf-header: 873 874Header 875------ 876 877The AMDGPU backend uses the following ELF header: 878 879 .. table:: AMDGPU ELF Header 880 :name: amdgpu-elf-header-table 881 882 ========================== =============================== 883 Field Value 884 ========================== =============================== 885 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 886 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 887 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 888 - ``ELFOSABI_AMDGPU_HSA`` 889 - ``ELFOSABI_AMDGPU_PAL`` 890 - ``ELFOSABI_AMDGPU_MESA3D`` 891 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 892 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 893 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 894 - ``ELFABIVERSION_AMDGPU_PAL`` 895 - ``ELFABIVERSION_AMDGPU_MESA3D`` 896 ``e_type`` - ``ET_REL`` 897 - ``ET_DYN`` 898 ``e_machine`` ``EM_AMDGPU`` 899 ``e_entry`` 0 900 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 901 :ref:`amdgpu-elf-header-e_flags-table-v3`, 902 and :ref:`amdgpu-elf-header-e_flags-table-v4` 903 ========================== =============================== 904 905.. 906 907 .. table:: AMDGPU ELF Header Enumeration Values 908 :name: amdgpu-elf-header-enumeration-values-table 909 910 =============================== ===== 911 Name Value 912 =============================== ===== 913 ``EM_AMDGPU`` 224 914 ``ELFOSABI_NONE`` 0 915 ``ELFOSABI_AMDGPU_HSA`` 64 916 ``ELFOSABI_AMDGPU_PAL`` 65 917 ``ELFOSABI_AMDGPU_MESA3D`` 66 918 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 919 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 920 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 921 ``ELFABIVERSION_AMDGPU_PAL`` 0 922 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 923 =============================== ===== 924 925``e_ident[EI_CLASS]`` 926 The ELF class is: 927 928 * ``ELFCLASS32`` for ``r600`` architecture. 929 930 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 931 process address space applications. 932 933``e_ident[EI_DATA]`` 934 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 935 936``e_ident[EI_OSABI]`` 937 One of the following AMDGPU target architecture specific OS ABIs 938 (see :ref:`amdgpu-os`): 939 940 * ``ELFOSABI_NONE`` for *unknown* OS. 941 942 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 943 944 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 945 946 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 947 948``e_ident[EI_ABIVERSION]`` 949 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 950 object conforms: 951 952 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 953 runtime ABI for code object V2. Specify using the Clang option 954 ``-mcode-object-version=2``. 955 956 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 957 runtime ABI for code object V3. Specify using the Clang option 958 ``-mcode-object-version=3``. This is the default code object 959 version if not specified. 960 961 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 962 runtime ABI for code object V4. Specify using the Clang option 963 ``-mcode-object-version=4``. 964 965 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 966 runtime ABI. 967 968 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 969 3D runtime ABI. 970 971``e_type`` 972 Can be one of the following values: 973 974 975 ``ET_REL`` 976 The type produced by the AMDGPU backend compiler as it is relocatable code 977 object. 978 979 ``ET_DYN`` 980 The type produced by the linker as it is a shared code object. 981 982 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 983 984``e_machine`` 985 The value ``EM_AMDGPU`` is used for the machine for all processors supported 986 by the ``r600`` and ``amdgcn`` architectures (see 987 :ref:`amdgpu-processor-table`). The specific processor is specified in the 988 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 989 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 990 ``e_flags`` for code object V3 to V4 (see 991 :ref:`amdgpu-elf-header-e_flags-table-v3` and 992 :ref:`amdgpu-elf-header-e_flags-table-v4`). 993 994``e_entry`` 995 The entry point is 0 as the entry points for individual kernels must be 996 selected in order to invoke them through AQL packets. 997 998``e_flags`` 999 The AMDGPU backend uses the following ELF header flags: 1000 1001 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1002 :name: amdgpu-elf-header-e_flags-v2-table 1003 1004 ===================================== ===== ============================= 1005 Name Value Description 1006 ===================================== ===== ============================= 1007 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1008 target feature is 1009 enabled for all code 1010 contained in the code object. 1011 If the processor 1012 does not support the 1013 ``xnack`` target 1014 feature then must 1015 be 0. 1016 See 1017 :ref:`amdgpu-target-features`. 1018 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1019 handler is enabled for all 1020 code contained in the code 1021 object. If the processor 1022 does not support a trap 1023 handler then must be 0. 1024 See 1025 :ref:`amdgpu-target-features`. 1026 ===================================== ===== ============================= 1027 1028 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1029 :name: amdgpu-elf-header-e_flags-table-v3 1030 1031 ================================= ===== ============================= 1032 Name Value Description 1033 ================================= ===== ============================= 1034 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1035 mask for 1036 ``EF_AMDGPU_MACH_xxx`` values 1037 defined in 1038 :ref:`amdgpu-ef-amdgpu-mach-table`. 1039 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1040 target feature is 1041 enabled for all code 1042 contained in the code object. 1043 If the processor 1044 does not support the 1045 ``xnack`` target 1046 feature then must 1047 be 0. 1048 See 1049 :ref:`amdgpu-target-features`. 1050 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1051 target feature is 1052 enabled for all code 1053 contained in the code object. 1054 If the processor 1055 does not support the 1056 ``sramecc`` target 1057 feature then must 1058 be 0. 1059 See 1060 :ref:`amdgpu-target-features`. 1061 ================================= ===== ============================= 1062 1063 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1064 :name: amdgpu-elf-header-e_flags-table-v4 1065 1066 ============================================ ===== =================================== 1067 Name Value Description 1068 ============================================ ===== =================================== 1069 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1070 mask for 1071 ``EF_AMDGPU_MACH_xxx`` values 1072 defined in 1073 :ref:`amdgpu-ef-amdgpu-mach-table`. 1074 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1075 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1076 values. 1077 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1078 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1079 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1080 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1081 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1082 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1083 values. 1084 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1085 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1086 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1087 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1088 ============================================ ===== =================================== 1089 1090 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1091 :name: amdgpu-ef-amdgpu-mach-table 1092 1093 ==================================== ========== ============================= 1094 Name Value Description (see 1095 :ref:`amdgpu-processor-table`) 1096 ==================================== ========== ============================= 1097 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1098 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1099 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1100 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1101 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1102 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1103 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1104 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1105 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1106 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1107 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1108 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1109 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1110 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1111 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1112 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1113 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1114 *reserved* 0x011 - Reserved for ``r600`` 1115 0x01f architecture processors. 1116 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1117 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1118 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1119 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1120 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1121 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1122 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1123 *reserved* 0x027 Reserved. 1124 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1125 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1126 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1127 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1128 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1129 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1130 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1131 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1132 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1133 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1134 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1135 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1136 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1137 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1138 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1139 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1140 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1141 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1142 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1143 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1144 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1145 *reserved* 0x03d Reserved. 1146 *reserved* 0x03e Reserved. 1147 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1148 ==================================== ========== ============================= 1149 1150Sections 1151-------- 1152 1153An AMDGPU target ELF code object has the standard ELF sections which include: 1154 1155 .. table:: AMDGPU ELF Sections 1156 :name: amdgpu-elf-sections-table 1157 1158 ================== ================ ================================= 1159 Name Type Attributes 1160 ================== ================ ================================= 1161 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1162 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1163 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1164 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1165 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1166 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1167 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1168 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1169 ``.note`` ``SHT_NOTE`` *none* 1170 ``.rela``\ *name* ``SHT_RELA`` *none* 1171 ``.rela.dyn`` ``SHT_RELA`` *none* 1172 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1173 ``.shstrtab`` ``SHT_STRTAB`` *none* 1174 ``.strtab`` ``SHT_STRTAB`` *none* 1175 ``.symtab`` ``SHT_SYMTAB`` *none* 1176 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1177 ================== ================ ================================= 1178 1179These sections have their standard meanings (see [ELF]_) and are only generated 1180if needed. 1181 1182``.debug``\ *\** 1183 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1184 information on the DWARF produced by the AMDGPU backend. 1185 1186``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1187 The standard sections used by a dynamic loader. 1188 1189``.note`` 1190 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1191 backend. 1192 1193``.rela``\ *name*, ``.rela.dyn`` 1194 For relocatable code objects, *name* is the name of the section that the 1195 relocation records apply. For example, ``.rela.text`` is the section name for 1196 relocation records associated with the ``.text`` section. 1197 1198 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1199 records from each of the relocatable code object's ``.rela``\ *name* sections. 1200 1201 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1202 the AMDGPU backend. 1203 1204``.text`` 1205 The executable machine code for the kernels and functions they call. Generated 1206 as position independent code. See :ref:`amdgpu-code-conventions` for 1207 information on conventions used in the isa generation. 1208 1209.. _amdgpu-note-records: 1210 1211Note Records 1212------------ 1213 1214The AMDGPU backend code object contains ELF note records in the ``.note`` 1215section. The set of generated notes and their semantics depend on the code 1216object version; see :ref:`amdgpu-note-records-v2` and 1217:ref:`amdgpu-note-records-v3-v4`. 1218 1219As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1220must be generated after the ``name`` field to ensure the ``desc`` field is 4 1221byte aligned. In addition, minimal zero-byte padding must be generated to 1222ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1223field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1224alignment. 1225 1226.. _amdgpu-note-records-v2: 1227 1228Code Object V2 Note Records 1229~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1230 1231.. warning:: 1232 Code object V2 is not the default code object version emitted by 1233 this version of LLVM. 1234 1235The AMDGPU backend code object uses the following ELF note record in the 1236``.note`` section when compiling for code object V2. 1237 1238The note record vendor field is "AMD". 1239 1240Additional note records may be present, but any which are not documented here 1241are deprecated and should not be used. 1242 1243 .. table:: AMDGPU Code Object V2 ELF Note Records 1244 :name: amdgpu-elf-note-records-v2-table 1245 1246 ===== ===================================== ====================================== 1247 Name Type Description 1248 ===== ===================================== ====================================== 1249 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1250 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1251 Finalizer and not the LLVM compiler. 1252 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1253 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1254 YAML [YAML]_ textual format. 1255 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1256 ===== ===================================== ====================================== 1257 1258.. 1259 1260 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1261 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1262 1263 ===================================== ===== 1264 Name Value 1265 ===================================== ===== 1266 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1267 ``NT_AMD_HSA_HSAIL`` 2 1268 ``NT_AMD_HSA_ISA_VERSION`` 3 1269 *reserved* 4-9 1270 ``NT_AMD_HSA_METADATA`` 10 1271 ``NT_AMD_HSA_ISA_NAME`` 11 1272 ===================================== ===== 1273 1274``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1275 Specifies the code object version number. The description field has the 1276 following layout: 1277 1278 .. code:: 1279 1280 struct amdgpu_hsa_note_code_object_version_s { 1281 uint32_t major_version; 1282 uint32_t minor_version; 1283 }; 1284 1285 The ``major_version`` has a value less than or equal to 2. 1286 1287``NT_AMD_HSA_HSAIL`` 1288 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1289 field has the following layout: 1290 1291 .. code:: 1292 1293 struct amdgpu_hsa_note_hsail_s { 1294 uint32_t hsail_major_version; 1295 uint32_t hsail_minor_version; 1296 uint8_t profile; 1297 uint8_t machine_model; 1298 uint8_t default_float_round; 1299 }; 1300 1301``NT_AMD_HSA_ISA_VERSION`` 1302 Specifies the target ISA version. The description field has the following layout: 1303 1304 .. code:: 1305 1306 struct amdgpu_hsa_note_isa_s { 1307 uint16_t vendor_name_size; 1308 uint16_t architecture_name_size; 1309 uint32_t major; 1310 uint32_t minor; 1311 uint32_t stepping; 1312 char vendor_and_architecture_name[1]; 1313 }; 1314 1315 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1316 vendor and architecture names respectively, including the NUL character. 1317 1318 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1319 vendor, immediately followed by the NUL terminated string for the 1320 architecture. 1321 1322 This note record is used by the HSA runtime loader. 1323 1324 Code object V2 only supports a limited number of processors and has fixed 1325 settings for target features. See 1326 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1327 processors and the corresponding target ID. In the table the note record ISA 1328 name is a concatenation of the vendor name, architecture name, major, minor, 1329 and stepping separated by a ":". 1330 1331 The target ID column shows the processor name and fixed target features used 1332 by the LLVM compiler. The LLVM compiler does not generate a 1333 ``NT_AMD_HSA_HSAIL`` note record. 1334 1335 A code object generated by the Finalizer also uses code object V2 and always 1336 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1337 ``sramecc`` target feature is as shown in 1338 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1339 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1340 bit. 1341 1342``NT_AMD_HSA_ISA_NAME`` 1343 Specifies the target ISA name as a non-NUL terminated string. 1344 1345 This note record is not used by the HSA runtime loader. 1346 1347 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1348 V2's limited support of processors and fixed settings for target features. 1349 1350 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1351 from the string to the corresponding target ID. If the ``xnack`` target 1352 feature is supported and enabled, the string produced by the LLVM compiler 1353 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1354 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1355 1356``NT_AMD_HSA_METADATA`` 1357 Specifies extensible metadata associated with the code objects executed on HSA 1358 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1359 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1360 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1361 metadata string. 1362 1363 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1364 :name: amdgpu-elf-note-record-supported_processors-v2-table 1365 1366 ==================== ========================== 1367 Note Record ISA Name Target ID 1368 ==================== ========================== 1369 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1370 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1371 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1372 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1373 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1374 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1375 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1376 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1377 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1378 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1379 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1380 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1381 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1382 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1383 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1384 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1385 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1386 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1387 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1388 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1389 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1390 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1391 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1392 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1393 ==================== ========================== 1394 1395.. _amdgpu-note-records-v3-v4: 1396 1397Code Object V3 to V4 Note Records 1398~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1399 1400The AMDGPU backend code object uses the following ELF note record in the 1401``.note`` section when compiling for code object V3 to V4. 1402 1403The note record vendor field is "AMDGPU". 1404 1405Additional note records may be present, but any which are not documented here 1406are deprecated and should not be used. 1407 1408 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1409 :name: amdgpu-elf-note-records-table-v3-v4 1410 1411 ======== ============================== ====================================== 1412 Name Type Description 1413 ======== ============================== ====================================== 1414 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1415 binary format. 1416 ======== ============================== ====================================== 1417 1418.. 1419 1420 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1421 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1422 1423 ============================== ===== 1424 Name Value 1425 ============================== ===== 1426 *reserved* 0-31 1427 ``NT_AMDGPU_METADATA`` 32 1428 ============================== ===== 1429 1430``NT_AMDGPU_METADATA`` 1431 Specifies extensible metadata associated with an AMDGPU code object. It is 1432 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1433 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1434 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1435 ``amdhsa`` OS. 1436 1437.. _amdgpu-symbols: 1438 1439Symbols 1440------- 1441 1442Symbols include the following: 1443 1444 .. table:: AMDGPU ELF Symbols 1445 :name: amdgpu-elf-symbols-table 1446 1447 ===================== ================== ================ ================== 1448 Name Type Section Description 1449 ===================== ================== ================ ================== 1450 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1451 - ``.rodata`` 1452 - ``.bss`` 1453 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1454 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1455 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1456 ===================== ================== ================ ================== 1457 1458Global variable 1459 Global variables both used and defined by the compilation unit. 1460 1461 If the symbol is defined in the compilation unit then it is allocated in the 1462 appropriate section according to if it has initialized data or is readonly. 1463 1464 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1465 will resolve relocations using the definition provided by another code object 1466 or explicitly defined by the runtime. 1467 1468 If the symbol resides in local/group memory (LDS) then its section is the 1469 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1470 ``st_value`` field describes alignment requirements as it does for common 1471 symbols. 1472 1473 .. TODO:: 1474 1475 Add description of linked shared object symbols. Seems undefined symbols 1476 are marked as STT_NOTYPE. 1477 1478Kernel descriptor 1479 Every HSA kernel has an associated kernel descriptor. It is the address of the 1480 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1481 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1482 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1483 1484Kernel entry point 1485 Every HSA kernel also has a symbol for its machine code entry point. 1486 1487.. _amdgpu-relocation-records: 1488 1489Relocation Records 1490------------------ 1491 1492AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1493relocatable fields are: 1494 1495``word32`` 1496 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1497 alignment. These values use the same byte order as other word values in the 1498 AMDGPU architecture. 1499 1500``word64`` 1501 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1502 alignment. These values use the same byte order as other word values in the 1503 AMDGPU architecture. 1504 1505Following notations are used for specifying relocation calculations: 1506 1507**A** 1508 Represents the addend used to compute the value of the relocatable field. 1509 1510**G** 1511 Represents the offset into the global offset table at which the relocation 1512 entry's symbol will reside during execution. 1513 1514**GOT** 1515 Represents the address of the global offset table. 1516 1517**P** 1518 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1519 of the storage unit being relocated (computed using ``r_offset``). 1520 1521**S** 1522 Represents the value of the symbol whose index resides in the relocation 1523 entry. Relocations not using this must specify a symbol index of 1524 ``STN_UNDEF``. 1525 1526**B** 1527 Represents the base address of a loaded executable or shared object which is 1528 the difference between the ELF address and the actual load address. 1529 Relocations using this are only valid in executable or shared objects. 1530 1531The following relocation types are supported: 1532 1533 .. table:: AMDGPU ELF Relocation Records 1534 :name: amdgpu-elf-relocation-records-table 1535 1536 ========================== ======= ===== ========== ============================== 1537 Relocation Type Kind Value Field Calculation 1538 ========================== ======= ===== ========== ============================== 1539 ``R_AMDGPU_NONE`` 0 *none* *none* 1540 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1541 Dynamic 1542 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1543 Dynamic 1544 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1545 Dynamic 1546 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1547 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1548 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1549 Dynamic 1550 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1551 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1552 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1553 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1554 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1555 *reserved* 12 1556 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1557 ========================== ======= ===== ========== ============================== 1558 1559``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1560the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1561 1562There is no current OS loader support for 32-bit programs and so 1563``R_AMDGPU_ABS32`` is not used. 1564 1565.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1566 1567Loaded Code Object Path Uniform Resource Identifier (URI) 1568--------------------------------------------------------- 1569 1570The AMD GPU code object loader represents the path of the ELF shared object from 1571which the code object was loaded as a textual Unifom Resource Identifier (URI). 1572Note that the code object is the in memory loaded relocated form of the ELF 1573shared object. Multiple code objects may be loaded at different memory 1574addresses in the same process from the same ELF shared object. 1575 1576The loaded code object path URI syntax is defined by the following BNF syntax: 1577 1578.. code:: 1579 1580 code_object_uri ::== file_uri | memory_uri 1581 file_uri ::== "file://" file_path [ range_specifier ] 1582 memory_uri ::== "memory://" process_id range_specifier 1583 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1584 file_path ::== URI_ENCODED_OS_FILE_PATH 1585 process_id ::== DECIMAL_NUMBER 1586 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1587 1588**number** 1589 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1590 and octal values by "0". 1591 1592**file_path** 1593 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1594 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1595 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1596 the path are separated by "/". 1597 1598**offset** 1599 Is a 0-based byte offset to the start of the code object. For a file URI, it 1600 is from the start of the file specified by the ``file_path``, and if omitted 1601 defaults to 0. For a memory URI, it is the memory address and is required. 1602 1603**size** 1604 Is the number of bytes in the code object. For a file URI, if omitted it 1605 defaults to the size of the file. It is required for a memory URI. 1606 1607**process_id** 1608 Is the identity of the process owning the memory. For Linux it is the C 1609 unsigned integral decimal literal for the process ID (PID). 1610 1611For example: 1612 1613.. code:: 1614 1615 file:///dir1/dir2/file1 1616 file:///dir3/dir4/file2#offset=0x2000&size=3000 1617 memory://1234#offset=0x20000&size=3000 1618 1619.. _amdgpu-dwarf-debug-information: 1620 1621DWARF Debug Information 1622======================= 1623 1624.. warning:: 1625 1626 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1627 is not currently fully implemented and is subject to change. 1628 1629AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1630:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1631object executable code and data to the source language constructs. It can be 1632used by tools such as debuggers and profilers. It uses features defined in 1633:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1634DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1635 1636This section defines the AMDGPU target architecture specific DWARF mappings. 1637 1638.. _amdgpu-dwarf-register-identifier: 1639 1640Register Identifier 1641------------------- 1642 1643This section defines the AMDGPU target architecture register numbers used in 1644DWARF operation expressions (see DWARF Version 5 section 2.5 and 1645:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1646instructions (see DWARF Version 5 section 6.4 and 1647:ref:`amdgpu-dwarf-call-frame-information`). 1648 1649A single code object can contain code for kernels that have different wavefront 1650sizes. The vector registers and some scalar registers are based on the wavefront 1651size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1652simplifies the consumer of the DWARF so that each register has a fixed size, 1653rather than being dynamic according to the wavefront size mode. Similarly, 1654distinct DWARF registers are defined for those registers that vary in size 1655according to the process address size. This allows a consumer to treat a 1656specific AMDGPU processor as a single architecture regardless of how it is 1657configured at run time. The compiler explicitly specifies the DWARF registers 1658that match the mode in which the code it is generating will be executed. 1659 1660DWARF registers are encoded as numbers, which are mapped to architecture 1661registers. The mapping for AMDGPU is defined in 1662:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1663mapping. 1664 1665.. table:: AMDGPU DWARF Register Mapping 1666 :name: amdgpu-dwarf-register-mapping-table 1667 1668 ============== ================= ======== ================================== 1669 DWARF Register AMDGPU Register Bit Size Description 1670 ============== ================= ======== ================================== 1671 0 PC_32 32 Program Counter (PC) when 1672 executing in a 32-bit process 1673 address space. Used in the CFI to 1674 describe the PC of the calling 1675 frame. 1676 1 EXEC_MASK_32 32 Execution Mask Register when 1677 executing in wavefront 32 mode. 1678 2-15 *Reserved* *Reserved for highly accessed 1679 registers using DWARF shortcut.* 1680 16 PC_64 64 Program Counter (PC) when 1681 executing in a 64-bit process 1682 address space. Used in the CFI to 1683 describe the PC of the calling 1684 frame. 1685 17 EXEC_MASK_64 64 Execution Mask Register when 1686 executing in wavefront 64 mode. 1687 18-31 *Reserved* *Reserved for highly accessed 1688 registers using DWARF shortcut.* 1689 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1690 Registers. 1691 96-127 *Reserved* *Reserved for frequently accessed 1692 registers using DWARF 1-byte ULEB.* 1693 128 STATUS 32 Status Register. 1694 129-511 *Reserved* *Reserved for future Scalar 1695 Architectural Registers.* 1696 512 VCC_32 32 Vector Condition Code Register 1697 when executing in wavefront 32 1698 mode. 1699 513-1023 *Reserved* *Reserved for future Vector 1700 Architectural Registers when 1701 executing in wavefront 32 mode.* 1702 768 VCC_64 64 Vector Condition Code Register 1703 when executing in wavefront 64 1704 mode. 1705 769-1023 *Reserved* *Reserved for future Vector 1706 Architectural Registers when 1707 executing in wavefront 64 mode.* 1708 1024-1087 *Reserved* *Reserved for padding.* 1709 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1710 1130-1535 *Reserved* *Reserved for future Scalar 1711 General Purpose Registers.* 1712 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1713 when executing in wavefront 32 1714 mode. 1715 1792-2047 *Reserved* *Reserved for future Vector 1716 General Purpose Registers when 1717 executing in wavefront 32 mode.* 1718 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1719 when executing in wavefront 32 1720 mode. 1721 2304-2559 *Reserved* *Reserved for future Vector 1722 Accumulation Registers when 1723 executing in wavefront 32 mode.* 1724 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1725 when executing in wavefront 64 1726 mode. 1727 2816-3071 *Reserved* *Reserved for future Vector 1728 General Purpose Registers when 1729 executing in wavefront 64 mode.* 1730 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1731 when executing in wavefront 64 1732 mode. 1733 3328-3583 *Reserved* *Reserved for future Vector 1734 Accumulation Registers when 1735 executing in wavefront 64 mode.* 1736 ============== ================= ======== ================================== 1737 1738The vector registers are represented as the full size for the wavefront. They 1739are organized as consecutive dwords (32-bits), one per lane, with the dword at 1740the least significant bit position corresponding to lane 0 and so forth. DWARF 1741location expressions involving the ``DW_OP_LLVM_offset`` and 1742``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1743register corresponding to the lane that is executing the current thread of 1744execution in languages that are implemented using a SIMD or SIMT execution 1745model. 1746 1747If the wavefront size is 32 lanes then the wavefront 32 mode register 1748definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1749mode register definitions are used. Some AMDGPU targets support executing in 1750both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1751to the wavefront mode of the generated code will be used. 1752 1753If code is generated to execute in a 32-bit process address space, then the 175432-bit process address space register definitions are used. If code is generated 1755to execute in a 64-bit process address space, then the 64-bit process address 1756space register definitions are used. The ``amdgcn`` target only supports the 175764-bit process address space. 1758 1759.. _amdgpu-dwarf-address-class-identifier: 1760 1761Address Class Identifier 1762------------------------ 1763 1764The DWARF address class represents the source language memory space. See DWARF 1765Version 5 section 2.12 which is updated by the *DWARF Extensions For 1766Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1767 1768The DWARF address class mapping used for AMDGPU is defined in 1769:ref:`amdgpu-dwarf-address-class-mapping-table`. 1770 1771.. table:: AMDGPU DWARF Address Class Mapping 1772 :name: amdgpu-dwarf-address-class-mapping-table 1773 1774 ========================= ====== ================= 1775 DWARF AMDGPU 1776 -------------------------------- ----------------- 1777 Address Class Name Value Address Space 1778 ========================= ====== ================= 1779 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1780 ``DW_ADDR_LLVM_global`` 0x0001 Global 1781 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1782 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1783 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1784 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1785 ========================= ====== ================= 1786 1787The DWARF address class values defined in the *DWARF Extensions For 1788Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1789 1790In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1791available for use for the AMD extension for access to the hardware GDS memory 1792which is scratchpad memory allocated per device. 1793 1794For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1795address class of ``DW_ADDR_none`` is used. 1796 1797See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1798mapping of DWARF address classes to DWARF address spaces, including address size 1799and NULL value. 1800 1801.. _amdgpu-dwarf-address-space-identifier: 1802 1803Address Space Identifier 1804------------------------ 1805 1806DWARF address spaces correspond to target architecture specific linear 1807addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1808For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1809 1810The DWARF address space mapping used for AMDGPU is defined in 1811:ref:`amdgpu-dwarf-address-space-mapping-table`. 1812 1813.. table:: AMDGPU DWARF Address Space Mapping 1814 :name: amdgpu-dwarf-address-space-mapping-table 1815 1816 ======================================= ===== ======= ======== ================= ======================= 1817 DWARF AMDGPU Notes 1818 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1819 Address Space Name Value Address Bit Size Address Space 1820 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1821 .. 64-bit 32-bit 1822 process process 1823 address address 1824 space space 1825 ======================================= ===== ======= ======== ================= ======================= 1826 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1827 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1828 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1829 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1830 *Reserved* 0x04 1831 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1832 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1833 ======================================= ===== ======= ======== ================= ======================= 1834 1835See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1836including address size and NULL value. 1837 1838The ``DW_ASPACE_none`` address space is the default target architecture address 1839space used in DWARF operations that do not specify an address space. It 1840therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1841related operations can refer to addresses in the program code. 1842 1843The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1844specify the flat address space. If the address corresponds to an address in the 1845local address space, then it corresponds to the wavefront that is executing the 1846focused thread of execution. If the address corresponds to an address in the 1847private address space, then it corresponds to the lane that is executing the 1848focused thread of execution for languages that are implemented using a SIMD or 1849SIMT execution model. 1850 1851.. note:: 1852 1853 CUDA-like languages such as HIP that do not have address spaces in the 1854 language type system, but do allow variables to be allocated in different 1855 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1856 address space in the DWARF expression operations as the default address space 1857 is the global address space. 1858 1859The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1860specify the local address space corresponding to the wavefront that is executing 1861the focused thread of execution. 1862 1863The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1864to specify the private address space corresponding to the lane that is executing 1865the focused thread of execution for languages that are implemented using a SIMD 1866or SIMT execution model. 1867 1868The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1869to specify the unswizzled private address space corresponding to the wavefront 1870that is executing the focused thread of execution. The wavefront view of private 1871memory is the per wavefront unswizzled backing memory layout defined in 1872:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1873location for the backing memory of the wavefront (namely the address is not 1874offset by ``wavefront-scratch-base``). The following formula can be used to 1875convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1876``DW_ASPACE_AMDGPU_private_wave`` address: 1877 1878:: 1879 1880 private-address-wavefront = 1881 ((private-address-lane / 4) * wavefront-size * 4) + 1882 (wavefront-lane-id * 4) + (private-address-lane % 4) 1883 1884If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1885of the dwords for each lane starting with lane 0 is required, then this 1886simplifies to: 1887 1888:: 1889 1890 private-address-wavefront = 1891 private-address-lane * wavefront-size 1892 1893A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1894complete spilled vector register back into a complete vector register in the 1895CFI. The frame pointer can be a private lane address which is dword aligned, 1896which can be shifted to multiply by the wavefront size, and then used to form a 1897private wavefront address that gives a location for a contiguous set of dwords, 1898one per lane, where the vector register dwords are spilled. The compiler knows 1899the wavefront size since it generates the code. Note that the type of the 1900address may have to be converted as the size of a 1901``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1902``DW_ASPACE_AMDGPU_private_wave`` address. 1903 1904.. _amdgpu-dwarf-lane-identifier: 1905 1906Lane identifier 1907--------------- 1908 1909DWARF lane identifies specify a target architecture lane position for hardware 1910that executes in a SIMD or SIMT manner, and on which a source language maps its 1911threads of execution onto those lanes. The DWARF lane identifier is pushed by 1912the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1913section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1914section :ref:`amdgpu-dwarf-operation-expressions`. 1915 1916For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1917wavefront. It is numbered from 0 to the wavefront size minus 1. 1918 1919Operation Expressions 1920--------------------- 1921 1922DWARF expressions are used to compute program values and the locations of 1923program objects. See DWARF Version 5 section 2.5 and 1924:ref:`amdgpu-dwarf-operation-expressions`. 1925 1926DWARF location descriptions describe how to access storage which includes memory 1927and registers. When accessing storage on AMDGPU, bytes are ordered with least 1928significant bytes first, and bits are ordered within bytes with least 1929significant bits first. 1930 1931For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1932unwinding vector registers that are spilled under the execution mask to memory: 1933the zero-single location description is the vector register, and the one-single 1934location description is the spilled memory location description. The 1935``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1936memory location description. 1937 1938In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1939``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1940controlled by the execution mask. An undefined location description together 1941with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1942to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1943 1944Debugger Information Entry Attributes 1945------------------------------------- 1946 1947This section describes how certain debugger information entry attributes are 1948used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1949by *DWARF Extensions For Heterogeneous Debugging* section 1950:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1951 1952.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1953 1954``DW_AT_LLVM_lane_pc`` 1955~~~~~~~~~~~~~~~~~~~~~~ 1956 1957For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1958location of the separate lanes of a SIMT thread. 1959 1960If the lane is an active lane then this will be the same as the current program 1961location. 1962 1963If the lane is inactive, but was active on entry to the subprogram, then this is 1964the program location in the subprogram at which execution of the lane is 1965conceptual positioned. 1966 1967If the lane was not active on entry to the subprogram, then this will be the 1968undefined location. A client debugger can check if the lane is part of a valid 1969work-group by checking that the lane is in the range of the associated 1970work-group within the grid, accounting for partial work-groups. If it is not, 1971then the debugger can omit any information for the lane. Otherwise, the debugger 1972may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1973calling subprogram until it finds a non-undefined location. Conceptually the 1974lane only has the call frames that it has a non-undefined 1975``DW_AT_LLVM_lane_pc``. 1976 1977The following example illustrates how the AMDGPU backend can generate a DWARF 1978location list expression for the nested ``IF/THEN/ELSE`` structures of the 1979following subprogram pseudo code for a target with 64 lanes per wavefront. 1980 1981.. code:: 1982 :number-lines: 1983 1984 SUBPROGRAM X 1985 BEGIN 1986 a; 1987 IF (c1) THEN 1988 b; 1989 IF (c2) THEN 1990 c; 1991 ELSE 1992 d; 1993 ENDIF 1994 e; 1995 ELSE 1996 f; 1997 ENDIF 1998 g; 1999 END 2000 2001The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2002execution mask (``EXEC``) to linearize the control flow. The condition is 2003evaluated to make a mask of the lanes for which the condition evaluates to true. 2004First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2005logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2006``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2007the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2008region the ``EXEC`` mask is restored to the value it had at the beginning of the 2009region. This is shown below. Other approaches are possible, but the basic 2010concept is the same. 2011 2012.. code:: 2013 :number-lines: 2014 2015 $lex_start: 2016 a; 2017 %1 = EXEC 2018 %2 = c1 2019 $lex_1_start: 2020 EXEC = %1 & %2 2021 $if_1_then: 2022 b; 2023 %3 = EXEC 2024 %4 = c2 2025 $lex_1_1_start: 2026 EXEC = %3 & %4 2027 $lex_1_1_then: 2028 c; 2029 EXEC = ~EXEC & %3 2030 $lex_1_1_else: 2031 d; 2032 EXEC = %3 2033 $lex_1_1_end: 2034 e; 2035 EXEC = ~EXEC & %1 2036 $lex_1_else: 2037 f; 2038 EXEC = %1 2039 $lex_1_end: 2040 g; 2041 $lex_end: 2042 2043To create the DWARF location list expression that defines the location 2044description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2045pseudo instruction can be used to annotate the linearized control flow. This can 2046be done by defining an artificial variable for the lane PC. The DWARF location 2047list expression created for it is used as the value of the 2048``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2049 2050A DWARF procedure is defined for each well nested structured control flow region 2051which provides the conceptual lane program location for a lane if it is not 2052active (namely it is divergent). The DWARF operation expression for each region 2053conceptually inherits the value of the immediately enclosing region and modifies 2054it according to the semantics of the region. 2055 2056For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2057the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2058region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2059region since the ``THEN`` region has completed. 2060 2061The lane PC artificial variable is assigned at each region transition. It uses 2062the immediately enclosing region's DWARF procedure to compute the program 2063location for each lane assuming they are divergent, and then modifies the result 2064by inserting the current program location for each lane that the ``EXEC`` mask 2065indicates is active. 2066 2067By having separate DWARF procedures for each region, they can be reused to 2068define the value for any nested region. This reduces the total size of the DWARF 2069operation expressions. 2070 2071The following provides an example using pseudo LLVM MIR. 2072 2073.. code:: 2074 :number-lines: 2075 2076 $lex_start: 2077 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2078 DW_AT_name = "__uint64"; 2079 DW_AT_byte_size = 8; 2080 DW_AT_encoding = DW_ATE_unsigned; 2081 ]; 2082 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2083 DW_AT_name = "__active_lane_pc"; 2084 DW_AT_location = [ 2085 DW_OP_regx PC; 2086 DW_OP_LLVM_extend 64, 64; 2087 DW_OP_regval_type EXEC, %uint_64; 2088 DW_OP_LLVM_select_bit_piece 64, 64; 2089 ]; 2090 ]; 2091 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2092 DW_AT_name = "__divergent_lane_pc"; 2093 DW_AT_location = [ 2094 DW_OP_LLVM_undefined; 2095 DW_OP_LLVM_extend 64, 64; 2096 ]; 2097 ]; 2098 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2099 DW_OP_call_ref %__divergent_lane_pc; 2100 DW_OP_call_ref %__active_lane_pc; 2101 ]; 2102 a; 2103 %1 = EXEC; 2104 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2105 %2 = c1; 2106 $lex_1_start: 2107 EXEC = %1 & %2; 2108 $lex_1_then: 2109 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2110 DW_AT_name = "__divergent_lane_pc_1_then"; 2111 DW_AT_location = DIExpression[ 2112 DW_OP_call_ref %__divergent_lane_pc; 2113 DW_OP_addrx &lex_1_start; 2114 DW_OP_stack_value; 2115 DW_OP_LLVM_extend 64, 64; 2116 DW_OP_call_ref %__lex_1_save_exec; 2117 DW_OP_deref_type 64, %__uint_64; 2118 DW_OP_LLVM_select_bit_piece 64, 64; 2119 ]; 2120 ]; 2121 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2122 DW_OP_call_ref %__divergent_lane_pc_1_then; 2123 DW_OP_call_ref %__active_lane_pc; 2124 ]; 2125 b; 2126 %3 = EXEC; 2127 DBG_VALUE %3, %__lex_1_1_save_exec; 2128 %4 = c2; 2129 $lex_1_1_start: 2130 EXEC = %3 & %4; 2131 $lex_1_1_then: 2132 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2133 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2134 DW_AT_location = DIExpression[ 2135 DW_OP_call_ref %__divergent_lane_pc_1_then; 2136 DW_OP_addrx &lex_1_1_start; 2137 DW_OP_stack_value; 2138 DW_OP_LLVM_extend 64, 64; 2139 DW_OP_call_ref %__lex_1_1_save_exec; 2140 DW_OP_deref_type 64, %__uint_64; 2141 DW_OP_LLVM_select_bit_piece 64, 64; 2142 ]; 2143 ]; 2144 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2145 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2146 DW_OP_call_ref %__active_lane_pc; 2147 ]; 2148 c; 2149 EXEC = ~EXEC & %3; 2150 $lex_1_1_else: 2151 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2152 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2153 DW_AT_location = DIExpression[ 2154 DW_OP_call_ref %__divergent_lane_pc_1_then; 2155 DW_OP_addrx &lex_1_1_end; 2156 DW_OP_stack_value; 2157 DW_OP_LLVM_extend 64, 64; 2158 DW_OP_call_ref %__lex_1_1_save_exec; 2159 DW_OP_deref_type 64, %__uint_64; 2160 DW_OP_LLVM_select_bit_piece 64, 64; 2161 ]; 2162 ]; 2163 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2164 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2165 DW_OP_call_ref %__active_lane_pc; 2166 ]; 2167 d; 2168 EXEC = %3; 2169 $lex_1_1_end: 2170 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2171 DW_OP_call_ref %__divergent_lane_pc; 2172 DW_OP_call_ref %__active_lane_pc; 2173 ]; 2174 e; 2175 EXEC = ~EXEC & %1; 2176 $lex_1_else: 2177 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2178 DW_AT_name = "__divergent_lane_pc_1_else"; 2179 DW_AT_location = DIExpression[ 2180 DW_OP_call_ref %__divergent_lane_pc; 2181 DW_OP_addrx &lex_1_end; 2182 DW_OP_stack_value; 2183 DW_OP_LLVM_extend 64, 64; 2184 DW_OP_call_ref %__lex_1_save_exec; 2185 DW_OP_deref_type 64, %__uint_64; 2186 DW_OP_LLVM_select_bit_piece 64, 64; 2187 ]; 2188 ]; 2189 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2190 DW_OP_call_ref %__divergent_lane_pc_1_else; 2191 DW_OP_call_ref %__active_lane_pc; 2192 ]; 2193 f; 2194 EXEC = %1; 2195 $lex_1_end: 2196 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2197 DW_OP_call_ref %__divergent_lane_pc; 2198 DW_OP_call_ref %__active_lane_pc; 2199 ]; 2200 g; 2201 $lex_end: 2202 2203The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2204that are active, with the current program location. 2205 2206Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2207the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2208instruction, location list entries will be created that describe where the 2209artificial variables are allocated at any given program location. The compiler 2210may allocate them to registers or spill them to memory. 2211 2212The DWARF procedures for each region use the values of the saved execution mask 2213artificial variables to only update the lanes that are active on entry to the 2214region. All other lanes retain the value of the enclosing region where they were 2215last active. If they were not active on entry to the subprogram, then will have 2216the undefined location description. 2217 2218Other structured control flow regions can be handled similarly. For example, 2219loops would set the divergent program location for the region at the end of the 2220loop. Any lanes active will be in the loop, and any lanes not active must have 2221exited the loop. 2222 2223An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2224``IF/THEN/ELSE`` regions. 2225 2226The DWARF procedures can use the active lane artificial variable described in 2227:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2228``EXEC`` mask in order to support whole or quad wavefront mode. 2229 2230.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2231 2232``DW_AT_LLVM_active_lane`` 2233~~~~~~~~~~~~~~~~~~~~~~~~~~ 2234 2235The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2236entry is used to specify the lanes that are conceptually active for a SIMT 2237thread. 2238 2239The execution mask may be modified to implement whole or quad wavefront mode 2240operations. For example, all lanes may need to temporarily be made active to 2241execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2242update it to enable the necessary lanes, perform the operations, and then 2243restore the ``EXEC`` mask from the saved value. While executing the whole 2244wavefront region, the conceptual execution mask is the saved value, not the 2245``EXEC`` value. 2246 2247This is handled by defining an artificial variable for the active lane mask. The 2248active lane mask artificial variable would be the actual ``EXEC`` mask for 2249normal regions, and the saved execution mask for regions where the mask is 2250temporarily updated. The location list expression created for this artificial 2251variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2252attribute. 2253 2254``DW_AT_LLVM_augmentation`` 2255~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2256 2257For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2258debugger information entry has the following value for the augmentation string: 2259 2260:: 2261 2262 [amdgpu:v0.0] 2263 2264The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2265extensions used in the DWARF of the compilation unit. The version number 2266conforms to [SEMVER]_. 2267 2268Call Frame Information 2269---------------------- 2270 2271DWARF Call Frame Information (CFI) describes how a consumer can virtually 2272*unwind* call frames in a running process or core dump. See DWARF Version 5 2273section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2274 2275For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2276 22771. ``augmentation`` string contains the following null-terminated UTF-8 string: 2278 2279 :: 2280 2281 [amd:v0.0] 2282 2283 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2284 extensions used in this CIE or to the FDEs that use it. The version number 2285 conforms to [SEMVER]_. 2286 22872. ``address_size`` for the ``Global`` address space is defined in 2288 :ref:`amdgpu-dwarf-address-space-identifier`. 2289 22903. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2291 22924. ``code_alignment_factor`` is 4 bytes. 2293 2294 .. TODO:: 2295 2296 Add to :ref:`amdgpu-processor-table` table. 2297 22985. ``data_alignment_factor`` is 4 bytes. 2299 2300 .. TODO:: 2301 2302 Add to :ref:`amdgpu-processor-table` table. 2303 23046. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2305 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2306 23077. ``initial_instructions`` Since a subprogram X with fewer registers can be 2308 called from subprogram Y that has more allocated, X will not change any of 2309 the extra registers as it cannot access them. Therefore, the default rule 2310 for all columns is ``same value``. 2311 2312For AMDGPU the register number follows the numbering defined in 2313:ref:`amdgpu-dwarf-register-identifier`. 2314 2315For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2316the return address to get the address of a byte within the call site 2317instructions. See DWARF Version 5 section 6.4.4. 2318 2319Accelerated Access 2320------------------ 2321 2322See DWARF Version 5 section 6.1. 2323 2324Lookup By Name Section Header 2325~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2326 2327See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2328 2329For AMDGPU the lookup by name section header table: 2330 2331``augmentation_string_size`` (uword) 2332 2333 Set to the length of the ``augmentation_string`` value which is always a 2334 multiple of 4. 2335 2336``augmentation_string`` (sequence of UTF-8 characters) 2337 2338 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2339 2340 :: 2341 2342 [amdgpu:v0.0] 2343 2344 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2345 extensions used in the DWARF of this index. The version number conforms to 2346 [SEMVER]_. 2347 2348 .. note:: 2349 2350 This is different to the DWARF Version 5 definition that requires the first 2351 4 characters to be the vendor ID. But this is consistent with the other 2352 augmentation strings and does allow multiple vendor contributions. However, 2353 backwards compatibility may be more desirable. 2354 2355Lookup By Address Section Header 2356~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2357 2358See DWARF Version 5 section 6.1.2. 2359 2360For AMDGPU the lookup by address section header table: 2361 2362``address_size`` (ubyte) 2363 2364 Match the address size for the ``Global`` address space defined in 2365 :ref:`amdgpu-dwarf-address-space-identifier`. 2366 2367``segment_selector_size`` (ubyte) 2368 2369 AMDGPU does not use a segment selector so this is 0. The entries in the 2370 ``.debug_aranges`` do not have a segment selector. 2371 2372Line Number Information 2373----------------------- 2374 2375See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2376 2377AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2378The instruction set must be obtained from the ELF file header ``e_flags`` field 2379in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2380<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2381 2382.. TODO:: 2383 2384 Should the ``isa`` state machine register be used to indicate if the code is 2385 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2386 2387For AMDGPU the line number program header fields have the following values (see 2388DWARF Version 5 section 6.2.4): 2389 2390``address_size`` (ubyte) 2391 Matches the address size for the ``Global`` address space defined in 2392 :ref:`amdgpu-dwarf-address-space-identifier`. 2393 2394``segment_selector_size`` (ubyte) 2395 AMDGPU does not use a segment selector so this is 0. 2396 2397``minimum_instruction_length`` (ubyte) 2398 For GFX9-GFX10 this is 4. 2399 2400``maximum_operations_per_instruction`` (ubyte) 2401 For GFX9-GFX10 this is 1. 2402 2403Source text for online-compiled programs (for example, those compiled by the 2404OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2405See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2406Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2407<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2408 2409The Clang option used to control source embedding in AMDGPU is defined in 2410:ref:`amdgpu-clang-debug-options-table`. 2411 2412 .. table:: AMDGPU Clang Debug Options 2413 :name: amdgpu-clang-debug-options-table 2414 2415 ==================== ================================================== 2416 Debug Flag Description 2417 ==================== ================================================== 2418 -g[no-]embed-source Enable/disable embedding source text in DWARF 2419 debug sections. Useful for environments where 2420 source cannot be written to disk, such as 2421 when performing online compilation. 2422 ==================== ================================================== 2423 2424For example: 2425 2426``-gembed-source`` 2427 Enable the embedded source. 2428 2429``-gno-embed-source`` 2430 Disable the embedded source. 2431 243232-Bit and 64-Bit DWARF Formats 2433------------------------------- 2434 2435See DWARF Version 5 section 7.4 and 2436:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2437 2438For AMDGPU: 2439 2440* For the ``amdgcn`` target architecture only the 64-bit process address space 2441 is supported. 2442 2443* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2444 the 32-bit DWARF format. 2445 2446Unit Headers 2447------------ 2448 2449For AMDGPU the following values apply for each of the unit headers described in 2450DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2451 2452``address_size`` (ubyte) 2453 Matches the address size for the ``Global`` address space defined in 2454 :ref:`amdgpu-dwarf-address-space-identifier`. 2455 2456.. _amdgpu-code-conventions: 2457 2458Code Conventions 2459================ 2460 2461This section provides code conventions used for each supported target triple OS 2462(see :ref:`amdgpu-target-triples`). 2463 2464AMDHSA 2465------ 2466 2467This section provides code conventions used when the target triple OS is 2468``amdhsa`` (see :ref:`amdgpu-target-triples`). 2469 2470.. _amdgpu-amdhsa-code-object-metadata: 2471 2472Code Object Metadata 2473~~~~~~~~~~~~~~~~~~~~ 2474 2475The code object metadata specifies extensible metadata associated with the code 2476objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2477encoding and semantics of this metadata depends on the code object version; see 2478:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2479:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2480:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2481 2482Code object metadata is specified in a note record (see 2483:ref:`amdgpu-note-records`) and is required when the target triple OS is 2484``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2485information necessary to support the HSA compatible runtime kernel queries. For 2486example, the segment sizes needed in a dispatch packet. In addition, a 2487high-level language runtime may require other information to be included. For 2488example, the AMD OpenCL runtime records kernel argument information. 2489 2490.. _amdgpu-amdhsa-code-object-metadata-v2: 2491 2492Code Object V2 Metadata 2493+++++++++++++++++++++++ 2494 2495.. warning:: 2496 Code object V2 is not the default code object version emitted by this version 2497 of LLVM. 2498 2499Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2500(see :ref:`amdgpu-note-records-v2`). 2501 2502The metadata is specified as a YAML formatted string (see [YAML]_ and 2503:doc:`YamlIO`). 2504 2505.. TODO:: 2506 2507 Is the string null terminated? It probably should not if YAML allows it to 2508 contain null characters, otherwise it should be. 2509 2510The metadata is represented as a single YAML document comprised of the mapping 2511defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2512referenced tables. 2513 2514For boolean values, the string values of ``false`` and ``true`` are used for 2515false and true respectively. 2516 2517Additional information can be added to the mappings. To avoid conflicts, any 2518non-AMD key names should be prefixed by "*vendor-name*.". 2519 2520 .. table:: AMDHSA Code Object V2 Metadata Map 2521 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2522 2523 ========== ============== ========= ======================================= 2524 String Key Value Type Required? Description 2525 ========== ============== ========= ======================================= 2526 "Version" sequence of Required - The first integer is the major 2527 2 integers version. Currently 1. 2528 - The second integer is the minor 2529 version. Currently 0. 2530 "Printf" sequence of Each string is encoded information 2531 strings about a printf function call. The 2532 encoded information is organized as 2533 fields separated by colon (':'): 2534 2535 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2536 2537 where: 2538 2539 ``ID`` 2540 A 32-bit integer as a unique id for 2541 each printf function call 2542 2543 ``N`` 2544 A 32-bit integer equal to the number 2545 of arguments of printf function call 2546 minus 1 2547 2548 ``S[i]`` (where i = 0, 1, ... , N-1) 2549 32-bit integers for the size in bytes 2550 of the i-th FormatString argument of 2551 the printf function call 2552 2553 FormatString 2554 The format string passed to the 2555 printf function call. 2556 "Kernels" sequence of Required Sequence of the mappings for each 2557 mapping kernel in the code object. See 2558 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2559 for the definition of the mapping. 2560 ========== ============== ========= ======================================= 2561 2562.. 2563 2564 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2565 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2566 2567 ================= ============== ========= ================================ 2568 String Key Value Type Required? Description 2569 ================= ============== ========= ================================ 2570 "Name" string Required Source name of the kernel. 2571 "SymbolName" string Required Name of the kernel 2572 descriptor ELF symbol. 2573 "Language" string Source language of the kernel. 2574 Values include: 2575 2576 - "OpenCL C" 2577 - "OpenCL C++" 2578 - "HCC" 2579 - "OpenMP" 2580 2581 "LanguageVersion" sequence of - The first integer is the major 2582 2 integers version. 2583 - The second integer is the 2584 minor version. 2585 "Attrs" mapping Mapping of kernel attributes. 2586 See 2587 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2588 for the mapping definition. 2589 "Args" sequence of Sequence of mappings of the 2590 mapping kernel arguments. See 2591 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2592 for the definition of the mapping. 2593 "CodeProps" mapping Mapping of properties related to 2594 the kernel code. See 2595 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2596 for the mapping definition. 2597 ================= ============== ========= ================================ 2598 2599.. 2600 2601 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2602 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2603 2604 =================== ============== ========= ============================== 2605 String Key Value Type Required? Description 2606 =================== ============== ========= ============================== 2607 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2608 3 integers must be >=1 and the dispatch 2609 work-group size X, Y, Z must 2610 correspond to the specified 2611 values. Defaults to 0, 0, 0. 2612 2613 Corresponds to the OpenCL 2614 ``reqd_work_group_size`` 2615 attribute. 2616 "WorkGroupSizeHint" sequence of The dispatch work-group size 2617 3 integers X, Y, Z is likely to be the 2618 specified values. 2619 2620 Corresponds to the OpenCL 2621 ``work_group_size_hint`` 2622 attribute. 2623 "VecTypeHint" string The name of a scalar or vector 2624 type. 2625 2626 Corresponds to the OpenCL 2627 ``vec_type_hint`` attribute. 2628 2629 "RuntimeHandle" string The external symbol name 2630 associated with a kernel. 2631 OpenCL runtime allocates a 2632 global buffer for the symbol 2633 and saves the kernel's address 2634 to it, which is used for 2635 device side enqueueing. Only 2636 available for device side 2637 enqueued kernels. 2638 =================== ============== ========= ============================== 2639 2640.. 2641 2642 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2643 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2644 2645 ================= ============== ========= ================================ 2646 String Key Value Type Required? Description 2647 ================= ============== ========= ================================ 2648 "Name" string Kernel argument name. 2649 "TypeName" string Kernel argument type name. 2650 "Size" integer Required Kernel argument size in bytes. 2651 "Align" integer Required Kernel argument alignment in 2652 bytes. Must be a power of two. 2653 "ValueKind" string Required Kernel argument kind that 2654 specifies how to set up the 2655 corresponding argument. 2656 Values include: 2657 2658 "ByValue" 2659 The argument is copied 2660 directly into the kernarg. 2661 2662 "GlobalBuffer" 2663 A global address space pointer 2664 to the buffer data is passed 2665 in the kernarg. 2666 2667 "DynamicSharedPointer" 2668 A group address space pointer 2669 to dynamically allocated LDS 2670 is passed in the kernarg. 2671 2672 "Sampler" 2673 A global address space 2674 pointer to a S# is passed in 2675 the kernarg. 2676 2677 "Image" 2678 A global address space 2679 pointer to a T# is passed in 2680 the kernarg. 2681 2682 "Pipe" 2683 A global address space pointer 2684 to an OpenCL pipe is passed in 2685 the kernarg. 2686 2687 "Queue" 2688 A global address space pointer 2689 to an OpenCL device enqueue 2690 queue is passed in the 2691 kernarg. 2692 2693 "HiddenGlobalOffsetX" 2694 The OpenCL grid dispatch 2695 global offset for the X 2696 dimension is passed in the 2697 kernarg. 2698 2699 "HiddenGlobalOffsetY" 2700 The OpenCL grid dispatch 2701 global offset for the Y 2702 dimension is passed in the 2703 kernarg. 2704 2705 "HiddenGlobalOffsetZ" 2706 The OpenCL grid dispatch 2707 global offset for the Z 2708 dimension is passed in the 2709 kernarg. 2710 2711 "HiddenNone" 2712 An argument that is not used 2713 by the kernel. Space needs to 2714 be left for it, but it does 2715 not need to be set up. 2716 2717 "HiddenPrintfBuffer" 2718 A global address space pointer 2719 to the runtime printf buffer 2720 is passed in kernarg. 2721 2722 "HiddenHostcallBuffer" 2723 A global address space pointer 2724 to the runtime hostcall buffer 2725 is passed in kernarg. 2726 2727 "HiddenDefaultQueue" 2728 A global address space pointer 2729 to the OpenCL device enqueue 2730 queue that should be used by 2731 the kernel by default is 2732 passed in the kernarg. 2733 2734 "HiddenCompletionAction" 2735 A global address space pointer 2736 to help link enqueued kernels into 2737 the ancestor tree for determining 2738 when the parent kernel has finished. 2739 2740 "HiddenMultiGridSyncArg" 2741 A global address space pointer for 2742 multi-grid synchronization is 2743 passed in the kernarg. 2744 2745 "ValueType" string Unused and deprecated. This should no longer 2746 be emitted, but is accepted for compatibility. 2747 2748 2749 "PointeeAlign" integer Alignment in bytes of pointee 2750 type for pointer type kernel 2751 argument. Must be a power 2752 of 2. Only present if 2753 "ValueKind" is 2754 "DynamicSharedPointer". 2755 "AddrSpaceQual" string Kernel argument address space 2756 qualifier. Only present if 2757 "ValueKind" is "GlobalBuffer" or 2758 "DynamicSharedPointer". Values 2759 are: 2760 2761 - "Private" 2762 - "Global" 2763 - "Constant" 2764 - "Local" 2765 - "Generic" 2766 - "Region" 2767 2768 .. TODO:: 2769 2770 Is GlobalBuffer only Global 2771 or Constant? Is 2772 DynamicSharedPointer always 2773 Local? Can HCC allow Generic? 2774 How can Private or Region 2775 ever happen? 2776 2777 "AccQual" string Kernel argument access 2778 qualifier. Only present if 2779 "ValueKind" is "Image" or 2780 "Pipe". Values 2781 are: 2782 2783 - "ReadOnly" 2784 - "WriteOnly" 2785 - "ReadWrite" 2786 2787 .. TODO:: 2788 2789 Does this apply to 2790 GlobalBuffer? 2791 2792 "ActualAccQual" string The actual memory accesses 2793 performed by the kernel on the 2794 kernel argument. Only present if 2795 "ValueKind" is "GlobalBuffer", 2796 "Image", or "Pipe". This may be 2797 more restrictive than indicated 2798 by "AccQual" to reflect what the 2799 kernel actual does. If not 2800 present then the runtime must 2801 assume what is implied by 2802 "AccQual" and "IsConst". Values 2803 are: 2804 2805 - "ReadOnly" 2806 - "WriteOnly" 2807 - "ReadWrite" 2808 2809 "IsConst" boolean Indicates if the kernel argument 2810 is const qualified. Only present 2811 if "ValueKind" is 2812 "GlobalBuffer". 2813 2814 "IsRestrict" boolean Indicates if the kernel argument 2815 is restrict qualified. Only 2816 present if "ValueKind" is 2817 "GlobalBuffer". 2818 2819 "IsVolatile" boolean Indicates if the kernel argument 2820 is volatile qualified. Only 2821 present if "ValueKind" is 2822 "GlobalBuffer". 2823 2824 "IsPipe" boolean Indicates if the kernel argument 2825 is pipe qualified. Only present 2826 if "ValueKind" is "Pipe". 2827 2828 .. TODO:: 2829 2830 Can GlobalBuffer be pipe 2831 qualified? 2832 2833 ================= ============== ========= ================================ 2834 2835.. 2836 2837 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2838 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2839 2840 ============================ ============== ========= ===================== 2841 String Key Value Type Required? Description 2842 ============================ ============== ========= ===================== 2843 "KernargSegmentSize" integer Required The size in bytes of 2844 the kernarg segment 2845 that holds the values 2846 of the arguments to 2847 the kernel. 2848 "GroupSegmentFixedSize" integer Required The amount of group 2849 segment memory 2850 required by a 2851 work-group in 2852 bytes. This does not 2853 include any 2854 dynamically allocated 2855 group segment memory 2856 that may be added 2857 when the kernel is 2858 dispatched. 2859 "PrivateSegmentFixedSize" integer Required The amount of fixed 2860 private address space 2861 memory required for a 2862 work-item in 2863 bytes. If the kernel 2864 uses a dynamic call 2865 stack then additional 2866 space must be added 2867 to this value for the 2868 call stack. 2869 "KernargSegmentAlign" integer Required The maximum byte 2870 alignment of 2871 arguments in the 2872 kernarg segment. Must 2873 be a power of 2. 2874 "WavefrontSize" integer Required Wavefront size. Must 2875 be a power of 2. 2876 "NumSGPRs" integer Required Number of scalar 2877 registers used by a 2878 wavefront for 2879 GFX6-GFX10. This 2880 includes the special 2881 SGPRs for VCC, Flat 2882 Scratch (GFX7-GFX10) 2883 and XNACK (for 2884 GFX8-GFX10). It does 2885 not include the 16 2886 SGPR added if a trap 2887 handler is 2888 enabled. It is not 2889 rounded up to the 2890 allocation 2891 granularity. 2892 "NumVGPRs" integer Required Number of vector 2893 registers used by 2894 each work-item for 2895 GFX6-GFX10 2896 "MaxFlatWorkGroupSize" integer Required Maximum flat 2897 work-group size 2898 supported by the 2899 kernel in work-items. 2900 Must be >=1 and 2901 consistent with 2902 ReqdWorkGroupSize if 2903 not 0, 0, 0. 2904 "NumSpilledSGPRs" integer Number of stores from 2905 a scalar register to 2906 a register allocator 2907 created spill 2908 location. 2909 "NumSpilledVGPRs" integer Number of stores from 2910 a vector register to 2911 a register allocator 2912 created spill 2913 location. 2914 ============================ ============== ========= ===================== 2915 2916.. _amdgpu-amdhsa-code-object-metadata-v3: 2917 2918Code Object V3 Metadata 2919+++++++++++++++++++++++ 2920 2921Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2922record (see :ref:`amdgpu-note-records-v3-v4`). 2923 2924The metadata is represented as Message Pack formatted binary data (see 2925[MsgPack]_). The top level is a Message Pack map that includes the 2926keys defined in table 2927:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2928tables. 2929 2930Additional information can be added to the maps. To avoid conflicts, 2931any key names should be prefixed by "*vendor-name*." where 2932``vendor-name`` can be the name of the vendor and specific vendor 2933tool that generates the information. The prefix is abbreviated to 2934simply "." when it appears within a map that has been added by the 2935same *vendor-name*. 2936 2937 .. table:: AMDHSA Code Object V3 Metadata Map 2938 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2939 2940 ================= ============== ========= ======================================= 2941 String Key Value Type Required? Description 2942 ================= ============== ========= ======================================= 2943 "amdhsa.version" sequence of Required - The first integer is the major 2944 2 integers version. Currently 1. 2945 - The second integer is the minor 2946 version. Currently 0. 2947 "amdhsa.printf" sequence of Each string is encoded information 2948 strings about a printf function call. The 2949 encoded information is organized as 2950 fields separated by colon (':'): 2951 2952 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2953 2954 where: 2955 2956 ``ID`` 2957 A 32-bit integer as a unique id for 2958 each printf function call 2959 2960 ``N`` 2961 A 32-bit integer equal to the number 2962 of arguments of printf function call 2963 minus 1 2964 2965 ``S[i]`` (where i = 0, 1, ... , N-1) 2966 32-bit integers for the size in bytes 2967 of the i-th FormatString argument of 2968 the printf function call 2969 2970 FormatString 2971 The format string passed to the 2972 printf function call. 2973 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2974 map kernel in the code object. See 2975 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2976 for the definition of the keys included 2977 in that map. 2978 ================= ============== ========= ======================================= 2979 2980.. 2981 2982 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2983 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2984 2985 =================================== ============== ========= ================================ 2986 String Key Value Type Required? Description 2987 =================================== ============== ========= ================================ 2988 ".name" string Required Source name of the kernel. 2989 ".symbol" string Required Name of the kernel 2990 descriptor ELF symbol. 2991 ".language" string Source language of the kernel. 2992 Values include: 2993 2994 - "OpenCL C" 2995 - "OpenCL C++" 2996 - "HCC" 2997 - "HIP" 2998 - "OpenMP" 2999 - "Assembler" 3000 3001 ".language_version" sequence of - The first integer is the major 3002 2 integers version. 3003 - The second integer is the 3004 minor version. 3005 ".args" sequence of Sequence of maps of the 3006 map kernel arguments. See 3007 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3008 for the definition of the keys 3009 included in that map. 3010 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3011 3 integers must be >=1 and the dispatch 3012 work-group size X, Y, Z must 3013 correspond to the specified 3014 values. Defaults to 0, 0, 0. 3015 3016 Corresponds to the OpenCL 3017 ``reqd_work_group_size`` 3018 attribute. 3019 ".workgroup_size_hint" sequence of The dispatch work-group size 3020 3 integers X, Y, Z is likely to be the 3021 specified values. 3022 3023 Corresponds to the OpenCL 3024 ``work_group_size_hint`` 3025 attribute. 3026 ".vec_type_hint" string The name of a scalar or vector 3027 type. 3028 3029 Corresponds to the OpenCL 3030 ``vec_type_hint`` attribute. 3031 3032 ".device_enqueue_symbol" string The external symbol name 3033 associated with a kernel. 3034 OpenCL runtime allocates a 3035 global buffer for the symbol 3036 and saves the kernel's address 3037 to it, which is used for 3038 device side enqueueing. Only 3039 available for device side 3040 enqueued kernels. 3041 ".kernarg_segment_size" integer Required The size in bytes of 3042 the kernarg segment 3043 that holds the values 3044 of the arguments to 3045 the kernel. 3046 ".group_segment_fixed_size" integer Required The amount of group 3047 segment memory 3048 required by a 3049 work-group in 3050 bytes. This does not 3051 include any 3052 dynamically allocated 3053 group segment memory 3054 that may be added 3055 when the kernel is 3056 dispatched. 3057 ".private_segment_fixed_size" integer Required The amount of fixed 3058 private address space 3059 memory required for a 3060 work-item in 3061 bytes. If the kernel 3062 uses a dynamic call 3063 stack then additional 3064 space must be added 3065 to this value for the 3066 call stack. 3067 ".kernarg_segment_align" integer Required The maximum byte 3068 alignment of 3069 arguments in the 3070 kernarg segment. Must 3071 be a power of 2. 3072 ".wavefront_size" integer Required Wavefront size. Must 3073 be a power of 2. 3074 ".sgpr_count" integer Required Number of scalar 3075 registers required by a 3076 wavefront for 3077 GFX6-GFX9. A register 3078 is required if it is 3079 used explicitly, or 3080 if a higher numbered 3081 register is used 3082 explicitly. This 3083 includes the special 3084 SGPRs for VCC, Flat 3085 Scratch (GFX7-GFX9) 3086 and XNACK (for 3087 GFX8-GFX9). It does 3088 not include the 16 3089 SGPR added if a trap 3090 handler is 3091 enabled. It is not 3092 rounded up to the 3093 allocation 3094 granularity. 3095 ".vgpr_count" integer Required Number of vector 3096 registers required by 3097 each work-item for 3098 GFX6-GFX9. A register 3099 is required if it is 3100 used explicitly, or 3101 if a higher numbered 3102 register is used 3103 explicitly. 3104 ".max_flat_workgroup_size" integer Required Maximum flat 3105 work-group size 3106 supported by the 3107 kernel in work-items. 3108 Must be >=1 and 3109 consistent with 3110 ReqdWorkGroupSize if 3111 not 0, 0, 0. 3112 ".sgpr_spill_count" integer Number of stores from 3113 a scalar register to 3114 a register allocator 3115 created spill 3116 location. 3117 ".vgpr_spill_count" integer Number of stores from 3118 a vector register to 3119 a register allocator 3120 created spill 3121 location. 3122 =================================== ============== ========= ================================ 3123 3124.. 3125 3126 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3127 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3128 3129 ====================== ============== ========= ================================ 3130 String Key Value Type Required? Description 3131 ====================== ============== ========= ================================ 3132 ".name" string Kernel argument name. 3133 ".type_name" string Kernel argument type name. 3134 ".size" integer Required Kernel argument size in bytes. 3135 ".offset" integer Required Kernel argument offset in 3136 bytes. The offset must be a 3137 multiple of the alignment 3138 required by the argument. 3139 ".value_kind" string Required Kernel argument kind that 3140 specifies how to set up the 3141 corresponding argument. 3142 Values include: 3143 3144 "by_value" 3145 The argument is copied 3146 directly into the kernarg. 3147 3148 "global_buffer" 3149 A global address space pointer 3150 to the buffer data is passed 3151 in the kernarg. 3152 3153 "dynamic_shared_pointer" 3154 A group address space pointer 3155 to dynamically allocated LDS 3156 is passed in the kernarg. 3157 3158 "sampler" 3159 A global address space 3160 pointer to a S# is passed in 3161 the kernarg. 3162 3163 "image" 3164 A global address space 3165 pointer to a T# is passed in 3166 the kernarg. 3167 3168 "pipe" 3169 A global address space pointer 3170 to an OpenCL pipe is passed in 3171 the kernarg. 3172 3173 "queue" 3174 A global address space pointer 3175 to an OpenCL device enqueue 3176 queue is passed in the 3177 kernarg. 3178 3179 "hidden_global_offset_x" 3180 The OpenCL grid dispatch 3181 global offset for the X 3182 dimension is passed in the 3183 kernarg. 3184 3185 "hidden_global_offset_y" 3186 The OpenCL grid dispatch 3187 global offset for the Y 3188 dimension is passed in the 3189 kernarg. 3190 3191 "hidden_global_offset_z" 3192 The OpenCL grid dispatch 3193 global offset for the Z 3194 dimension is passed in the 3195 kernarg. 3196 3197 "hidden_none" 3198 An argument that is not used 3199 by the kernel. Space needs to 3200 be left for it, but it does 3201 not need to be set up. 3202 3203 "hidden_printf_buffer" 3204 A global address space pointer 3205 to the runtime printf buffer 3206 is passed in kernarg. 3207 3208 "hidden_hostcall_buffer" 3209 A global address space pointer 3210 to the runtime hostcall buffer 3211 is passed in kernarg. 3212 3213 "hidden_default_queue" 3214 A global address space pointer 3215 to the OpenCL device enqueue 3216 queue that should be used by 3217 the kernel by default is 3218 passed in the kernarg. 3219 3220 "hidden_completion_action" 3221 A global address space pointer 3222 to help link enqueued kernels into 3223 the ancestor tree for determining 3224 when the parent kernel has finished. 3225 3226 "hidden_multigrid_sync_arg" 3227 A global address space pointer for 3228 multi-grid synchronization is 3229 passed in the kernarg. 3230 3231 ".value_type" string Unused and deprecated. This should no longer 3232 be emitted, but is accepted for compatibility. 3233 3234 ".pointee_align" integer Alignment in bytes of pointee 3235 type for pointer type kernel 3236 argument. Must be a power 3237 of 2. Only present if 3238 ".value_kind" is 3239 "dynamic_shared_pointer". 3240 ".address_space" string Kernel argument address space 3241 qualifier. Only present if 3242 ".value_kind" is "global_buffer" or 3243 "dynamic_shared_pointer". Values 3244 are: 3245 3246 - "private" 3247 - "global" 3248 - "constant" 3249 - "local" 3250 - "generic" 3251 - "region" 3252 3253 .. TODO:: 3254 3255 Is "global_buffer" only "global" 3256 or "constant"? Is 3257 "dynamic_shared_pointer" always 3258 "local"? Can HCC allow "generic"? 3259 How can "private" or "region" 3260 ever happen? 3261 3262 ".access" string Kernel argument access 3263 qualifier. Only present if 3264 ".value_kind" is "image" or 3265 "pipe". Values 3266 are: 3267 3268 - "read_only" 3269 - "write_only" 3270 - "read_write" 3271 3272 .. TODO:: 3273 3274 Does this apply to 3275 "global_buffer"? 3276 3277 ".actual_access" string The actual memory accesses 3278 performed by the kernel on the 3279 kernel argument. Only present if 3280 ".value_kind" is "global_buffer", 3281 "image", or "pipe". This may be 3282 more restrictive than indicated 3283 by ".access" to reflect what the 3284 kernel actual does. If not 3285 present then the runtime must 3286 assume what is implied by 3287 ".access" and ".is_const" . Values 3288 are: 3289 3290 - "read_only" 3291 - "write_only" 3292 - "read_write" 3293 3294 ".is_const" boolean Indicates if the kernel argument 3295 is const qualified. Only present 3296 if ".value_kind" is 3297 "global_buffer". 3298 3299 ".is_restrict" boolean Indicates if the kernel argument 3300 is restrict qualified. Only 3301 present if ".value_kind" is 3302 "global_buffer". 3303 3304 ".is_volatile" boolean Indicates if the kernel argument 3305 is volatile qualified. Only 3306 present if ".value_kind" is 3307 "global_buffer". 3308 3309 ".is_pipe" boolean Indicates if the kernel argument 3310 is pipe qualified. Only present 3311 if ".value_kind" is "pipe". 3312 3313 .. TODO:: 3314 3315 Can "global_buffer" be pipe 3316 qualified? 3317 3318 ====================== ============== ========= ================================ 3319 3320.. _amdgpu-amdhsa-code-object-metadata-v4: 3321 3322Code Object V4 Metadata 3323+++++++++++++++++++++++ 3324 3325.. warning:: 3326 Code object V4 is not the default code object version emitted by this version 3327 of LLVM. 3328 3329Code object V4 metadata is the same as 3330:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3331defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3332 3333 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3334 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3335 3336 ================= ============== ========= ======================================= 3337 String Key Value Type Required? Description 3338 ================= ============== ========= ======================================= 3339 "amdhsa.version" sequence of Required - The first integer is the major 3340 2 integers version. Currently 1. 3341 - The second integer is the minor 3342 version. Currently 1. 3343 "amdhsa.target" string Required The target name of the code using the syntax: 3344 3345 .. code:: 3346 3347 <target-triple> [ "-" <target-id> ] 3348 3349 A canonical target ID must be 3350 used. See :ref:`amdgpu-target-triples` 3351 and :ref:`amdgpu-target-id`. 3352 ================= ============== ========= ======================================= 3353 3354.. 3355 3356Kernel Dispatch 3357~~~~~~~~~~~~~~~ 3358 3359The HSA architected queuing language (AQL) defines a user space memory interface 3360that can be used to control the dispatch of kernels, in an agent independent 3361way. An agent can have zero or more AQL queues created for it using an HSA 3362compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3363are 64 bytes) can be placed. See the *HSA Platform System Architecture 3364Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3365 3366The packet processor of a kernel agent is responsible for detecting and 3367dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3368packet processor is implemented by the hardware command processor (CP), 3369asynchronous dispatch controller (ADC) and shader processor input controller 3370(SPI). 3371 3372An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3373the kernel mode driver to initialize and register the AQL queue with CP. 3374 3375To dispatch a kernel the following actions are performed. This can occur in the 3376CPU host program, or from an HSA kernel executing on a GPU. 3377 33781. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3379 executed is obtained. 33802. A pointer to the kernel descriptor (see 3381 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3382 It must be for a kernel that is contained in a code object that that was 3383 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3384 queue is associated. 33853. Space is allocated for the kernel arguments using the HSA compatible runtime 3386 allocator for a memory region with the kernarg property for the kernel agent 3387 that will execute the kernel. It must be at least 16-byte aligned. 33884. Kernel argument values are assigned to the kernel argument memory 3389 allocation. The layout is defined in the *HSA Programmer's Language 3390 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3391 kernel argument memory in the same way constant memory is accessed. (Note 3392 that the HSA specification allows an implementation to copy the kernel 3393 argument contents to another location that is accessed by the kernel.) 33945. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3395 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3396 for the packet. The packet must be set up, and the final write must use an 3397 atomic store release to set the packet kind to ensure the packet contents are 3398 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3399 notify the kernel agent that the AQL queue has been updated. These rules, and 3400 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3401 System Architecture Specification* [HSA]_. 34026. A kernel dispatch packet includes information about the actual dispatch, 3403 such as grid and work-group size, together with information from the code 3404 object about the kernel, such as segment sizes. The HSA compatible runtime 3405 queries on the kernel symbol can be used to obtain the code object values 3406 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 34077. CP executes micro-code and is responsible for detecting and setting up the 3408 GPU to execute the wavefronts of a kernel dispatch. 34098. CP ensures that when the a wavefront starts executing the kernel machine 3410 code, the scalar general purpose registers (SGPR) and vector general purpose 3411 registers (VGPR) are set up as required by the machine code. The required 3412 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3413 register state is defined in 3414 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 34159. The prolog of the kernel machine code (see 3416 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3417 before continuing executing the machine code that corresponds to the kernel. 341810. When the kernel dispatch has completed execution, CP signals the completion 3419 signal specified in the kernel dispatch packet if not 0. 3420 3421.. _amdgpu-amdhsa-memory-spaces: 3422 3423Memory Spaces 3424~~~~~~~~~~~~~ 3425 3426The memory space properties are: 3427 3428 .. table:: AMDHSA Memory Spaces 3429 :name: amdgpu-amdhsa-memory-spaces-table 3430 3431 ================= =========== ======== ======= ================== 3432 Memory Space Name HSA Segment Hardware Address NULL Value 3433 Name Name Size 3434 ================= =========== ======== ======= ================== 3435 Private private scratch 32 0x00000000 3436 Local group LDS 32 0xFFFFFFFF 3437 Global global global 64 0x0000000000000000 3438 Constant constant *same as 64 0x0000000000000000 3439 global* 3440 Generic flat flat 64 0x0000000000000000 3441 Region N/A GDS 32 *not implemented 3442 for AMDHSA* 3443 ================= =========== ======== ======= ================== 3444 3445The global and constant memory spaces both use global virtual addresses, which 3446are the same virtual address space used by the CPU. However, some virtual 3447addresses may only be accessible to the CPU, some only accessible by the GPU, 3448and some by both. 3449 3450Using the constant memory space indicates that the data will not change during 3451the execution of the kernel. This allows scalar read instructions to be 3452used. The vector and scalar L1 caches are invalidated of volatile data before 3453each kernel dispatch execution to allow constant memory to change values between 3454kernel dispatches. 3455 3456The local memory space uses the hardware Local Data Store (LDS) which is 3457automatically allocated when the hardware creates work-groups of wavefronts, and 3458freed when all the wavefronts of a work-group have terminated. The data store 3459(DS) instructions can be used to access it. 3460 3461The private memory space uses the hardware scratch memory support. If the kernel 3462uses scratch, then the hardware allocates memory that is accessed using 3463wavefront lane dword (4 byte) interleaving. The mapping used from private 3464address to physical address is: 3465 3466 ``wavefront-scratch-base + 3467 (private-address * wavefront-size * 4) + 3468 (wavefront-lane-id * 4)`` 3469 3470There are different ways that the wavefront scratch base address is determined 3471by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3472memory can be accessed in an interleaved manner using buffer instruction with 3473the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3474instructions, or by flat instructions. If each lane of a wavefront accesses the 3475same private address, the interleaving results in adjacent dwords being accessed 3476and hence requires fewer cache lines to be fetched. Multi-dword access is not 3477supported except by flat and scratch instructions in GFX9-GFX10. 3478 3479The generic address space uses the hardware flat address support available in 3480GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3481local apertures), that are outside the range of addressible global memory, to 3482map from a flat address to a private or local address. 3483 3484FLAT instructions can take a flat address and access global, private (scratch) 3485and group (LDS) memory depending in if the address is within one of the 3486aperture ranges. Flat access to scratch requires hardware aperture setup and 3487setup in the kernel prologue (see 3488:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3489hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3490:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3491 3492To convert between a segment address and a flat address the base address of the 3493apertures address can be used. For GFX7-GFX8 these are available in the 3494:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3495Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3496GFX9-GFX10 the aperture base addresses are directly available as inline constant 3497registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3498address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3499which makes it easier to convert from flat to segment or segment to flat. 3500 3501Image and Samplers 3502~~~~~~~~~~~~~~~~~~ 3503 3504Image and sample handles created by an HSA compatible runtime (see 3505:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3506object respectively. In order to support the HSA ``query_sampler`` operations 3507two extra dwords are used to store the HSA BRIG enumeration values for the 3508queries that are not trivially deducible from the S# representation. 3509 3510HSA Signals 3511~~~~~~~~~~~ 3512 3513HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3514are 64-bit addresses of a structure allocated in memory accessible from both the 3515CPU and GPU. The structure is defined by the runtime and subject to change 3516between releases. For example, see [AMD-ROCm-github]_. 3517 3518.. _amdgpu-amdhsa-hsa-aql-queue: 3519 3520HSA AQL Queue 3521~~~~~~~~~~~~~ 3522 3523The HSA AQL queue structure is defined by an HSA compatible runtime (see 3524:ref:`amdgpu-os`) and subject to change between releases. For example, see 3525[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3526certain language features such as the flat address aperture bases. It also 3527contains fields used by CP such as managing the allocation of scratch memory. 3528 3529.. _amdgpu-amdhsa-kernel-descriptor: 3530 3531Kernel Descriptor 3532~~~~~~~~~~~~~~~~~ 3533 3534A kernel descriptor consists of the information needed by CP to initiate the 3535execution of a kernel, including the entry point address of the machine code 3536that implements the kernel. 3537 3538Code Object V3 Kernel Descriptor 3539++++++++++++++++++++++++++++++++ 3540 3541CP microcode requires the Kernel descriptor to be allocated on 64-byte 3542alignment. 3543 3544The fields used by CP for code objects before V3 also match those specified in 3545:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3546 3547 .. table:: Code Object V3 Kernel Descriptor 3548 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3549 3550 ======= ======= =============================== ============================ 3551 Bits Size Field Name Description 3552 ======= ======= =============================== ============================ 3553 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3554 address space memory 3555 required for a work-group 3556 in bytes. This does not 3557 include any dynamically 3558 allocated local address 3559 space memory that may be 3560 added when the kernel is 3561 dispatched. 3562 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3563 private address space 3564 memory required for a 3565 work-item in bytes. 3566 Additional space may need to 3567 be added to this value if 3568 the call stack has 3569 non-inlined function calls. 3570 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3571 memory pointed to by the 3572 AQL dispatch packet. The 3573 kernarg memory is used to 3574 pass arguments to the 3575 kernel. 3576 3577 * If the kernarg pointer in 3578 the dispatch packet is NULL 3579 then there are no kernel 3580 arguments. 3581 * If the kernarg pointer in 3582 the dispatch packet is 3583 not NULL and this value 3584 is 0 then the kernarg 3585 memory size is 3586 unspecified. 3587 * If the kernarg pointer in 3588 the dispatch packet is 3589 not NULL and this value 3590 is not 0 then the value 3591 specifies the kernarg 3592 memory size in bytes. It 3593 is recommended to provide 3594 a value as it may be used 3595 by CP to optimize making 3596 the kernarg memory 3597 visible to the kernel 3598 code. 3599 3600 127:96 4 bytes Reserved, must be 0. 3601 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3602 negative) from base 3603 address of kernel 3604 descriptor to kernel's 3605 entry point instruction 3606 which must be 256 byte 3607 aligned. 3608 351:272 20 Reserved, must be 0. 3609 bytes 3610 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3611 Reserved, must be 0. 3612 GFX90A 3613 Compute Shader (CS) 3614 program settings used by 3615 CP to set up 3616 ``COMPUTE_PGM_RSRC3`` 3617 configuration 3618 register. See 3619 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3620 GFX10 3621 Compute Shader (CS) 3622 program settings used by 3623 CP to set up 3624 ``COMPUTE_PGM_RSRC3`` 3625 configuration 3626 register. See 3627 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3628 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3629 program settings used by 3630 CP to set up 3631 ``COMPUTE_PGM_RSRC1`` 3632 configuration 3633 register. See 3634 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3635 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3636 program settings used by 3637 CP to set up 3638 ``COMPUTE_PGM_RSRC2`` 3639 configuration 3640 register. See 3641 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3642 458:448 7 bits *See separate bits below.* Enable the setup of the 3643 SGPR user data registers 3644 (see 3645 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3646 3647 The total number of SGPR 3648 user data registers 3649 requested must not exceed 3650 16 and match value in 3651 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3652 Any requests beyond 16 3653 will be ignored. 3654 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3655 _BUFFER 3656 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3657 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3658 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3659 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3660 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT 3661 3662 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3663 _SIZE 3664 457:455 3 bits Reserved, must be 0. 3665 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3666 Reserved, must be 0. 3667 GFX10 3668 - If 0 execute in 3669 wavefront size 64 mode. 3670 - If 1 execute in 3671 native wavefront size 3672 32 mode. 3673 463:459 1 bit Reserved, must be 0. 3674 464 1 bit RESERVED_464 Deprecated, must be 0. 3675 467:465 3 bits Reserved, must be 0. 3676 468 1 bit RESERVED_468 Deprecated, must be 0. 3677 469:471 3 bits Reserved, must be 0. 3678 511:472 5 bytes Reserved, must be 0. 3679 512 **Total size 64 bytes.** 3680 ======= ==================================================================== 3681 3682.. 3683 3684 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3685 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3686 3687 ======= ======= =============================== =========================================================================== 3688 Bits Size Field Name Description 3689 ======= ======= =============================== =========================================================================== 3690 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3691 blocks used by each work-item; 3692 granularity is device 3693 specific: 3694 3695 GFX6-GFX9 3696 - vgprs_used 0..256 3697 - max(0, ceil(vgprs_used / 4) - 1) 3698 GFX90A 3699 - vgprs_used 0..512 3700 - vgprs_used = align(arch_vgprs, 4) 3701 + acc_vgprs 3702 - max(0, ceil(vgprs_used / 8) - 1) 3703 GFX10 (wavefront size 64) 3704 - max_vgpr 1..256 3705 - max(0, ceil(vgprs_used / 4) - 1) 3706 GFX10 (wavefront size 32) 3707 - max_vgpr 1..256 3708 - max(0, ceil(vgprs_used / 8) - 1) 3709 3710 Where vgprs_used is defined 3711 as the highest VGPR number 3712 explicitly referenced plus 3713 one. 3714 3715 Used by CP to set up 3716 ``COMPUTE_PGM_RSRC1.VGPRS``. 3717 3718 The 3719 :ref:`amdgpu-assembler` 3720 calculates this 3721 automatically for the 3722 selected processor from 3723 values provided to the 3724 `.amdhsa_kernel` directive 3725 by the 3726 `.amdhsa_next_free_vgpr` 3727 nested directive (see 3728 :ref:`amdhsa-kernel-directives-table`). 3729 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3730 blocks used by a wavefront; 3731 granularity is device 3732 specific: 3733 3734 GFX6-GFX8 3735 - sgprs_used 0..112 3736 - max(0, ceil(sgprs_used / 8) - 1) 3737 GFX9 3738 - sgprs_used 0..112 3739 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3740 GFX10 3741 Reserved, must be 0. 3742 (128 SGPRs always 3743 allocated.) 3744 3745 Where sgprs_used is 3746 defined as the highest 3747 SGPR number explicitly 3748 referenced plus one, plus 3749 a target specific number 3750 of additional special 3751 SGPRs for VCC, 3752 FLAT_SCRATCH (GFX7+) and 3753 XNACK_MASK (GFX8+), and 3754 any additional 3755 target specific 3756 limitations. It does not 3757 include the 16 SGPRs added 3758 if a trap handler is 3759 enabled. 3760 3761 The target specific 3762 limitations and special 3763 SGPR layout are defined in 3764 the hardware 3765 documentation, which can 3766 be found in the 3767 :ref:`amdgpu-processors` 3768 table. 3769 3770 Used by CP to set up 3771 ``COMPUTE_PGM_RSRC1.SGPRS``. 3772 3773 The 3774 :ref:`amdgpu-assembler` 3775 calculates this 3776 automatically for the 3777 selected processor from 3778 values provided to the 3779 `.amdhsa_kernel` directive 3780 by the 3781 `.amdhsa_next_free_sgpr` 3782 and `.amdhsa_reserve_*` 3783 nested directives (see 3784 :ref:`amdhsa-kernel-directives-table`). 3785 11:10 2 bits PRIORITY Must be 0. 3786 3787 Start executing wavefront 3788 at the specified priority. 3789 3790 CP is responsible for 3791 filling in 3792 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3793 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3794 with specified rounding 3795 mode for single (32 3796 bit) floating point 3797 precision floating point 3798 operations. 3799 3800 Floating point rounding 3801 mode values are defined in 3802 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3803 3804 Used by CP to set up 3805 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3806 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3807 with specified rounding 3808 denorm mode for half/double (16 3809 and 64-bit) floating point 3810 precision floating point 3811 operations. 3812 3813 Floating point rounding 3814 mode values are defined in 3815 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3816 3817 Used by CP to set up 3818 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3819 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3820 with specified denorm mode 3821 for single (32 3822 bit) floating point 3823 precision floating point 3824 operations. 3825 3826 Floating point denorm mode 3827 values are defined in 3828 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3829 3830 Used by CP to set up 3831 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3832 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3833 with specified denorm mode 3834 for half/double (16 3835 and 64-bit) floating point 3836 precision floating point 3837 operations. 3838 3839 Floating point denorm mode 3840 values are defined in 3841 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3842 3843 Used by CP to set up 3844 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3845 20 1 bit PRIV Must be 0. 3846 3847 Start executing wavefront 3848 in privilege trap handler 3849 mode. 3850 3851 CP is responsible for 3852 filling in 3853 ``COMPUTE_PGM_RSRC1.PRIV``. 3854 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3855 with DX10 clamp mode 3856 enabled. Used by the vector 3857 ALU to force DX10 style 3858 treatment of NaN's (when 3859 set, clamp NaN to zero, 3860 otherwise pass NaN 3861 through). 3862 3863 Used by CP to set up 3864 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3865 22 1 bit DEBUG_MODE Must be 0. 3866 3867 Start executing wavefront 3868 in single step mode. 3869 3870 CP is responsible for 3871 filling in 3872 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3873 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3874 with IEEE mode 3875 enabled. Floating point 3876 opcodes that support 3877 exception flag gathering 3878 will quiet and propagate 3879 signaling-NaN inputs per 3880 IEEE 754-2008. Min_dx10 and 3881 max_dx10 become IEEE 3882 754-2008 compliant due to 3883 signaling-NaN propagation 3884 and quieting. 3885 3886 Used by CP to set up 3887 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3888 24 1 bit BULKY Must be 0. 3889 3890 Only one work-group allowed 3891 to execute on a compute 3892 unit. 3893 3894 CP is responsible for 3895 filling in 3896 ``COMPUTE_PGM_RSRC1.BULKY``. 3897 25 1 bit CDBG_USER Must be 0. 3898 3899 Flag that can be used to 3900 control debugging code. 3901 3902 CP is responsible for 3903 filling in 3904 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3905 26 1 bit FP16_OVFL GFX6-GFX8 3906 Reserved, must be 0. 3907 GFX9-GFX10 3908 Wavefront starts execution 3909 with specified fp16 overflow 3910 mode. 3911 3912 - If 0, fp16 overflow generates 3913 +/-INF values. 3914 - If 1, fp16 overflow that is the 3915 result of an +/-INF input value 3916 or divide by 0 produces a +/-INF, 3917 otherwise clamps computed 3918 overflow to +/-MAX_FP16 as 3919 appropriate. 3920 3921 Used by CP to set up 3922 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3923 28:27 2 bits Reserved, must be 0. 3924 29 1 bit WGP_MODE GFX6-GFX9 3925 Reserved, must be 0. 3926 GFX10 3927 - If 0 execute work-groups in 3928 CU wavefront execution mode. 3929 - If 1 execute work-groups on 3930 in WGP wavefront execution mode. 3931 3932 See :ref:`amdgpu-amdhsa-memory-model`. 3933 3934 Used by CP to set up 3935 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3936 30 1 bit MEM_ORDERED GFX6-GFX9 3937 Reserved, must be 0. 3938 GFX10 3939 Controls the behavior of the 3940 s_waitcnt's vmcnt and vscnt 3941 counters. 3942 3943 - If 0 vmcnt reports completion 3944 of load and atomic with return 3945 out of order with sample 3946 instructions, and the vscnt 3947 reports the completion of 3948 store and atomic without 3949 return in order. 3950 - If 1 vmcnt reports completion 3951 of load, atomic with return 3952 and sample instructions in 3953 order, and the vscnt reports 3954 the completion of store and 3955 atomic without return in order. 3956 3957 Used by CP to set up 3958 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3959 31 1 bit FWD_PROGRESS GFX6-GFX9 3960 Reserved, must be 0. 3961 GFX10 3962 - If 0 execute SIMD wavefronts 3963 using oldest first policy. 3964 - If 1 execute SIMD wavefronts to 3965 ensure wavefronts will make some 3966 forward progress. 3967 3968 Used by CP to set up 3969 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3970 32 **Total size 4 bytes** 3971 ======= =================================================================================================================== 3972 3973.. 3974 3975 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3976 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3977 3978 ======= ======= =============================== =========================================================================== 3979 Bits Size Field Name Description 3980 ======= ======= =============================== =========================================================================== 3981 0 1 bit ENABLE_PRIVATE_SEGMENT Enable the setup of the 3982 private segment. 3983 3984 In addition, enable the 3985 setup of the SGPR 3986 wavefront scratch offset 3987 system register (see 3988 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3989 3990 Used by CP to set up 3991 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3992 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3993 user data registers 3994 requested. This number must 3995 match the number of user 3996 data registers enabled. 3997 3998 Used by CP to set up 3999 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4000 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4001 4002 This bit represents 4003 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4004 which is set by the CP if 4005 the runtime has installed a 4006 trap handler. 4007 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4008 system SGPR register for 4009 the work-group id in the X 4010 dimension (see 4011 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4012 4013 Used by CP to set up 4014 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4015 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4016 system SGPR register for 4017 the work-group id in the Y 4018 dimension (see 4019 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4020 4021 Used by CP to set up 4022 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4023 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4024 system SGPR register for 4025 the work-group id in the Z 4026 dimension (see 4027 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4028 4029 Used by CP to set up 4030 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4031 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4032 system SGPR register for 4033 work-group information (see 4034 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4035 4036 Used by CP to set up 4037 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4038 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4039 VGPR system registers used 4040 for the work-item ID. 4041 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4042 defines the values. 4043 4044 Used by CP to set up 4045 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4046 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4047 4048 Wavefront starts execution 4049 with address watch 4050 exceptions enabled which 4051 are generated when L1 has 4052 witnessed a thread access 4053 an *address of 4054 interest*. 4055 4056 CP is responsible for 4057 filling in the address 4058 watch bit in 4059 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4060 according to what the 4061 runtime requests. 4062 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4063 4064 Wavefront starts execution 4065 with memory violation 4066 exceptions exceptions 4067 enabled which are generated 4068 when a memory violation has 4069 occurred for this wavefront from 4070 L1 or LDS 4071 (write-to-read-only-memory, 4072 mis-aligned atomic, LDS 4073 address out of range, 4074 illegal address, etc.). 4075 4076 CP sets the memory 4077 violation bit in 4078 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4079 according to what the 4080 runtime requests. 4081 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4082 4083 CP uses the rounded value 4084 from the dispatch packet, 4085 not this value, as the 4086 dispatch may contain 4087 dynamically allocated group 4088 segment memory. CP writes 4089 directly to 4090 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4091 4092 Amount of group segment 4093 (LDS) to allocate for each 4094 work-group. Granularity is 4095 device specific: 4096 4097 GFX6: 4098 roundup(lds-size / (64 * 4)) 4099 GFX7-GFX10: 4100 roundup(lds-size / (128 * 4)) 4101 4102 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4103 _INVALID_OPERATION with specified exceptions 4104 enabled. 4105 4106 Used by CP to set up 4107 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4108 (set from bits 0..6). 4109 4110 IEEE 754 FP Invalid 4111 Operation 4112 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4113 _SOURCE input operands is a 4114 denormal number 4115 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4116 _DIVISION_BY_ZERO Zero 4117 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4118 _OVERFLOW 4119 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4120 _UNDERFLOW 4121 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4122 _INEXACT 4123 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4124 _ZERO (rcp_iflag_f32 instruction 4125 only) 4126 31 1 bit Reserved, must be 0. 4127 32 **Total size 4 bytes.** 4128 ======= =================================================================================================================== 4129 4130.. 4131 4132 .. table:: compute_pgm_rsrc3 for GFX90A 4133 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4134 4135 ======= ======= =============================== =========================================================================== 4136 Bits Size Field Name Description 4137 ======= ======= =============================== =========================================================================== 4138 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4139 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4140 63 - accum-offset = 256. 4141 6:15 10 Reserved, must be 0. 4142 bits 4143 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4144 launched in the same CU. 4145 - If 1 the waves of a work-group can be 4146 launched in different CUs. The waves 4147 cannot use S_BARRIER or LDS. 4148 17:31 15 Reserved, must be 0. 4149 bits 4150 32 **Total size 4 bytes.** 4151 ======= =================================================================================================================== 4152 4153.. 4154 4155 .. table:: compute_pgm_rsrc3 for GFX10 4156 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4157 4158 ======= ======= =============================== =========================================================================== 4159 Bits Size Field Name Description 4160 ======= ======= =============================== =========================================================================== 4161 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4162 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4163 31:4 28 Reserved, must be 0. 4164 bits 4165 32 **Total size 4 bytes.** 4166 ======= =================================================================================================================== 4167 4168.. 4169 4170 .. table:: Floating Point Rounding Mode Enumeration Values 4171 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4172 4173 ====================================== ===== ============================== 4174 Enumeration Name Value Description 4175 ====================================== ===== ============================== 4176 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4177 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4178 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4179 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4180 ====================================== ===== ============================== 4181 4182.. 4183 4184 .. table:: Floating Point Denorm Mode Enumeration Values 4185 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4186 4187 ====================================== ===== ============================== 4188 Enumeration Name Value Description 4189 ====================================== ===== ============================== 4190 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4191 Denorms 4192 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4193 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4194 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4195 ====================================== ===== ============================== 4196 4197.. 4198 4199 .. table:: System VGPR Work-Item ID Enumeration Values 4200 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4201 4202 ======================================== ===== ============================ 4203 Enumeration Name Value Description 4204 ======================================== ===== ============================ 4205 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4206 ID. 4207 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4208 dimensions ID. 4209 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4210 dimensions ID. 4211 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4212 ======================================== ===== ============================ 4213 4214.. _amdgpu-amdhsa-initial-kernel-execution-state: 4215 4216Initial Kernel Execution State 4217~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4218 4219This section defines the register state that will be set up by the packet 4220processor prior to the start of execution of every wavefront. This is limited by 4221the constraints of the hardware controllers of CP/ADC/SPI. 4222 4223The order of the SGPR registers is defined, but the compiler can specify which 4224ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4225fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4226for enabled registers are dense starting at SGPR0: the first enabled register is 4227SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4228an SGPR number. 4229 4230The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4231all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4232using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4233actually initialized. These are then immediately followed by the System SGPRs 4234that are set up by ADC/SPI and can have different values for each wavefront of 4235the grid dispatch. 4236 4237SGPR register initial state is defined in 4238:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4239 4240 .. table:: SGPR Register Set Up Order 4241 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4242 4243 ========== ========================== ====== ============================== 4244 SGPR Order Name Number Description 4245 (kernel descriptor enable of 4246 field) SGPRs 4247 ========== ========================== ====== ============================== 4248 First Private Segment Buffer 4 See 4249 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4250 _segment_buffer) 4251 then Dispatch Ptr 2 64-bit address of AQL dispatch 4252 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4253 actually executing. 4254 then Queue Ptr 2 64-bit address of amd_queue_t 4255 (enable_sgpr_queue_ptr) object for AQL queue on which 4256 the dispatch packet was 4257 queued. 4258 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4259 (enable_sgpr_kernarg segment. This is directly 4260 _segment_ptr) copied from the 4261 kernarg_address in the kernel 4262 dispatch packet. 4263 4264 Having CP load it once avoids 4265 loading it at the beginning of 4266 every wavefront. 4267 then Dispatch Id 2 64-bit Dispatch ID of the 4268 (enable_sgpr_dispatch_id) dispatch packet being 4269 executed. 4270 then Flat Scratch Init 2 See 4271 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4272 then Private Segment Size 1 The 32-bit byte size of a 4273 (enable_sgpr_private single 4274 work-item's 4275 scratch_segment_size) memory 4276 allocation. This is the 4277 value from the kernel 4278 dispatch packet Private 4279 Segment Byte Size rounded up 4280 by CP to a multiple of 4281 DWORD. 4282 4283 Having CP load it once avoids 4284 loading it at the beginning of 4285 every wavefront. 4286 4287 This is not used for 4288 GFX7-GFX8 since it is the same 4289 value as the second SGPR of 4290 Flat Scratch Init. However, it 4291 may be needed for GFX9-GFX10 which 4292 changes the meaning of the 4293 Flat Scratch Init value. 4294 then Grid Work-Group Count X 1 32-bit count of the number of 4295 (enable_sgpr_grid work-groups in the X dimension 4296 _workgroup_count_X) for the grid being 4297 executed. Computed from the 4298 fields in the kernel dispatch 4299 packet as ((grid_size.x + 4300 workgroup_size.x - 1) / 4301 workgroup_size.x). 4302 then Grid Work-Group Count Y 1 32-bit count of the number of 4303 (enable_sgpr_grid work-groups in the Y dimension 4304 _workgroup_count_Y && for the grid being 4305 less than 16 previous executed. Computed from the 4306 SGPRs) fields in the kernel dispatch 4307 packet as ((grid_size.y + 4308 workgroup_size.y - 1) / 4309 workgroupSize.y). 4310 4311 Only initialized if <16 4312 previous SGPRs initialized. 4313 then Grid Work-Group Count Z 1 32-bit count of the number of 4314 (enable_sgpr_grid work-groups in the Z dimension 4315 _workgroup_count_Z && for the grid being 4316 less than 16 previous executed. Computed from the 4317 SGPRs) fields in the kernel dispatch 4318 packet as ((grid_size.z + 4319 workgroup_size.z - 1) / 4320 workgroupSize.z). 4321 4322 Only initialized if <16 4323 previous SGPRs initialized. 4324 then Work-Group Id X 1 32-bit work-group id in X 4325 (enable_sgpr_workgroup_id dimension of grid for 4326 _X) wavefront. 4327 then Work-Group Id Y 1 32-bit work-group id in Y 4328 (enable_sgpr_workgroup_id dimension of grid for 4329 _Y) wavefront. 4330 then Work-Group Id Z 1 32-bit work-group id in Z 4331 (enable_sgpr_workgroup_id dimension of grid for 4332 _Z) wavefront. 4333 then Work-Group Info 1 {first_wavefront, 14'b0000, 4334 (enable_sgpr_workgroup ordered_append_term[10:0], 4335 _info) threadgroup_size_in_wavefronts[5:0]} 4336 then Scratch Wavefront Offset 1 See 4337 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4338 _segment_wavefront_offset) and 4339 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4340 ========== ========================== ====== ============================== 4341 4342The order of the VGPR registers is defined, but the compiler can specify which 4343ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4344fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4345for enabled registers are dense starting at VGPR0: the first enabled register is 4346VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4347VGPR number. 4348 4349There are different methods used for the VGPR initial state: 4350 4351* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4352 specifies otherwise, a separate VGPR register is used per work-item ID. The 4353 VGPR register initial state for this method is defined in 4354 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4355* If *Target Properties* column of :ref:`amdgpu-processor-table` 4356 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4357 for all work-item IDs. The register layout for this method is defined in 4358 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4359 4360 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4361 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4362 4363 ========== ========================== ====== ============================== 4364 VGPR Order Name Number Description 4365 (kernel descriptor enable of 4366 field) VGPRs 4367 ========== ========================== ====== ============================== 4368 First Work-Item Id X 1 32-bit work-item id in X 4369 (Always initialized) dimension of work-group for 4370 wavefront lane. 4371 then Work-Item Id Y 1 32-bit work-item id in Y 4372 (enable_vgpr_workitem_id dimension of work-group for 4373 > 0) wavefront lane. 4374 then Work-Item Id Z 1 32-bit work-item id in Z 4375 (enable_vgpr_workitem_id dimension of work-group for 4376 > 1) wavefront lane. 4377 ========== ========================== ====== ============================== 4378 4379.. 4380 4381 .. table:: Register Layout for Packed Work-Item ID Method 4382 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4383 4384 ======= ======= ================ ========================================= 4385 Bits Size Field Name Description 4386 ======= ======= ================ ========================================= 4387 0:9 10 bits Work-Item Id X Work-item id in X 4388 dimension of work-group for 4389 wavefront lane. 4390 4391 Always initialized. 4392 4393 10:19 10 bits Work-Item Id Y Work-item id in Y 4394 dimension of work-group for 4395 wavefront lane. 4396 4397 Initialized if enable_vgpr_workitem_id > 4398 0, otherwise set to 0. 4399 20:29 10 bits Work-Item Id Z Work-item id in Z 4400 dimension of work-group for 4401 wavefront lane. 4402 4403 Initialized if enable_vgpr_workitem_id > 4404 1, otherwise set to 0. 4405 30:31 2 bits Reserved, set to 0. 4406 ======= ======= ================ ========================================= 4407 4408The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4409 44101. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4411 registers. 44122. Work-group Id registers X, Y, Z are set by ADC which supports any 4413 combination including none. 44143. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4415 its value cannot be included with the flat scratch init value which is per 4416 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 44174. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4418 or (X, Y, Z). 44195. Flat Scratch register pair initialization is described in 4420 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4421 4422The global segment can be accessed either using buffer instructions (GFX6 which 4423has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4424instructions (GFX9-GFX10). 4425 4426If buffer operations are used, then the compiler can generate a V# with the 4427following properties: 4428 4429* base address of 0 4430* no swizzle 4431* ATC: 1 if IOMMU present (such as APU) 4432* ptr64: 1 4433* MTYPE set to support memory coherence that matches the runtime (such as CC for 4434 APU and NC for dGPU). 4435 4436.. _amdgpu-amdhsa-kernel-prolog: 4437 4438Kernel Prolog 4439~~~~~~~~~~~~~ 4440 4441The compiler performs initialization in the kernel prologue depending on the 4442target and information about things like stack usage in the kernel and called 4443functions. Some of this initialization requires the compiler to request certain 4444User and System SGPRs be present in the 4445:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4446:ref:`amdgpu-amdhsa-kernel-descriptor`. 4447 4448.. _amdgpu-amdhsa-kernel-prolog-cfi: 4449 4450CFI 4451+++ 4452 44531. The CFI return address is undefined. 4454 44552. The CFI CFA is defined using an expression which evaluates to a location 4456 description that comprises one memory location description for the 4457 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4458 4459.. _amdgpu-amdhsa-kernel-prolog-m0: 4460 4461M0 4462++ 4463 4464GFX6-GFX8 4465 The M0 register must be initialized with a value at least the total LDS size 4466 if the kernel may access LDS via DS or flat operations. Total LDS size is 4467 available in dispatch packet. For M0, it is also possible to use maximum 4468 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4469 GFX7-GFX8). 4470GFX9-GFX10 4471 The M0 register is not used for range checking LDS accesses and so does not 4472 need to be initialized in the prolog. 4473 4474.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4475 4476Stack Pointer 4477+++++++++++++ 4478 4479If the kernel has function calls it must set up the ABI stack pointer described 4480in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4481SGPR32 to the unswizzled scratch offset of the address past the last local 4482allocation. 4483 4484.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4485 4486Frame Pointer 4487+++++++++++++ 4488 4489If the kernel needs a frame pointer for the reasons defined in 4490``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4491kernel prolog. If a frame pointer is not required then all uses of the frame 4492pointer are replaced with immediate ``0`` offsets. 4493 4494.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4495 4496Flat Scratch 4497++++++++++++ 4498 4499There are different methods used for initializing flat scratch: 4500 4501* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4502 specifies *Does not support generic address space*: 4503 4504 Flat scratch is not supported and there is no flat scratch register pair. 4505 4506* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4507 specifies *Offset flat scratch*: 4508 4509 If the kernel or any function it calls may use flat operations to access 4510 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4511 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4512 Scratch Wavefront Offset SGPR registers (see 4513 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4514 4515 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4516 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4517 being managed by SPI for the queue executing the kernel dispatch. This is 4518 the same value used in the Scratch Segment Buffer V# base address. 4519 4520 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4521 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4522 4523 The prolog must add the value of Scratch Wavefront Offset to get the 4524 wavefront's byte scratch backing memory offset from 4525 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4526 4527 The Scratch Wavefront Offset must also be used as an offset with Private 4528 segment address when using the Scratch Segment Buffer. 4529 4530 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4531 shifted by 8 before moving into FLAT_SCRATCH_HI. 4532 4533 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4534 SGPRn is the highest numbered SGPR allocated to the wavefront). 4535 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4536 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4537 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4538 aperture. 4539 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4540 work-items scratch memory usage. 4541 4542 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4543 checks that the value in the kernel dispatch packet Private Segment Byte 4544 Size is not larger and requests the runtime to increase the queue's scratch 4545 size if necessary. 4546 4547 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4548 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4549 loading it at the beginning of every wavefront. 4550 4551 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4552 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4553 in flat memory instructions. 4554 4555* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4556 specifies *Absolute flat scratch*: 4557 4558 If the kernel or any function it calls may use flat operations to access 4559 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4560 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4561 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4562 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4563 4564 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4565 memory being managed by SPI for the queue executing the kernel dispatch. 4566 4567 CP obtains this from the runtime. 4568 4569 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4570 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4571 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4572 memory instructions. 4573 4574 The Scratch Wavefront Offset must also be used as an offset with Private 4575 segment address when using the Scratch Segment Buffer (see 4576 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4577 4578.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4579 4580Private Segment Buffer 4581++++++++++++++++++++++ 4582 4583Private Segment Buffer SGPR register is used to initilize 4 SGPRs 4584that are used as a V# to access scratch. CP uses the value provided by the 4585runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4586access the private memory space using a segment address. See 4587:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4588 4589The scratch V# is a four-aligned SGPR and always selected for the kernel as 4590follows: 4591 4592 - If it is known during instruction selection that there is stack usage, 4593 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4594 optimizations are disabled (``-O0``), if stack objects already exist (for 4595 locals, etc.), or if there are any function calls. 4596 4597 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4598 are reserved for the tentative scratch V#. These will be used if it is 4599 determined that spilling is needed. 4600 4601 - If no use is made of the tentative scratch V#, then it is unreserved, 4602 and the register count is determined ignoring it. 4603 - If use is made of the tentative scratch V#, then its register numbers 4604 are shifted to the first four-aligned SGPR index after the highest one 4605 allocated by the register allocator, and all uses are updated. The 4606 register count includes them in the shifted location. 4607 - In either case, if the processor has the SGPR allocation bug, the 4608 tentative allocation is not shifted or unreserved in order to ensure 4609 the register count is higher to workaround the bug. 4610 4611 .. note:: 4612 4613 This approach of using a tentative scratch V# and shifting the register 4614 numbers if used avoids having to perform register allocation a second 4615 time if the tentative V# is eliminated. This is more efficient and 4616 avoids the problem that the second register allocation may perform 4617 spilling which will fail as there is no longer a scratch V#. 4618 4619When the kernel prolog code is being emitted it is known whether the scratch V# 4620described above is actually used. If it is, the prolog code must set it up by 4621copying the Private Segment Buffer to the scratch V# registers and then adding 4622the Private Segment Wavefront Offset to the queue base address in the V#. The 4623result is a V# with a base address pointing to the beginning of the wavefront 4624scratch backing memory. 4625 4626The Private Segment Buffer is always requested, but the Private Segment 4627Wavefront Offset is only requested if it is used (see 4628:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4629 4630.. _amdgpu-amdhsa-memory-model: 4631 4632Memory Model 4633~~~~~~~~~~~~ 4634 4635This section describes the mapping of the LLVM memory model onto AMDGPU machine 4636code (see :ref:`memmodel`). 4637 4638The AMDGPU backend supports the memory synchronization scopes specified in 4639:ref:`amdgpu-memory-scopes`. 4640 4641The code sequences used to implement the memory model specify the order of 4642instructions that a single thread must execute. The ``s_waitcnt`` and cache 4643management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4644to other memory instructions executed by the same thread. This allows them to be 4645moved earlier or later which can allow them to be combined with other instances 4646of the same instruction, or hoisted/sunk out of loops to improve performance. 4647Only the instructions related to the memory model are given; additional 4648``s_waitcnt`` instructions are required to ensure registers are defined before 4649being used. These may be able to be combined with the memory model ``s_waitcnt`` 4650instructions as described above. 4651 4652The AMDGPU backend supports the following memory models: 4653 4654 HSA Memory Model [HSA]_ 4655 The HSA memory model uses a single happens-before relation for all address 4656 spaces (see :ref:`amdgpu-address-spaces`). 4657 OpenCL Memory Model [OpenCL]_ 4658 The OpenCL memory model which has separate happens-before relations for the 4659 global and local address spaces. Only a fence specifying both global and 4660 local address space, and seq_cst instructions join the relationships. Since 4661 the LLVM ``memfence`` instruction does not allow an address space to be 4662 specified the OpenCL fence has to conservatively assume both local and 4663 global address space was specified. However, optimizations can often be 4664 done to eliminate the additional ``s_waitcnt`` instructions when there are 4665 no intervening memory instructions which access the corresponding address 4666 space. The code sequences in the table indicate what can be omitted for the 4667 OpenCL memory. The target triple environment is used to determine if the 4668 source language is OpenCL (see :ref:`amdgpu-opencl`). 4669 4670``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4671operations. 4672 4673``buffer/global/flat_load/store/atomic`` instructions to global memory are 4674termed vector memory operations. 4675 4676Private address space uses ``buffer_load/store`` using the scratch V# 4677(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4678is accessing the memory, atomic memory orderings are not meaningful, and all 4679accesses are treated as non-atomic. 4680 4681Constant address space uses ``buffer/global_load`` instructions (or equivalent 4682scalar memory instructions). Since the constant address space contents do not 4683change during the execution of a kernel dispatch it is not legal to perform 4684stores, and atomic memory orderings are not meaningful, and all accesses are 4685treated as non-atomic. 4686 4687A memory synchronization scope wider than work-group is not meaningful for the 4688group (LDS) address space and is treated as work-group. 4689 4690The memory model does not support the region address space which is treated as 4691non-atomic. 4692 4693Acquire memory ordering is not meaningful on store atomic instructions and is 4694treated as non-atomic. 4695 4696Release memory ordering is not meaningful on load atomic instructions and is 4697treated a non-atomic. 4698 4699Acquire-release memory ordering is not meaningful on load or store atomic 4700instructions and is treated as acquire and release respectively. 4701 4702The memory order also adds the single thread optimization constraints defined in 4703table 4704:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4705 4706 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4707 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4708 4709 ============ ============================================================== 4710 LLVM Memory Optimization Constraints 4711 Ordering 4712 ============ ============================================================== 4713 unordered *none* 4714 monotonic *none* 4715 acquire - If a load atomic/atomicrmw then no following load/load 4716 atomic/store/store atomic/atomicrmw/fence instruction can be 4717 moved before the acquire. 4718 - If a fence then same as load atomic, plus no preceding 4719 associated fence-paired-atomic can be moved after the fence. 4720 release - If a store atomic/atomicrmw then no preceding load/load 4721 atomic/store/store atomic/atomicrmw/fence instruction can be 4722 moved after the release. 4723 - If a fence then same as store atomic, plus no following 4724 associated fence-paired-atomic can be moved before the 4725 fence. 4726 acq_rel Same constraints as both acquire and release. 4727 seq_cst - If a load atomic then same constraints as acquire, plus no 4728 preceding sequentially consistent load atomic/store 4729 atomic/atomicrmw/fence instruction can be moved after the 4730 seq_cst. 4731 - If a store atomic then the same constraints as release, plus 4732 no following sequentially consistent load atomic/store 4733 atomic/atomicrmw/fence instruction can be moved before the 4734 seq_cst. 4735 - If an atomicrmw/fence then same constraints as acq_rel. 4736 ============ ============================================================== 4737 4738The code sequences used to implement the memory model are defined in the 4739following sections: 4740 4741* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4742* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4743* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4744 4745.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4746 4747Memory Model GFX6-GFX9 4748++++++++++++++++++++++ 4749 4750For GFX6-GFX9: 4751 4752* Each agent has multiple shader arrays (SA). 4753* Each SA has multiple compute units (CU). 4754* Each CU has multiple SIMDs that execute wavefronts. 4755* The wavefronts for a single work-group are executed in the same CU but may be 4756 executed by different SIMDs. 4757* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4758 executing on it. 4759* All LDS operations of a CU are performed as wavefront wide operations in a 4760 global order and involve no caching. Completion is reported to a wavefront in 4761 execution order. 4762* The LDS memory has multiple request queues shared by the SIMDs of a 4763 CU. Therefore, the LDS operations performed by different wavefronts of a 4764 work-group can be reordered relative to each other, which can result in 4765 reordering the visibility of vector memory operations with respect to LDS 4766 operations of other wavefronts in the same work-group. A ``s_waitcnt 4767 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4768 vector memory operations between wavefronts of a work-group, but not between 4769 operations performed by the same wavefront. 4770* The vector memory operations are performed as wavefront wide operations and 4771 completion is reported to a wavefront in execution order. The exception is 4772 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4773 vector memory order if they access LDS memory, and out of LDS operation order 4774 if they access global memory. 4775* The vector memory operations access a single vector L1 cache shared by all 4776 SIMDs a CU. Therefore, no special action is required for coherence between the 4777 lanes of a single wavefront, or for coherence between wavefronts in the same 4778 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4779 wavefronts executing in different work-groups as they may be executing on 4780 different CUs. 4781* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4782 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4783 scalar operations are used in a restricted way so do not impact the memory 4784 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4785* The vector and scalar memory operations use an L2 cache shared by all CUs on 4786 the same agent. 4787* The L2 cache has independent channels to service disjoint ranges of virtual 4788 addresses. 4789* Each CU has a separate request queue per channel. Therefore, the vector and 4790 scalar memory operations performed by wavefronts executing in different 4791 work-groups (which may be executing on different CUs) of an agent can be 4792 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4793 ensure synchronization between vector memory operations of different CUs. It 4794 ensures a previous vector memory operation has completed before executing a 4795 subsequent vector memory or LDS operation and so can be used to meet the 4796 requirements of acquire and release. 4797* The L2 cache can be kept coherent with other agents on some targets, or ranges 4798 of virtual addresses can be set up to bypass it to ensure system coherence. 4799 4800Scalar memory operations are only used to access memory that is proven to not 4801change during the execution of the kernel dispatch. This includes constant 4802address space and global address space for program scope ``const`` variables. 4803Therefore, the kernel machine code does not have to maintain the scalar cache to 4804ensure it is coherent with the vector caches. The scalar and vector caches are 4805invalidated between kernel dispatches by CP since constant address space data 4806may change between kernel dispatch executions. See 4807:ref:`amdgpu-amdhsa-memory-spaces`. 4808 4809The one exception is if scalar writes are used to spill SGPR registers. In this 4810case the AMDGPU backend ensures the memory location used to spill is never 4811accessed by vector memory operations at the same time. If scalar writes are used 4812then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4813return since the locations may be used for vector memory instructions by a 4814future wavefront that uses the same scratch area, or a function call that 4815creates a frame at the same address, respectively. There is no need for a 4816``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4817 4818For kernarg backing memory: 4819 4820* CP invalidates the L1 cache at the start of each kernel dispatch. 4821* On dGPU the kernarg backing memory is allocated in host memory accessed as 4822 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4823 causes it to be treated as non-volatile and so is not invalidated by 4824 ``*_vol``. 4825* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4826 and so the L2 cache will be coherent with the CPU and other agents. 4827 4828Scratch backing memory (which is used for the private address space) is accessed 4829with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4830only accessed by a single thread, and is always write-before-read, there is 4831never a need to invalidate these entries from the L1 cache. Hence all cache 4832invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4833 4834The code sequences used to implement the memory model for GFX6-GFX9 are defined 4835in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4836 4837 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4838 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4839 4840 ============ ============ ============== ========== ================================ 4841 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4842 Ordering Sync Scope Address GFX6-GFX9 4843 Space 4844 ============ ============ ============== ========== ================================ 4845 **Non-Atomic** 4846 ------------------------------------------------------------------------------------ 4847 load *none* *none* - global - !volatile & !nontemporal 4848 - generic 4849 - private 1. buffer/global/flat_load 4850 - constant 4851 - !volatile & nontemporal 4852 4853 1. buffer/global/flat_load 4854 glc=1 slc=1 4855 4856 - volatile 4857 4858 1. buffer/global/flat_load 4859 glc=1 4860 2. s_waitcnt vmcnt(0) 4861 4862 - Must happen before 4863 any following volatile 4864 global/generic 4865 load/store. 4866 - Ensures that 4867 volatile 4868 operations to 4869 different 4870 addresses will not 4871 be reordered by 4872 hardware. 4873 4874 load *none* *none* - local 1. ds_load 4875 store *none* *none* - global - !volatile & !nontemporal 4876 - generic 4877 - private 1. buffer/global/flat_store 4878 - constant 4879 - !volatile & nontemporal 4880 4881 1. buffer/global/flat_store 4882 glc=1 slc=1 4883 4884 - volatile 4885 4886 1. buffer/global/flat_store 4887 2. s_waitcnt vmcnt(0) 4888 4889 - Must happen before 4890 any following volatile 4891 global/generic 4892 load/store. 4893 - Ensures that 4894 volatile 4895 operations to 4896 different 4897 addresses will not 4898 be reordered by 4899 hardware. 4900 4901 store *none* *none* - local 1. ds_store 4902 **Unordered Atomic** 4903 ------------------------------------------------------------------------------------ 4904 load atomic unordered *any* *any* *Same as non-atomic*. 4905 store atomic unordered *any* *any* *Same as non-atomic*. 4906 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4907 **Monotonic Atomic** 4908 ------------------------------------------------------------------------------------ 4909 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4910 - wavefront - local 4911 - workgroup - generic 4912 load atomic monotonic - agent - global 1. buffer/global/flat_load 4913 - system - generic glc=1 4914 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4915 - wavefront - generic 4916 - workgroup 4917 - agent 4918 - system 4919 store atomic monotonic - singlethread - local 1. ds_store 4920 - wavefront 4921 - workgroup 4922 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4923 - wavefront - generic 4924 - workgroup 4925 - agent 4926 - system 4927 atomicrmw monotonic - singlethread - local 1. ds_atomic 4928 - wavefront 4929 - workgroup 4930 **Acquire Atomic** 4931 ------------------------------------------------------------------------------------ 4932 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4933 - wavefront - local 4934 - generic 4935 load atomic acquire - workgroup - global 1. buffer/global_load 4936 load atomic acquire - workgroup - local 1. ds/flat_load 4937 - generic 2. s_waitcnt lgkmcnt(0) 4938 4939 - If OpenCL, omit. 4940 - Must happen before 4941 any following 4942 global/generic 4943 load/load 4944 atomic/store/store 4945 atomic/atomicrmw. 4946 - Ensures any 4947 following global 4948 data read is no 4949 older than a local load 4950 atomic value being 4951 acquired. 4952 4953 load atomic acquire - agent - global 1. buffer/global_load 4954 - system glc=1 4955 2. s_waitcnt vmcnt(0) 4956 4957 - Must happen before 4958 following 4959 buffer_wbinvl1_vol. 4960 - Ensures the load 4961 has completed 4962 before invalidating 4963 the cache. 4964 4965 3. buffer_wbinvl1_vol 4966 4967 - Must happen before 4968 any following 4969 global/generic 4970 load/load 4971 atomic/atomicrmw. 4972 - Ensures that 4973 following 4974 loads will not see 4975 stale global data. 4976 4977 load atomic acquire - agent - generic 1. flat_load glc=1 4978 - system 2. s_waitcnt vmcnt(0) & 4979 lgkmcnt(0) 4980 4981 - If OpenCL omit 4982 lgkmcnt(0). 4983 - Must happen before 4984 following 4985 buffer_wbinvl1_vol. 4986 - Ensures the flat_load 4987 has completed 4988 before invalidating 4989 the cache. 4990 4991 3. buffer_wbinvl1_vol 4992 4993 - Must happen before 4994 any following 4995 global/generic 4996 load/load 4997 atomic/atomicrmw. 4998 - Ensures that 4999 following loads 5000 will not see stale 5001 global data. 5002 5003 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5004 - wavefront - local 5005 - generic 5006 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5007 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5008 - generic 2. s_waitcnt lgkmcnt(0) 5009 5010 - If OpenCL, omit. 5011 - Must happen before 5012 any following 5013 global/generic 5014 load/load 5015 atomic/store/store 5016 atomic/atomicrmw. 5017 - Ensures any 5018 following global 5019 data read is no 5020 older than a local 5021 atomicrmw value 5022 being acquired. 5023 5024 atomicrmw acquire - agent - global 1. buffer/global_atomic 5025 - system 2. s_waitcnt vmcnt(0) 5026 5027 - Must happen before 5028 following 5029 buffer_wbinvl1_vol. 5030 - Ensures the 5031 atomicrmw has 5032 completed before 5033 invalidating the 5034 cache. 5035 5036 3. buffer_wbinvl1_vol 5037 5038 - Must happen before 5039 any following 5040 global/generic 5041 load/load 5042 atomic/atomicrmw. 5043 - Ensures that 5044 following loads 5045 will not see stale 5046 global data. 5047 5048 atomicrmw acquire - agent - generic 1. flat_atomic 5049 - system 2. s_waitcnt vmcnt(0) & 5050 lgkmcnt(0) 5051 5052 - If OpenCL, omit 5053 lgkmcnt(0). 5054 - Must happen before 5055 following 5056 buffer_wbinvl1_vol. 5057 - Ensures the 5058 atomicrmw has 5059 completed before 5060 invalidating the 5061 cache. 5062 5063 3. buffer_wbinvl1_vol 5064 5065 - Must happen before 5066 any following 5067 global/generic 5068 load/load 5069 atomic/atomicrmw. 5070 - Ensures that 5071 following loads 5072 will not see stale 5073 global data. 5074 5075 fence acquire - singlethread *none* *none* 5076 - wavefront 5077 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5078 5079 - If OpenCL and 5080 address space is 5081 not generic, omit. 5082 - However, since LLVM 5083 currently has no 5084 address space on 5085 the fence need to 5086 conservatively 5087 always generate. If 5088 fence had an 5089 address space then 5090 set to address 5091 space of OpenCL 5092 fence flag, or to 5093 generic if both 5094 local and global 5095 flags are 5096 specified. 5097 - Must happen after 5098 any preceding 5099 local/generic load 5100 atomic/atomicrmw 5101 with an equal or 5102 wider sync scope 5103 and memory ordering 5104 stronger than 5105 unordered (this is 5106 termed the 5107 fence-paired-atomic). 5108 - Must happen before 5109 any following 5110 global/generic 5111 load/load 5112 atomic/store/store 5113 atomic/atomicrmw. 5114 - Ensures any 5115 following global 5116 data read is no 5117 older than the 5118 value read by the 5119 fence-paired-atomic. 5120 5121 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5122 - system vmcnt(0) 5123 5124 - If OpenCL and 5125 address space is 5126 not generic, omit 5127 lgkmcnt(0). 5128 - However, since LLVM 5129 currently has no 5130 address space on 5131 the fence need to 5132 conservatively 5133 always generate 5134 (see comment for 5135 previous fence). 5136 - Could be split into 5137 separate s_waitcnt 5138 vmcnt(0) and 5139 s_waitcnt 5140 lgkmcnt(0) to allow 5141 them to be 5142 independently moved 5143 according to the 5144 following rules. 5145 - s_waitcnt vmcnt(0) 5146 must happen after 5147 any preceding 5148 global/generic load 5149 atomic/atomicrmw 5150 with an equal or 5151 wider sync scope 5152 and memory ordering 5153 stronger than 5154 unordered (this is 5155 termed the 5156 fence-paired-atomic). 5157 - s_waitcnt lgkmcnt(0) 5158 must happen after 5159 any preceding 5160 local/generic load 5161 atomic/atomicrmw 5162 with an equal or 5163 wider sync scope 5164 and memory ordering 5165 stronger than 5166 unordered (this is 5167 termed the 5168 fence-paired-atomic). 5169 - Must happen before 5170 the following 5171 buffer_wbinvl1_vol. 5172 - Ensures that the 5173 fence-paired atomic 5174 has completed 5175 before invalidating 5176 the 5177 cache. Therefore 5178 any following 5179 locations read must 5180 be no older than 5181 the value read by 5182 the 5183 fence-paired-atomic. 5184 5185 2. buffer_wbinvl1_vol 5186 5187 - Must happen before any 5188 following global/generic 5189 load/load 5190 atomic/store/store 5191 atomic/atomicrmw. 5192 - Ensures that 5193 following loads 5194 will not see stale 5195 global data. 5196 5197 **Release Atomic** 5198 ------------------------------------------------------------------------------------ 5199 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5200 - wavefront - local 5201 - generic 5202 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5203 - generic 5204 - If OpenCL, omit. 5205 - Must happen after 5206 any preceding 5207 local/generic 5208 load/store/load 5209 atomic/store 5210 atomic/atomicrmw. 5211 - Must happen before 5212 the following 5213 store. 5214 - Ensures that all 5215 memory operations 5216 to local have 5217 completed before 5218 performing the 5219 store that is being 5220 released. 5221 5222 2. buffer/global/flat_store 5223 store atomic release - workgroup - local 1. ds_store 5224 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5225 - system - generic vmcnt(0) 5226 5227 - If OpenCL and 5228 address space is 5229 not generic, omit 5230 lgkmcnt(0). 5231 - Could be split into 5232 separate s_waitcnt 5233 vmcnt(0) and 5234 s_waitcnt 5235 lgkmcnt(0) to allow 5236 them to be 5237 independently moved 5238 according to the 5239 following rules. 5240 - s_waitcnt vmcnt(0) 5241 must happen after 5242 any preceding 5243 global/generic 5244 load/store/load 5245 atomic/store 5246 atomic/atomicrmw. 5247 - s_waitcnt lgkmcnt(0) 5248 must happen after 5249 any preceding 5250 local/generic 5251 load/store/load 5252 atomic/store 5253 atomic/atomicrmw. 5254 - Must happen before 5255 the following 5256 store. 5257 - Ensures that all 5258 memory operations 5259 to memory have 5260 completed before 5261 performing the 5262 store that is being 5263 released. 5264 5265 2. buffer/global/flat_store 5266 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5267 - wavefront - local 5268 - generic 5269 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5270 - generic 5271 - If OpenCL, omit. 5272 - Must happen after 5273 any preceding 5274 local/generic 5275 load/store/load 5276 atomic/store 5277 atomic/atomicrmw. 5278 - Must happen before 5279 the following 5280 atomicrmw. 5281 - Ensures that all 5282 memory operations 5283 to local have 5284 completed before 5285 performing the 5286 atomicrmw that is 5287 being released. 5288 5289 2. buffer/global/flat_atomic 5290 atomicrmw release - workgroup - local 1. ds_atomic 5291 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5292 - system - generic vmcnt(0) 5293 5294 - If OpenCL, omit 5295 lgkmcnt(0). 5296 - Could be split into 5297 separate s_waitcnt 5298 vmcnt(0) and 5299 s_waitcnt 5300 lgkmcnt(0) to allow 5301 them to be 5302 independently moved 5303 according to the 5304 following rules. 5305 - s_waitcnt vmcnt(0) 5306 must happen after 5307 any preceding 5308 global/generic 5309 load/store/load 5310 atomic/store 5311 atomic/atomicrmw. 5312 - s_waitcnt lgkmcnt(0) 5313 must happen after 5314 any preceding 5315 local/generic 5316 load/store/load 5317 atomic/store 5318 atomic/atomicrmw. 5319 - Must happen before 5320 the following 5321 atomicrmw. 5322 - Ensures that all 5323 memory operations 5324 to global and local 5325 have completed 5326 before performing 5327 the atomicrmw that 5328 is being released. 5329 5330 2. buffer/global/flat_atomic 5331 fence release - singlethread *none* *none* 5332 - wavefront 5333 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5334 5335 - If OpenCL and 5336 address space is 5337 not generic, omit. 5338 - However, since LLVM 5339 currently has no 5340 address space on 5341 the fence need to 5342 conservatively 5343 always generate. If 5344 fence had an 5345 address space then 5346 set to address 5347 space of OpenCL 5348 fence flag, or to 5349 generic if both 5350 local and global 5351 flags are 5352 specified. 5353 - Must happen after 5354 any preceding 5355 local/generic 5356 load/load 5357 atomic/store/store 5358 atomic/atomicrmw. 5359 - Must happen before 5360 any following store 5361 atomic/atomicrmw 5362 with an equal or 5363 wider sync scope 5364 and memory ordering 5365 stronger than 5366 unordered (this is 5367 termed the 5368 fence-paired-atomic). 5369 - Ensures that all 5370 memory operations 5371 to local have 5372 completed before 5373 performing the 5374 following 5375 fence-paired-atomic. 5376 5377 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5378 - system vmcnt(0) 5379 5380 - If OpenCL and 5381 address space is 5382 not generic, omit 5383 lgkmcnt(0). 5384 - If OpenCL and 5385 address space is 5386 local, omit 5387 vmcnt(0). 5388 - However, since LLVM 5389 currently has no 5390 address space on 5391 the fence need to 5392 conservatively 5393 always generate. If 5394 fence had an 5395 address space then 5396 set to address 5397 space of OpenCL 5398 fence flag, or to 5399 generic if both 5400 local and global 5401 flags are 5402 specified. 5403 - Could be split into 5404 separate s_waitcnt 5405 vmcnt(0) and 5406 s_waitcnt 5407 lgkmcnt(0) to allow 5408 them to be 5409 independently moved 5410 according to the 5411 following rules. 5412 - s_waitcnt vmcnt(0) 5413 must happen after 5414 any preceding 5415 global/generic 5416 load/store/load 5417 atomic/store 5418 atomic/atomicrmw. 5419 - s_waitcnt lgkmcnt(0) 5420 must happen after 5421 any preceding 5422 local/generic 5423 load/store/load 5424 atomic/store 5425 atomic/atomicrmw. 5426 - Must happen before 5427 any following store 5428 atomic/atomicrmw 5429 with an equal or 5430 wider sync scope 5431 and memory ordering 5432 stronger than 5433 unordered (this is 5434 termed the 5435 fence-paired-atomic). 5436 - Ensures that all 5437 memory operations 5438 have 5439 completed before 5440 performing the 5441 following 5442 fence-paired-atomic. 5443 5444 **Acquire-Release Atomic** 5445 ------------------------------------------------------------------------------------ 5446 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5447 - wavefront - local 5448 - generic 5449 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5450 5451 - If OpenCL, omit. 5452 - Must happen after 5453 any preceding 5454 local/generic 5455 load/store/load 5456 atomic/store 5457 atomic/atomicrmw. 5458 - Must happen before 5459 the following 5460 atomicrmw. 5461 - Ensures that all 5462 memory operations 5463 to local have 5464 completed before 5465 performing the 5466 atomicrmw that is 5467 being released. 5468 5469 2. buffer/global_atomic 5470 5471 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5472 2. s_waitcnt lgkmcnt(0) 5473 5474 - If OpenCL, omit. 5475 - Must happen before 5476 any following 5477 global/generic 5478 load/load 5479 atomic/store/store 5480 atomic/atomicrmw. 5481 - Ensures any 5482 following global 5483 data read is no 5484 older than the local load 5485 atomic value being 5486 acquired. 5487 5488 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5489 5490 - If OpenCL, omit. 5491 - Must happen after 5492 any preceding 5493 local/generic 5494 load/store/load 5495 atomic/store 5496 atomic/atomicrmw. 5497 - Must happen before 5498 the following 5499 atomicrmw. 5500 - Ensures that all 5501 memory operations 5502 to local have 5503 completed before 5504 performing the 5505 atomicrmw that is 5506 being released. 5507 5508 2. flat_atomic 5509 3. s_waitcnt lgkmcnt(0) 5510 5511 - If OpenCL, omit. 5512 - Must happen before 5513 any following 5514 global/generic 5515 load/load 5516 atomic/store/store 5517 atomic/atomicrmw. 5518 - Ensures any 5519 following global 5520 data read is no 5521 older than a local load 5522 atomic value being 5523 acquired. 5524 5525 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5526 - system vmcnt(0) 5527 5528 - If OpenCL, omit 5529 lgkmcnt(0). 5530 - Could be split into 5531 separate s_waitcnt 5532 vmcnt(0) and 5533 s_waitcnt 5534 lgkmcnt(0) to allow 5535 them to be 5536 independently moved 5537 according to the 5538 following rules. 5539 - s_waitcnt vmcnt(0) 5540 must happen after 5541 any preceding 5542 global/generic 5543 load/store/load 5544 atomic/store 5545 atomic/atomicrmw. 5546 - s_waitcnt lgkmcnt(0) 5547 must happen after 5548 any preceding 5549 local/generic 5550 load/store/load 5551 atomic/store 5552 atomic/atomicrmw. 5553 - Must happen before 5554 the following 5555 atomicrmw. 5556 - Ensures that all 5557 memory operations 5558 to global have 5559 completed before 5560 performing the 5561 atomicrmw that is 5562 being released. 5563 5564 2. buffer/global_atomic 5565 3. s_waitcnt vmcnt(0) 5566 5567 - Must happen before 5568 following 5569 buffer_wbinvl1_vol. 5570 - Ensures the 5571 atomicrmw has 5572 completed before 5573 invalidating the 5574 cache. 5575 5576 4. buffer_wbinvl1_vol 5577 5578 - Must happen before 5579 any following 5580 global/generic 5581 load/load 5582 atomic/atomicrmw. 5583 - Ensures that 5584 following loads 5585 will not see stale 5586 global data. 5587 5588 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5589 - system vmcnt(0) 5590 5591 - If OpenCL, omit 5592 lgkmcnt(0). 5593 - Could be split into 5594 separate s_waitcnt 5595 vmcnt(0) and 5596 s_waitcnt 5597 lgkmcnt(0) to allow 5598 them to be 5599 independently moved 5600 according to the 5601 following rules. 5602 - s_waitcnt vmcnt(0) 5603 must happen after 5604 any preceding 5605 global/generic 5606 load/store/load 5607 atomic/store 5608 atomic/atomicrmw. 5609 - s_waitcnt lgkmcnt(0) 5610 must happen after 5611 any preceding 5612 local/generic 5613 load/store/load 5614 atomic/store 5615 atomic/atomicrmw. 5616 - Must happen before 5617 the following 5618 atomicrmw. 5619 - Ensures that all 5620 memory operations 5621 to global have 5622 completed before 5623 performing the 5624 atomicrmw that is 5625 being released. 5626 5627 2. flat_atomic 5628 3. s_waitcnt vmcnt(0) & 5629 lgkmcnt(0) 5630 5631 - If OpenCL, omit 5632 lgkmcnt(0). 5633 - Must happen before 5634 following 5635 buffer_wbinvl1_vol. 5636 - Ensures the 5637 atomicrmw has 5638 completed before 5639 invalidating the 5640 cache. 5641 5642 4. buffer_wbinvl1_vol 5643 5644 - Must happen before 5645 any following 5646 global/generic 5647 load/load 5648 atomic/atomicrmw. 5649 - Ensures that 5650 following loads 5651 will not see stale 5652 global data. 5653 5654 fence acq_rel - singlethread *none* *none* 5655 - wavefront 5656 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5657 5658 - If OpenCL and 5659 address space is 5660 not generic, omit. 5661 - However, 5662 since LLVM 5663 currently has no 5664 address space on 5665 the fence need to 5666 conservatively 5667 always generate 5668 (see comment for 5669 previous fence). 5670 - Must happen after 5671 any preceding 5672 local/generic 5673 load/load 5674 atomic/store/store 5675 atomic/atomicrmw. 5676 - Must happen before 5677 any following 5678 global/generic 5679 load/load 5680 atomic/store/store 5681 atomic/atomicrmw. 5682 - Ensures that all 5683 memory operations 5684 to local have 5685 completed before 5686 performing any 5687 following global 5688 memory operations. 5689 - Ensures that the 5690 preceding 5691 local/generic load 5692 atomic/atomicrmw 5693 with an equal or 5694 wider sync scope 5695 and memory ordering 5696 stronger than 5697 unordered (this is 5698 termed the 5699 acquire-fence-paired-atomic) 5700 has completed 5701 before following 5702 global memory 5703 operations. This 5704 satisfies the 5705 requirements of 5706 acquire. 5707 - Ensures that all 5708 previous memory 5709 operations have 5710 completed before a 5711 following 5712 local/generic store 5713 atomic/atomicrmw 5714 with an equal or 5715 wider sync scope 5716 and memory ordering 5717 stronger than 5718 unordered (this is 5719 termed the 5720 release-fence-paired-atomic). 5721 This satisfies the 5722 requirements of 5723 release. 5724 5725 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5726 - system vmcnt(0) 5727 5728 - If OpenCL and 5729 address space is 5730 not generic, omit 5731 lgkmcnt(0). 5732 - However, since LLVM 5733 currently has no 5734 address space on 5735 the fence need to 5736 conservatively 5737 always generate 5738 (see comment for 5739 previous fence). 5740 - Could be split into 5741 separate s_waitcnt 5742 vmcnt(0) and 5743 s_waitcnt 5744 lgkmcnt(0) to allow 5745 them to be 5746 independently moved 5747 according to the 5748 following rules. 5749 - s_waitcnt vmcnt(0) 5750 must happen after 5751 any preceding 5752 global/generic 5753 load/store/load 5754 atomic/store 5755 atomic/atomicrmw. 5756 - s_waitcnt lgkmcnt(0) 5757 must happen after 5758 any preceding 5759 local/generic 5760 load/store/load 5761 atomic/store 5762 atomic/atomicrmw. 5763 - Must happen before 5764 the following 5765 buffer_wbinvl1_vol. 5766 - Ensures that the 5767 preceding 5768 global/local/generic 5769 load 5770 atomic/atomicrmw 5771 with an equal or 5772 wider sync scope 5773 and memory ordering 5774 stronger than 5775 unordered (this is 5776 termed the 5777 acquire-fence-paired-atomic) 5778 has completed 5779 before invalidating 5780 the cache. This 5781 satisfies the 5782 requirements of 5783 acquire. 5784 - Ensures that all 5785 previous memory 5786 operations have 5787 completed before a 5788 following 5789 global/local/generic 5790 store 5791 atomic/atomicrmw 5792 with an equal or 5793 wider sync scope 5794 and memory ordering 5795 stronger than 5796 unordered (this is 5797 termed the 5798 release-fence-paired-atomic). 5799 This satisfies the 5800 requirements of 5801 release. 5802 5803 2. buffer_wbinvl1_vol 5804 5805 - Must happen before 5806 any following 5807 global/generic 5808 load/load 5809 atomic/store/store 5810 atomic/atomicrmw. 5811 - Ensures that 5812 following loads 5813 will not see stale 5814 global data. This 5815 satisfies the 5816 requirements of 5817 acquire. 5818 5819 **Sequential Consistent Atomic** 5820 ------------------------------------------------------------------------------------ 5821 load atomic seq_cst - singlethread - global *Same as corresponding 5822 - wavefront - local load atomic acquire, 5823 - generic except must generated 5824 all instructions even 5825 for OpenCL.* 5826 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5827 - generic 5828 5829 - Must 5830 happen after 5831 preceding 5832 local/generic load 5833 atomic/store 5834 atomic/atomicrmw 5835 with memory 5836 ordering of seq_cst 5837 and with equal or 5838 wider sync scope. 5839 (Note that seq_cst 5840 fences have their 5841 own s_waitcnt 5842 lgkmcnt(0) and so do 5843 not need to be 5844 considered.) 5845 - Ensures any 5846 preceding 5847 sequential 5848 consistent local 5849 memory instructions 5850 have completed 5851 before executing 5852 this sequentially 5853 consistent 5854 instruction. This 5855 prevents reordering 5856 a seq_cst store 5857 followed by a 5858 seq_cst load. (Note 5859 that seq_cst is 5860 stronger than 5861 acquire/release as 5862 the reordering of 5863 load acquire 5864 followed by a store 5865 release is 5866 prevented by the 5867 s_waitcnt of 5868 the release, but 5869 there is nothing 5870 preventing a store 5871 release followed by 5872 load acquire from 5873 completing out of 5874 order. The s_waitcnt 5875 could be placed after 5876 seq_store or before 5877 the seq_load. We 5878 choose the load to 5879 make the s_waitcnt be 5880 as late as possible 5881 so that the store 5882 may have already 5883 completed.) 5884 5885 2. *Following 5886 instructions same as 5887 corresponding load 5888 atomic acquire, 5889 except must generated 5890 all instructions even 5891 for OpenCL.* 5892 load atomic seq_cst - workgroup - local *Same as corresponding 5893 load atomic acquire, 5894 except must generated 5895 all instructions even 5896 for OpenCL.* 5897 5898 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5899 - system - generic vmcnt(0) 5900 5901 - Could be split into 5902 separate s_waitcnt 5903 vmcnt(0) 5904 and s_waitcnt 5905 lgkmcnt(0) to allow 5906 them to be 5907 independently moved 5908 according to the 5909 following rules. 5910 - s_waitcnt lgkmcnt(0) 5911 must happen after 5912 preceding 5913 global/generic load 5914 atomic/store 5915 atomic/atomicrmw 5916 with memory 5917 ordering of seq_cst 5918 and with equal or 5919 wider sync scope. 5920 (Note that seq_cst 5921 fences have their 5922 own s_waitcnt 5923 lgkmcnt(0) and so do 5924 not need to be 5925 considered.) 5926 - s_waitcnt vmcnt(0) 5927 must happen after 5928 preceding 5929 global/generic load 5930 atomic/store 5931 atomic/atomicrmw 5932 with memory 5933 ordering of seq_cst 5934 and with equal or 5935 wider sync scope. 5936 (Note that seq_cst 5937 fences have their 5938 own s_waitcnt 5939 vmcnt(0) and so do 5940 not need to be 5941 considered.) 5942 - Ensures any 5943 preceding 5944 sequential 5945 consistent global 5946 memory instructions 5947 have completed 5948 before executing 5949 this sequentially 5950 consistent 5951 instruction. This 5952 prevents reordering 5953 a seq_cst store 5954 followed by a 5955 seq_cst load. (Note 5956 that seq_cst is 5957 stronger than 5958 acquire/release as 5959 the reordering of 5960 load acquire 5961 followed by a store 5962 release is 5963 prevented by the 5964 s_waitcnt of 5965 the release, but 5966 there is nothing 5967 preventing a store 5968 release followed by 5969 load acquire from 5970 completing out of 5971 order. The s_waitcnt 5972 could be placed after 5973 seq_store or before 5974 the seq_load. We 5975 choose the load to 5976 make the s_waitcnt be 5977 as late as possible 5978 so that the store 5979 may have already 5980 completed.) 5981 5982 2. *Following 5983 instructions same as 5984 corresponding load 5985 atomic acquire, 5986 except must generated 5987 all instructions even 5988 for OpenCL.* 5989 store atomic seq_cst - singlethread - global *Same as corresponding 5990 - wavefront - local store atomic release, 5991 - workgroup - generic except must generated 5992 - agent all instructions even 5993 - system for OpenCL.* 5994 atomicrmw seq_cst - singlethread - global *Same as corresponding 5995 - wavefront - local atomicrmw acq_rel, 5996 - workgroup - generic except must generated 5997 - agent all instructions even 5998 - system for OpenCL.* 5999 fence seq_cst - singlethread *none* *Same as corresponding 6000 - wavefront fence acq_rel, 6001 - workgroup except must generated 6002 - agent all instructions even 6003 - system for OpenCL.* 6004 ============ ============ ============== ========== ================================ 6005 6006.. _amdgpu-amdhsa-memory-model-gfx90a: 6007 6008Memory Model GFX90A 6009+++++++++++++++++++ 6010 6011For GFX90A: 6012 6013* Each agent has multiple shader arrays (SA). 6014* Each SA has multiple compute units (CU). 6015* Each CU has multiple SIMDs that execute wavefronts. 6016* The wavefronts for a single work-group are executed in the same CU but may be 6017 executed by different SIMDs. The exception is when in tgsplit execution mode 6018 when the wavefronts may be executed by different SIMDs in different CUs. 6019* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6020 executing on it. The exception is when in tgsplit execution mode when no LDS 6021 is allocated as wavefronts of the same work-group can be in different CUs. 6022* All LDS operations of a CU are performed as wavefront wide operations in a 6023 global order and involve no caching. Completion is reported to a wavefront in 6024 execution order. 6025* The LDS memory has multiple request queues shared by the SIMDs of a 6026 CU. Therefore, the LDS operations performed by different wavefronts of a 6027 work-group can be reordered relative to each other, which can result in 6028 reordering the visibility of vector memory operations with respect to LDS 6029 operations of other wavefronts in the same work-group. A ``s_waitcnt 6030 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6031 vector memory operations between wavefronts of a work-group, but not between 6032 operations performed by the same wavefront. 6033* The vector memory operations are performed as wavefront wide operations and 6034 completion is reported to a wavefront in execution order. The exception is 6035 that ``flat_load/store/atomic`` instructions can report out of vector memory 6036 order if they access LDS memory, and out of LDS operation order if they access 6037 global memory. 6038* The vector memory operations access a single vector L1 cache shared by all 6039 SIMDs a CU. Therefore: 6040 6041 * No special action is required for coherence between the lanes of a single 6042 wavefront. 6043 6044 * No special action is required for coherence between wavefronts in the same 6045 work-group since they exeute on the same CU. The exception is when in 6046 tgsplit execution mode as wavefronts of the same work-group can be in 6047 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6048 the following item. 6049 6050 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6051 executing in different work-groups as they may be executing on different 6052 CUs. 6053 6054* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6055 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6056 scalar operations are used in a restricted way so do not impact the memory 6057 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6058* The vector and scalar memory operations use an L2 cache shared by all CUs on 6059 the same agent. 6060 6061 * The L2 cache has independent channels to service disjoint ranges of virtual 6062 addresses. 6063 * Each CU has a separate request queue per channel. Therefore, the vector and 6064 scalar memory operations performed by wavefronts executing in different 6065 work-groups (which may be executing on different CUs), or the same 6066 work-group if executing in tgsplit mode, of an agent can be reordered 6067 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6068 synchronization between vector memory operations of different CUs. It 6069 ensures a previous vector memory operation has completed before executing a 6070 subsequent vector memory or LDS operation and so can be used to meet the 6071 requirements of acquire and release. 6072 * The L2 cache of one agent can be kept coherent with other agents by using 6073 the MTYPE RW (read-write) for memory local to the L2, and MTYPE NC 6074 (non-coherent) with the PTE C-bit set for memory not local to the L2. 6075 6076 * Any local memory cache lines will be automatically invalidated by writes 6077 from CUs associated with other L2 caches, or writes from the CPU, due to 6078 the cache probe caused by the PTE C-bit. 6079 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6080 Subsequent access from the GPU will automatically invalidate or writeback 6081 the CPU cache due to the L2 probe filter. 6082 * Since all work-groups on the same agent share the same L2, no L2 6083 invalidation or writeback is required for coherence. 6084 * To ensure coherence of local memory writes of work-groups in different 6085 agents a ``buffer_wbl2`` is required. It will writeback dirty L2 cache 6086 lines. 6087 * To ensure coherence of local memory reads of work-groups in different 6088 agents a ``buffer_invl2`` is required. It will invalidate non-local L2 6089 cache lines. 6090 6091 * PCIe access from the GPU to the CPU memory can be kept coherent by using the 6092 MTYPE UC (uncached) which bypasses the L2. 6093 6094Scalar memory operations are only used to access memory that is proven to not 6095change during the execution of the kernel dispatch. This includes constant 6096address space and global address space for program scope ``const`` variables. 6097Therefore, the kernel machine code does not have to maintain the scalar cache to 6098ensure it is coherent with the vector caches. The scalar and vector caches are 6099invalidated between kernel dispatches by CP since constant address space data 6100may change between kernel dispatch executions. See 6101:ref:`amdgpu-amdhsa-memory-spaces`. 6102 6103The one exception is if scalar writes are used to spill SGPR registers. In this 6104case the AMDGPU backend ensures the memory location used to spill is never 6105accessed by vector memory operations at the same time. If scalar writes are used 6106then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6107return since the locations may be used for vector memory instructions by a 6108future wavefront that uses the same scratch area, or a function call that 6109creates a frame at the same address, respectively. There is no need for a 6110``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6111 6112For kernarg backing memory: 6113 6114* CP invalidates the L1 cache at the start of each kernel dispatch. 6115* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6116 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6117 cache. This also causes it to be treated as non-volatile and so is not 6118 invalidated by ``*_vol``. 6119* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6120 so the L2 cache will be coherent with the CPU and other agents. 6121 6122Scratch backing memory (which is used for the private address space) is accessed 6123with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6124only accessed by a single thread, and is always write-before-read, there is 6125never a need to invalidate these entries from the L1 cache. Hence all cache 6126invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6127 6128The code sequences used to implement the memory model for GFX90A are defined 6129in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6130 6131 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6132 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6133 6134 ============ ============ ============== ========== ================================ 6135 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6136 Ordering Sync Scope Address GFX90A 6137 Space 6138 ============ ============ ============== ========== ================================ 6139 **Non-Atomic** 6140 ------------------------------------------------------------------------------------ 6141 load *none* *none* - global - !volatile & !nontemporal 6142 - generic 6143 - private 1. buffer/global/flat_load 6144 - constant 6145 - !volatile & nontemporal 6146 6147 1. buffer/global/flat_load 6148 glc=1 slc=1 6149 6150 - volatile 6151 6152 1. buffer/global/flat_load 6153 glc=1 scc=1 6154 2. s_waitcnt vmcnt(0) 6155 6156 - Must happen before 6157 any following volatile 6158 global/generic 6159 load/store. 6160 - Ensures that 6161 volatile 6162 operations to 6163 different 6164 addresses will not 6165 be reordered by 6166 hardware. 6167 6168 load *none* *none* - local 1. ds_load 6169 store *none* *none* - global - !volatile & !nontemporal 6170 - generic 6171 - private 1. buffer/global/flat_store 6172 - constant 6173 - !volatile & nontemporal 6174 6175 1. buffer/global/flat_store 6176 glc=1 slc=1 6177 6178 - volatile 6179 6180 1. buffer/global/flat_store 6181 scc=1 6182 2. s_waitcnt vmcnt(0) 6183 6184 - Must happen before 6185 any following volatile 6186 global/generic 6187 load/store. 6188 - Ensures that 6189 volatile 6190 operations to 6191 different 6192 addresses will not 6193 be reordered by 6194 hardware. 6195 6196 store *none* *none* - local 1. ds_store 6197 **Unordered Atomic** 6198 ------------------------------------------------------------------------------------ 6199 load atomic unordered *any* *any* *Same as non-atomic*. 6200 store atomic unordered *any* *any* *Same as non-atomic*. 6201 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6202 **Monotonic Atomic** 6203 ------------------------------------------------------------------------------------ 6204 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6205 - wavefront - generic 6206 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6207 - generic glc=1 6208 6209 - If not TgSplit execution 6210 mode, omit glc=1. 6211 6212 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6213 - wavefront local address space cannot 6214 - workgroup be used.* 6215 6216 1. ds_load 6217 load atomic monotonic - agent - global 1. buffer/global/flat_load 6218 - generic glc=1 6219 load atomic monotonic - system - global 1. buffer/global/flat_load 6220 - generic glc=1 scc=1 6221 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6222 - wavefront - generic 6223 - workgroup 6224 - agent 6225 store atomic monotonic - system - global 1. buffer/global/flat_store 6226 - generic scc=1 6227 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6228 - wavefront local address space cannot 6229 - workgroup be used.* 6230 6231 1. ds_store 6232 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6233 - wavefront - generic 6234 - workgroup 6235 - agent 6236 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6237 - generic scc=1 6238 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6239 - wavefront local address space cannot 6240 - workgroup be used.* 6241 6242 1. ds_atomic 6243 **Acquire Atomic** 6244 ------------------------------------------------------------------------------------ 6245 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6246 - wavefront - local 6247 - generic 6248 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6249 6250 - If not TgSplit execution 6251 mode, omit glc=1. 6252 6253 2. s_waitcnt vmcnt(0) 6254 6255 - If not TgSplit execution 6256 mode, omit. 6257 - Must happen before the 6258 following buffer_wbinvl1_vol. 6259 6260 3. buffer_wbinvl1_vol 6261 6262 - If not TgSplit execution 6263 mode, omit. 6264 - Must happen before 6265 any following 6266 global/generic 6267 load/load 6268 atomic/store/store 6269 atomic/atomicrmw. 6270 - Ensures that 6271 following 6272 loads will not see 6273 stale data. 6274 6275 load atomic acquire - workgroup - local *If TgSplit execution mode, 6276 local address space cannot 6277 be used.* 6278 6279 1. ds_load 6280 2. s_waitcnt lgkmcnt(0) 6281 6282 - If OpenCL, omit. 6283 - Must happen before 6284 any following 6285 global/generic 6286 load/load 6287 atomic/store/store 6288 atomic/atomicrmw. 6289 - Ensures any 6290 following global 6291 data read is no 6292 older than the local load 6293 atomic value being 6294 acquired. 6295 6296 load atomic acquire - workgroup - generic 1. flat_load glc=1 6297 6298 - If not TgSplit execution 6299 mode, omit glc=1. 6300 6301 2. s_waitcnt lgkm/vmcnt(0) 6302 6303 - Use lgkmcnt(0) if not 6304 TgSplit execution mode 6305 and vmcnt(0) if TgSplit 6306 execution mode. 6307 - If OpenCL, omit lgkmcnt(0). 6308 - Must happen before 6309 the following 6310 buffer_wbinvl1_vol and any 6311 following global/generic 6312 load/load 6313 atomic/store/store 6314 atomic/atomicrmw. 6315 - Ensures any 6316 following global 6317 data read is no 6318 older than a local load 6319 atomic value being 6320 acquired. 6321 6322 3. buffer_wbinvl1_vol 6323 6324 - If not TgSplit execution 6325 mode, omit. 6326 - Ensures that 6327 following 6328 loads will not see 6329 stale data. 6330 6331 load atomic acquire - agent - global 1. buffer/global_load 6332 glc=1 6333 2. s_waitcnt vmcnt(0) 6334 6335 - Must happen before 6336 following 6337 buffer_wbinvl1_vol. 6338 - Ensures the load 6339 has completed 6340 before invalidating 6341 the cache. 6342 6343 3. buffer_wbinvl1_vol 6344 6345 - Must happen before 6346 any following 6347 global/generic 6348 load/load 6349 atomic/atomicrmw. 6350 - Ensures that 6351 following 6352 loads will not see 6353 stale global data. 6354 6355 load atomic acquire - system - global 1. buffer/global/flat_load 6356 glc=1 scc=1 6357 2. s_waitcnt vmcnt(0) 6358 6359 - Must happen before 6360 following buffer_invl2 and 6361 buffer_wbinvl1_vol. 6362 - Ensures the load 6363 has completed 6364 before invalidating 6365 the cache. 6366 6367 3. buffer_invl2; 6368 buffer_wbinvl1_vol 6369 6370 - Must happen before 6371 any following 6372 global/generic 6373 load/load 6374 atomic/atomicrmw. 6375 - Ensures that 6376 following 6377 loads will not see 6378 stale MTYPE NC global data. 6379 MTYPE RW and CC memory will 6380 never be stale due to the 6381 memory probes. 6382 6383 load atomic acquire - agent - generic 1. flat_load glc=1 6384 2. s_waitcnt vmcnt(0) & 6385 lgkmcnt(0) 6386 6387 - If TgSplit execution mode, 6388 omit lgkmcnt(0). 6389 - If OpenCL omit 6390 lgkmcnt(0). 6391 - Must happen before 6392 following 6393 buffer_wbinvl1_vol. 6394 - Ensures the flat_load 6395 has completed 6396 before invalidating 6397 the cache. 6398 6399 3. buffer_wbinvl1_vol 6400 6401 - Must happen before 6402 any following 6403 global/generic 6404 load/load 6405 atomic/atomicrmw. 6406 - Ensures that 6407 following loads 6408 will not see stale 6409 global data. 6410 6411 load atomic acquire - system - generic 1. flat_load glc=1 scc=1 6412 2. s_waitcnt vmcnt(0) & 6413 lgkmcnt(0) 6414 6415 - If TgSplit execution mode, 6416 omit lgkmcnt(0). 6417 - If OpenCL omit 6418 lgkmcnt(0). 6419 - Must happen before 6420 following 6421 buffer_invl2 and 6422 buffer_wbinvl1_vol. 6423 - Ensures the flat_load 6424 has completed 6425 before invalidating 6426 the caches. 6427 6428 3. buffer_invl2; 6429 buffer_wbinvl1_vol 6430 6431 - Must happen before 6432 any following 6433 global/generic 6434 load/load 6435 atomic/atomicrmw. 6436 - Ensures that 6437 following 6438 loads will not see 6439 stale MTYPE NC global data. 6440 MTYPE RW and CC memory will 6441 never be stale due to the 6442 memory probes. 6443 6444 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6445 - wavefront - generic 6446 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6447 - wavefront local address space cannot 6448 be used.* 6449 6450 1. ds_atomic 6451 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6452 2. s_waitcnt vmcnt(0) 6453 6454 - If not TgSplit execution 6455 mode, omit. 6456 - Must happen before the 6457 following buffer_wbinvl1_vol. 6458 - Ensures the atomicrmw 6459 has completed 6460 before invalidating 6461 the cache. 6462 6463 3. buffer_wbinvl1_vol 6464 6465 - If not TgSplit execution 6466 mode, omit. 6467 - Must happen before 6468 any following 6469 global/generic 6470 load/load 6471 atomic/atomicrmw. 6472 - Ensures that 6473 following loads 6474 will not see stale 6475 global data. 6476 6477 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6478 local address space cannot 6479 be used.* 6480 6481 1. ds_atomic 6482 2. s_waitcnt lgkmcnt(0) 6483 6484 - If OpenCL, omit. 6485 - Must happen before 6486 any following 6487 global/generic 6488 load/load 6489 atomic/store/store 6490 atomic/atomicrmw. 6491 - Ensures any 6492 following global 6493 data read is no 6494 older than the local 6495 atomicrmw value 6496 being acquired. 6497 6498 atomicrmw acquire - workgroup - generic 1. flat_atomic 6499 2. s_waitcnt lgkm/vmcnt(0) 6500 6501 - Use lgkmcnt(0) if not 6502 TgSplit execution mode 6503 and vmcnt(0) if TgSplit 6504 execution mode. 6505 - If OpenCL, omit lgkmcnt(0). 6506 - Must happen before 6507 the following 6508 buffer_wbinvl1_vol and 6509 any following 6510 global/generic 6511 load/load 6512 atomic/store/store 6513 atomic/atomicrmw. 6514 - Ensures any 6515 following global 6516 data read is no 6517 older than a local 6518 atomicrmw value 6519 being acquired. 6520 6521 3. buffer_wbinvl1_vol 6522 6523 - If not TgSplit execution 6524 mode, omit. 6525 - Ensures that 6526 following 6527 loads will not see 6528 stale data. 6529 6530 atomicrmw acquire - agent - global 1. buffer/global_atomic 6531 2. s_waitcnt vmcnt(0) 6532 6533 - Must happen before 6534 following 6535 buffer_wbinvl1_vol. 6536 - Ensures the 6537 atomicrmw has 6538 completed before 6539 invalidating the 6540 cache. 6541 6542 3. buffer_wbinvl1_vol 6543 6544 - Must happen before 6545 any following 6546 global/generic 6547 load/load 6548 atomic/atomicrmw. 6549 - Ensures that 6550 following loads 6551 will not see stale 6552 global data. 6553 6554 atomicrmw acquire - system - global 1. buffer/global_atomic 6555 scc=1 6556 2. s_waitcnt vmcnt(0) 6557 6558 - Must happen before 6559 following buffer_invl2 and 6560 buffer_wbinvl1_vol. 6561 - Ensures the 6562 atomicrmw has 6563 completed before 6564 invalidating the 6565 caches. 6566 6567 3. buffer_invl2; 6568 buffer_wbinvl1_vol 6569 6570 - Must happen before 6571 any following 6572 global/generic 6573 load/load 6574 atomic/atomicrmw. 6575 - Ensures that 6576 following 6577 loads will not see 6578 stale MTYPE NC global data. 6579 MTYPE RW and CC memory will 6580 never be stale due to the 6581 memory probes. 6582 6583 atomicrmw acquire - agent - generic 1. flat_atomic 6584 2. s_waitcnt vmcnt(0) & 6585 lgkmcnt(0) 6586 6587 - If TgSplit execution mode, 6588 omit lgkmcnt(0). 6589 - If OpenCL, omit 6590 lgkmcnt(0). 6591 - Must happen before 6592 following 6593 buffer_wbinvl1_vol. 6594 - Ensures the 6595 atomicrmw has 6596 completed before 6597 invalidating the 6598 cache. 6599 6600 3. buffer_wbinvl1_vol 6601 6602 - Must happen before 6603 any following 6604 global/generic 6605 load/load 6606 atomic/atomicrmw. 6607 - Ensures that 6608 following loads 6609 will not see stale 6610 global data. 6611 6612 atomicrmw acquire - system - generic 1. flat_atomic scc=1 6613 2. s_waitcnt vmcnt(0) & 6614 lgkmcnt(0) 6615 6616 - If TgSplit execution mode, 6617 omit lgkmcnt(0). 6618 - If OpenCL, omit 6619 lgkmcnt(0). 6620 - Must happen before 6621 following 6622 buffer_invl2 and 6623 buffer_wbinvl1_vol. 6624 - Ensures the 6625 atomicrmw has 6626 completed before 6627 invalidating the 6628 caches. 6629 6630 3. buffer_invl2; 6631 buffer_wbinvl1_vol 6632 6633 - Must happen before 6634 any following 6635 global/generic 6636 load/load 6637 atomic/atomicrmw. 6638 - Ensures that 6639 following 6640 loads will not see 6641 stale MTYPE NC global data. 6642 MTYPE RW and CC memory will 6643 never be stale due to the 6644 memory probes. 6645 6646 fence acquire - singlethread *none* *none* 6647 - wavefront 6648 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6649 6650 - Use lgkmcnt(0) if not 6651 TgSplit execution mode 6652 and vmcnt(0) if TgSplit 6653 execution mode. 6654 - If OpenCL and 6655 address space is 6656 not generic, omit 6657 lgkmcnt(0). 6658 - If OpenCL and 6659 address space is 6660 local, omit 6661 vmcnt(0). 6662 - However, since LLVM 6663 currently has no 6664 address space on 6665 the fence need to 6666 conservatively 6667 always generate. If 6668 fence had an 6669 address space then 6670 set to address 6671 space of OpenCL 6672 fence flag, or to 6673 generic if both 6674 local and global 6675 flags are 6676 specified. 6677 - s_waitcnt vmcnt(0) 6678 must happen after 6679 any preceding 6680 global/generic load 6681 atomic/ 6682 atomicrmw 6683 with an equal or 6684 wider sync scope 6685 and memory ordering 6686 stronger than 6687 unordered (this is 6688 termed the 6689 fence-paired-atomic). 6690 - s_waitcnt lgkmcnt(0) 6691 must happen after 6692 any preceding 6693 local/generic load 6694 atomic/atomicrmw 6695 with an equal or 6696 wider sync scope 6697 and memory ordering 6698 stronger than 6699 unordered (this is 6700 termed the 6701 fence-paired-atomic). 6702 - Must happen before 6703 the following 6704 buffer_wbinvl1_vol and 6705 any following 6706 global/generic 6707 load/load 6708 atomic/store/store 6709 atomic/atomicrmw. 6710 - Ensures any 6711 following global 6712 data read is no 6713 older than the 6714 value read by the 6715 fence-paired-atomic. 6716 6717 3. buffer_wbinvl1_vol 6718 6719 - If not TgSplit execution 6720 mode, omit. 6721 - Ensures that 6722 following 6723 loads will not see 6724 stale data. 6725 6726 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6727 vmcnt(0) 6728 6729 - If TgSplit execution mode, 6730 omit lgkmcnt(0). 6731 - If OpenCL and 6732 address space is 6733 not generic, omit 6734 lgkmcnt(0). 6735 - However, since LLVM 6736 currently has no 6737 address space on 6738 the fence need to 6739 conservatively 6740 always generate 6741 (see comment for 6742 previous fence). 6743 - Could be split into 6744 separate s_waitcnt 6745 vmcnt(0) and 6746 s_waitcnt 6747 lgkmcnt(0) to allow 6748 them to be 6749 independently moved 6750 according to the 6751 following rules. 6752 - s_waitcnt vmcnt(0) 6753 must happen after 6754 any preceding 6755 global/generic load 6756 atomic/atomicrmw 6757 with an equal or 6758 wider sync scope 6759 and memory ordering 6760 stronger than 6761 unordered (this is 6762 termed the 6763 fence-paired-atomic). 6764 - s_waitcnt lgkmcnt(0) 6765 must happen after 6766 any preceding 6767 local/generic load 6768 atomic/atomicrmw 6769 with an equal or 6770 wider sync scope 6771 and memory ordering 6772 stronger than 6773 unordered (this is 6774 termed the 6775 fence-paired-atomic). 6776 - Must happen before 6777 the following 6778 buffer_wbinvl1_vol. 6779 - Ensures that the 6780 fence-paired atomic 6781 has completed 6782 before invalidating 6783 the 6784 cache. Therefore 6785 any following 6786 locations read must 6787 be no older than 6788 the value read by 6789 the 6790 fence-paired-atomic. 6791 6792 2. buffer_wbinvl1_vol 6793 6794 - Must happen before any 6795 following global/generic 6796 load/load 6797 atomic/store/store 6798 atomic/atomicrmw. 6799 - Ensures that 6800 following loads 6801 will not see stale 6802 global data. 6803 6804 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6805 vmcnt(0) 6806 6807 - If TgSplit execution mode, 6808 omit lgkmcnt(0). 6809 - If OpenCL and 6810 address space is 6811 not generic, omit 6812 lgkmcnt(0). 6813 - However, since LLVM 6814 currently has no 6815 address space on 6816 the fence need to 6817 conservatively 6818 always generate 6819 (see comment for 6820 previous fence). 6821 - Could be split into 6822 separate s_waitcnt 6823 vmcnt(0) and 6824 s_waitcnt 6825 lgkmcnt(0) to allow 6826 them to be 6827 independently moved 6828 according to the 6829 following rules. 6830 - s_waitcnt vmcnt(0) 6831 must happen after 6832 any preceding 6833 global/generic load 6834 atomic/atomicrmw 6835 with an equal or 6836 wider sync scope 6837 and memory ordering 6838 stronger than 6839 unordered (this is 6840 termed the 6841 fence-paired-atomic). 6842 - s_waitcnt lgkmcnt(0) 6843 must happen after 6844 any preceding 6845 local/generic load 6846 atomic/atomicrmw 6847 with an equal or 6848 wider sync scope 6849 and memory ordering 6850 stronger than 6851 unordered (this is 6852 termed the 6853 fence-paired-atomic). 6854 - Must happen before 6855 the following buffer_invl2 and 6856 buffer_wbinvl1_vol. 6857 - Ensures that the 6858 fence-paired atomic 6859 has completed 6860 before invalidating 6861 the 6862 cache. Therefore 6863 any following 6864 locations read must 6865 be no older than 6866 the value read by 6867 the 6868 fence-paired-atomic. 6869 6870 2. buffer_invl2; 6871 buffer_wbinvl1_vol 6872 6873 - Must happen before any 6874 following global/generic 6875 load/load 6876 atomic/store/store 6877 atomic/atomicrmw. 6878 - Ensures that 6879 following loads 6880 will not see stale 6881 global data. 6882 6883 **Release Atomic** 6884 ------------------------------------------------------------------------------------ 6885 store atomic release - singlethread - global 1. buffer/global/flat_store 6886 - wavefront - generic 6887 store atomic release - singlethread - local *If TgSplit execution mode, 6888 - wavefront local address space cannot 6889 be used.* 6890 6891 1. ds_store 6892 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6893 - generic 6894 - Use lgkmcnt(0) if not 6895 TgSplit execution mode 6896 and vmcnt(0) if TgSplit 6897 execution mode. 6898 - If OpenCL, omit lgkmcnt(0). 6899 - s_waitcnt vmcnt(0) 6900 must happen after 6901 any preceding 6902 global/generic load/store/ 6903 load atomic/store atomic/ 6904 atomicrmw. 6905 - s_waitcnt lgkmcnt(0) 6906 must happen after 6907 any preceding 6908 local/generic 6909 load/store/load 6910 atomic/store 6911 atomic/atomicrmw. 6912 - Must happen before 6913 the following 6914 store. 6915 - Ensures that all 6916 memory operations 6917 have 6918 completed before 6919 performing the 6920 store that is being 6921 released. 6922 6923 2. buffer/global/flat_store 6924 store atomic release - workgroup - local *If TgSplit execution mode, 6925 local address space cannot 6926 be used.* 6927 6928 1. ds_store 6929 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6930 - generic vmcnt(0) 6931 6932 - If TgSplit execution mode, 6933 omit lgkmcnt(0). 6934 - If OpenCL and 6935 address space is 6936 not generic, omit 6937 lgkmcnt(0). 6938 - Could be split into 6939 separate s_waitcnt 6940 vmcnt(0) and 6941 s_waitcnt 6942 lgkmcnt(0) to allow 6943 them to be 6944 independently moved 6945 according to the 6946 following rules. 6947 - s_waitcnt vmcnt(0) 6948 must happen after 6949 any preceding 6950 global/generic 6951 load/store/load 6952 atomic/store 6953 atomic/atomicrmw. 6954 - s_waitcnt lgkmcnt(0) 6955 must happen after 6956 any preceding 6957 local/generic 6958 load/store/load 6959 atomic/store 6960 atomic/atomicrmw. 6961 - Must happen before 6962 the following 6963 store. 6964 - Ensures that all 6965 memory operations 6966 to memory have 6967 completed before 6968 performing the 6969 store that is being 6970 released. 6971 6972 2. buffer/global/flat_store 6973 store atomic release - system - global 1. buffer_wbl2 6974 - generic 6975 - Must happen before 6976 following s_waitcnt. 6977 - Performs L2 writeback to 6978 ensure previous 6979 global/generic 6980 store/atomicrmw are 6981 visible at system scope. 6982 6983 2. s_waitcnt lgkmcnt(0) & 6984 vmcnt(0) 6985 6986 - If TgSplit execution mode, 6987 omit lgkmcnt(0). 6988 - If OpenCL and 6989 address space is 6990 not generic, omit 6991 lgkmcnt(0). 6992 - Could be split into 6993 separate s_waitcnt 6994 vmcnt(0) and 6995 s_waitcnt 6996 lgkmcnt(0) to allow 6997 them to be 6998 independently moved 6999 according to the 7000 following rules. 7001 - s_waitcnt vmcnt(0) 7002 must happen after any 7003 preceding 7004 global/generic 7005 load/store/load 7006 atomic/store 7007 atomic/atomicrmw. 7008 - s_waitcnt lgkmcnt(0) 7009 must happen after any 7010 preceding 7011 local/generic 7012 load/store/load 7013 atomic/store 7014 atomic/atomicrmw. 7015 - Must happen before 7016 the following 7017 store. 7018 - Ensures that all 7019 memory operations 7020 to memory and the L2 7021 writeback have 7022 completed before 7023 performing the 7024 store that is being 7025 released. 7026 7027 2. buffer/global/flat_store 7028 scc=1 7029 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7030 - wavefront - generic 7031 atomicrmw release - singlethread - local *If TgSplit execution mode, 7032 - wavefront local address space cannot 7033 be used.* 7034 7035 1. ds_atomic 7036 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7037 - generic 7038 - Use lgkmcnt(0) if not 7039 TgSplit execution mode 7040 and vmcnt(0) if TgSplit 7041 execution mode. 7042 - If OpenCL, omit 7043 lgkmcnt(0). 7044 - s_waitcnt vmcnt(0) 7045 must happen after 7046 any preceding 7047 global/generic load/store/ 7048 load atomic/store atomic/ 7049 atomicrmw. 7050 - s_waitcnt lgkmcnt(0) 7051 must happen after 7052 any preceding 7053 local/generic 7054 load/store/load 7055 atomic/store 7056 atomic/atomicrmw. 7057 - Must happen before 7058 the following 7059 atomicrmw. 7060 - Ensures that all 7061 memory operations 7062 have 7063 completed before 7064 performing the 7065 atomicrmw that is 7066 being released. 7067 7068 2. buffer/global/flat_atomic 7069 atomicrmw release - workgroup - local *If TgSplit execution mode, 7070 local address space cannot 7071 be used.* 7072 7073 1. ds_atomic 7074 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7075 - generic vmcnt(0) 7076 7077 - If TgSplit execution mode, 7078 omit lgkmcnt(0). 7079 - If OpenCL, omit 7080 lgkmcnt(0). 7081 - Could be split into 7082 separate s_waitcnt 7083 vmcnt(0) and 7084 s_waitcnt 7085 lgkmcnt(0) to allow 7086 them to be 7087 independently moved 7088 according to the 7089 following rules. 7090 - s_waitcnt vmcnt(0) 7091 must happen after 7092 any preceding 7093 global/generic 7094 load/store/load 7095 atomic/store 7096 atomic/atomicrmw. 7097 - s_waitcnt lgkmcnt(0) 7098 must happen after 7099 any preceding 7100 local/generic 7101 load/store/load 7102 atomic/store 7103 atomic/atomicrmw. 7104 - Must happen before 7105 the following 7106 atomicrmw. 7107 - Ensures that all 7108 memory operations 7109 to global and local 7110 have completed 7111 before performing 7112 the atomicrmw that 7113 is being released. 7114 7115 2. buffer/global/flat_atomic 7116 atomicrmw release - system - global 1. buffer_wbl2 7117 - generic 7118 - Must happen before 7119 following s_waitcnt. 7120 - Performs L2 writeback to 7121 ensure previous 7122 global/generic 7123 store/atomicrmw are 7124 visible at system scope. 7125 7126 2. s_waitcnt lgkmcnt(0) & 7127 vmcnt(0) 7128 7129 - If TgSplit execution mode, 7130 omit lgkmcnt(0). 7131 - If OpenCL, omit 7132 lgkmcnt(0). 7133 - Could be split into 7134 separate s_waitcnt 7135 vmcnt(0) and 7136 s_waitcnt 7137 lgkmcnt(0) to allow 7138 them to be 7139 independently moved 7140 according to the 7141 following rules. 7142 - s_waitcnt vmcnt(0) 7143 must happen after 7144 any preceding 7145 global/generic 7146 load/store/load 7147 atomic/store 7148 atomic/atomicrmw. 7149 - s_waitcnt lgkmcnt(0) 7150 must happen after 7151 any preceding 7152 local/generic 7153 load/store/load 7154 atomic/store 7155 atomic/atomicrmw. 7156 - Must happen before 7157 the following 7158 atomicrmw. 7159 - Ensures that all 7160 memory operations 7161 to memory and the L2 7162 writeback have 7163 completed before 7164 performing the 7165 store that is being 7166 released. 7167 7168 3. buffer/global/flat_atomic 7169 scc=1 7170 fence release - singlethread *none* *none* 7171 - wavefront 7172 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7173 7174 - Use lgkmcnt(0) if not 7175 TgSplit execution mode 7176 and vmcnt(0) if TgSplit 7177 execution mode. 7178 - If OpenCL and 7179 address space is 7180 not generic, omit 7181 lgkmcnt(0). 7182 - If OpenCL and 7183 address space is 7184 local, omit 7185 vmcnt(0). 7186 - However, since LLVM 7187 currently has no 7188 address space on 7189 the fence need to 7190 conservatively 7191 always generate. If 7192 fence had an 7193 address space then 7194 set to address 7195 space of OpenCL 7196 fence flag, or to 7197 generic if both 7198 local and global 7199 flags are 7200 specified. 7201 - s_waitcnt vmcnt(0) 7202 must happen after 7203 any preceding 7204 global/generic 7205 load/store/ 7206 load atomic/store atomic/ 7207 atomicrmw. 7208 - s_waitcnt lgkmcnt(0) 7209 must happen after 7210 any preceding 7211 local/generic 7212 load/load 7213 atomic/store/store 7214 atomic/atomicrmw. 7215 - Must happen before 7216 any following store 7217 atomic/atomicrmw 7218 with an equal or 7219 wider sync scope 7220 and memory ordering 7221 stronger than 7222 unordered (this is 7223 termed the 7224 fence-paired-atomic). 7225 - Ensures that all 7226 memory operations 7227 have 7228 completed before 7229 performing the 7230 following 7231 fence-paired-atomic. 7232 7233 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7234 vmcnt(0) 7235 7236 - If TgSplit execution mode, 7237 omit lgkmcnt(0). 7238 - If OpenCL and 7239 address space is 7240 not generic, omit 7241 lgkmcnt(0). 7242 - If OpenCL and 7243 address space is 7244 local, omit 7245 vmcnt(0). 7246 - However, since LLVM 7247 currently has no 7248 address space on 7249 the fence need to 7250 conservatively 7251 always generate. If 7252 fence had an 7253 address space then 7254 set to address 7255 space of OpenCL 7256 fence flag, or to 7257 generic if both 7258 local and global 7259 flags are 7260 specified. 7261 - Could be split into 7262 separate s_waitcnt 7263 vmcnt(0) and 7264 s_waitcnt 7265 lgkmcnt(0) to allow 7266 them to be 7267 independently moved 7268 according to the 7269 following rules. 7270 - s_waitcnt vmcnt(0) 7271 must happen after 7272 any preceding 7273 global/generic 7274 load/store/load 7275 atomic/store 7276 atomic/atomicrmw. 7277 - s_waitcnt lgkmcnt(0) 7278 must happen after 7279 any preceding 7280 local/generic 7281 load/store/load 7282 atomic/store 7283 atomic/atomicrmw. 7284 - Must happen before 7285 any following store 7286 atomic/atomicrmw 7287 with an equal or 7288 wider sync scope 7289 and memory ordering 7290 stronger than 7291 unordered (this is 7292 termed the 7293 fence-paired-atomic). 7294 - Ensures that all 7295 memory operations 7296 have 7297 completed before 7298 performing the 7299 following 7300 fence-paired-atomic. 7301 7302 fence release - system *none* 1. buffer_wbl2 7303 7304 - If OpenCL and 7305 address space is 7306 local, omit. 7307 - Must happen before 7308 following s_waitcnt. 7309 - Performs L2 writeback to 7310 ensure previous 7311 global/generic 7312 store/atomicrmw are 7313 visible at system scope. 7314 7315 2. s_waitcnt lgkmcnt(0) & 7316 vmcnt(0) 7317 7318 - If TgSplit execution mode, 7319 omit lgkmcnt(0). 7320 - If OpenCL and 7321 address space is 7322 not generic, omit 7323 lgkmcnt(0). 7324 - If OpenCL and 7325 address space is 7326 local, omit 7327 vmcnt(0). 7328 - However, since LLVM 7329 currently has no 7330 address space on 7331 the fence need to 7332 conservatively 7333 always generate. If 7334 fence had an 7335 address space then 7336 set to address 7337 space of OpenCL 7338 fence flag, or to 7339 generic if both 7340 local and global 7341 flags are 7342 specified. 7343 - Could be split into 7344 separate s_waitcnt 7345 vmcnt(0) and 7346 s_waitcnt 7347 lgkmcnt(0) to allow 7348 them to be 7349 independently moved 7350 according to the 7351 following rules. 7352 - s_waitcnt vmcnt(0) 7353 must happen after 7354 any preceding 7355 global/generic 7356 load/store/load 7357 atomic/store 7358 atomic/atomicrmw. 7359 - s_waitcnt lgkmcnt(0) 7360 must happen after 7361 any preceding 7362 local/generic 7363 load/store/load 7364 atomic/store 7365 atomic/atomicrmw. 7366 - Must happen before 7367 any following store 7368 atomic/atomicrmw 7369 with an equal or 7370 wider sync scope 7371 and memory ordering 7372 stronger than 7373 unordered (this is 7374 termed the 7375 fence-paired-atomic). 7376 - Ensures that all 7377 memory operations 7378 have 7379 completed before 7380 performing the 7381 following 7382 fence-paired-atomic. 7383 7384 **Acquire-Release Atomic** 7385 ------------------------------------------------------------------------------------ 7386 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7387 - wavefront - generic 7388 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7389 - wavefront local address space cannot 7390 be used.* 7391 7392 1. ds_atomic 7393 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7394 7395 - Use lgkmcnt(0) if not 7396 TgSplit execution mode 7397 and vmcnt(0) if TgSplit 7398 execution mode. 7399 - If OpenCL, omit 7400 lgkmcnt(0). 7401 - Must happen after 7402 any preceding 7403 local/generic 7404 load/store/load 7405 atomic/store 7406 atomic/atomicrmw. 7407 - s_waitcnt vmcnt(0) 7408 must happen after 7409 any preceding 7410 global/generic load/store/ 7411 load atomic/store atomic/ 7412 atomicrmw. 7413 - s_waitcnt lgkmcnt(0) 7414 must happen after 7415 any preceding 7416 local/generic 7417 load/store/load 7418 atomic/store 7419 atomic/atomicrmw. 7420 - Must happen before 7421 the following 7422 atomicrmw. 7423 - Ensures that all 7424 memory operations 7425 have 7426 completed before 7427 performing the 7428 atomicrmw that is 7429 being released. 7430 7431 2. buffer/global_atomic 7432 3. s_waitcnt vmcnt(0) 7433 7434 - If not TgSplit execution 7435 mode, omit. 7436 - Must happen before 7437 the following 7438 buffer_wbinvl1_vol. 7439 - Ensures any 7440 following global 7441 data read is no 7442 older than the 7443 atomicrmw value 7444 being acquired. 7445 7446 4. buffer_wbinvl1_vol 7447 7448 - If not TgSplit execution 7449 mode, omit. 7450 - Ensures that 7451 following 7452 loads will not see 7453 stale data. 7454 7455 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7456 local address space cannot 7457 be used.* 7458 7459 1. ds_atomic 7460 2. s_waitcnt lgkmcnt(0) 7461 7462 - If OpenCL, omit. 7463 - Must happen before 7464 any following 7465 global/generic 7466 load/load 7467 atomic/store/store 7468 atomic/atomicrmw. 7469 - Ensures any 7470 following global 7471 data read is no 7472 older than the local load 7473 atomic value being 7474 acquired. 7475 7476 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7477 7478 - Use lgkmcnt(0) if not 7479 TgSplit execution mode 7480 and vmcnt(0) if TgSplit 7481 execution mode. 7482 - If OpenCL, omit 7483 lgkmcnt(0). 7484 - s_waitcnt vmcnt(0) 7485 must happen after 7486 any preceding 7487 global/generic load/store/ 7488 load atomic/store atomic/ 7489 atomicrmw. 7490 - s_waitcnt lgkmcnt(0) 7491 must happen after 7492 any preceding 7493 local/generic 7494 load/store/load 7495 atomic/store 7496 atomic/atomicrmw. 7497 - Must happen before 7498 the following 7499 atomicrmw. 7500 - Ensures that all 7501 memory operations 7502 have 7503 completed before 7504 performing the 7505 atomicrmw that is 7506 being released. 7507 7508 2. flat_atomic 7509 3. s_waitcnt lgkmcnt(0) & 7510 vmcnt(0) 7511 7512 - If not TgSplit execution 7513 mode, omit vmcnt(0). 7514 - If OpenCL, omit 7515 lgkmcnt(0). 7516 - Must happen before 7517 the following 7518 buffer_wbinvl1_vol and 7519 any following 7520 global/generic 7521 load/load 7522 atomic/store/store 7523 atomic/atomicrmw. 7524 - Ensures any 7525 following global 7526 data read is no 7527 older than a local load 7528 atomic value being 7529 acquired. 7530 7531 3. buffer_wbinvl1_vol 7532 7533 - If not TgSplit execution 7534 mode, omit. 7535 - Ensures that 7536 following 7537 loads will not see 7538 stale data. 7539 7540 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7541 vmcnt(0) 7542 7543 - If TgSplit execution mode, 7544 omit lgkmcnt(0). 7545 - If OpenCL, omit 7546 lgkmcnt(0). 7547 - Could be split into 7548 separate s_waitcnt 7549 vmcnt(0) and 7550 s_waitcnt 7551 lgkmcnt(0) to allow 7552 them to be 7553 independently moved 7554 according to the 7555 following rules. 7556 - s_waitcnt vmcnt(0) 7557 must happen after 7558 any preceding 7559 global/generic 7560 load/store/load 7561 atomic/store 7562 atomic/atomicrmw. 7563 - s_waitcnt lgkmcnt(0) 7564 must happen after 7565 any preceding 7566 local/generic 7567 load/store/load 7568 atomic/store 7569 atomic/atomicrmw. 7570 - Must happen before 7571 the following 7572 atomicrmw. 7573 - Ensures that all 7574 memory operations 7575 to global have 7576 completed before 7577 performing the 7578 atomicrmw that is 7579 being released. 7580 7581 2. buffer/global_atomic 7582 3. s_waitcnt vmcnt(0) 7583 7584 - Must happen before 7585 following 7586 buffer_wbinvl1_vol. 7587 - Ensures the 7588 atomicrmw has 7589 completed before 7590 invalidating the 7591 cache. 7592 7593 4. buffer_wbinvl1_vol 7594 7595 - Must happen before 7596 any following 7597 global/generic 7598 load/load 7599 atomic/atomicrmw. 7600 - Ensures that 7601 following loads 7602 will not see stale 7603 global data. 7604 7605 atomicrmw acq_rel - system - global 1. buffer_wbl2 7606 7607 - Must happen before 7608 following s_waitcnt. 7609 - Performs L2 writeback to 7610 ensure previous 7611 global/generic 7612 store/atomicrmw are 7613 visible at system scope. 7614 7615 2. s_waitcnt lgkmcnt(0) & 7616 vmcnt(0) 7617 7618 - If TgSplit execution mode, 7619 omit lgkmcnt(0). 7620 - If OpenCL, omit 7621 lgkmcnt(0). 7622 - Could be split into 7623 separate s_waitcnt 7624 vmcnt(0) and 7625 s_waitcnt 7626 lgkmcnt(0) to allow 7627 them to be 7628 independently moved 7629 according to the 7630 following rules. 7631 - s_waitcnt vmcnt(0) 7632 must happen after 7633 any preceding 7634 global/generic 7635 load/store/load 7636 atomic/store 7637 atomic/atomicrmw. 7638 - s_waitcnt lgkmcnt(0) 7639 must happen after 7640 any preceding 7641 local/generic 7642 load/store/load 7643 atomic/store 7644 atomic/atomicrmw. 7645 - Must happen before 7646 the following 7647 atomicrmw. 7648 - Ensures that all 7649 memory operations 7650 to global and L2 writeback 7651 have completed before 7652 performing the 7653 atomicrmw that is 7654 being released. 7655 7656 3. buffer/global_atomic 7657 scc=1 7658 4. s_waitcnt vmcnt(0) 7659 7660 - Must happen before 7661 following buffer_invl2 and 7662 buffer_wbinvl1_vol. 7663 - Ensures the 7664 atomicrmw has 7665 completed before 7666 invalidating the 7667 caches. 7668 7669 5. buffer_invl2; 7670 buffer_wbinvl1_vol 7671 7672 - Must happen before 7673 any following 7674 global/generic 7675 load/load 7676 atomic/atomicrmw. 7677 - Ensures that 7678 following loads 7679 will not see stale 7680 MTYPE NC global data. 7681 MTYPE RW and CC memory will 7682 never be stale due to the 7683 memory probes. 7684 7685 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7686 vmcnt(0) 7687 7688 - If TgSplit execution mode, 7689 omit lgkmcnt(0). 7690 - If OpenCL, omit 7691 lgkmcnt(0). 7692 - Could be split into 7693 separate s_waitcnt 7694 vmcnt(0) and 7695 s_waitcnt 7696 lgkmcnt(0) to allow 7697 them to be 7698 independently moved 7699 according to the 7700 following rules. 7701 - s_waitcnt vmcnt(0) 7702 must happen after 7703 any preceding 7704 global/generic 7705 load/store/load 7706 atomic/store 7707 atomic/atomicrmw. 7708 - s_waitcnt lgkmcnt(0) 7709 must happen after 7710 any preceding 7711 local/generic 7712 load/store/load 7713 atomic/store 7714 atomic/atomicrmw. 7715 - Must happen before 7716 the following 7717 atomicrmw. 7718 - Ensures that all 7719 memory operations 7720 to global have 7721 completed before 7722 performing the 7723 atomicrmw that is 7724 being released. 7725 7726 2. flat_atomic 7727 3. s_waitcnt vmcnt(0) & 7728 lgkmcnt(0) 7729 7730 - If TgSplit execution mode, 7731 omit lgkmcnt(0). 7732 - If OpenCL, omit 7733 lgkmcnt(0). 7734 - Must happen before 7735 following 7736 buffer_wbinvl1_vol. 7737 - Ensures the 7738 atomicrmw has 7739 completed before 7740 invalidating the 7741 cache. 7742 7743 4. buffer_wbinvl1_vol 7744 7745 - Must happen before 7746 any following 7747 global/generic 7748 load/load 7749 atomic/atomicrmw. 7750 - Ensures that 7751 following loads 7752 will not see stale 7753 global data. 7754 7755 atomicrmw acq_rel - system - generic 1. buffer_wbl2 7756 7757 - Must happen before 7758 following s_waitcnt. 7759 - Performs L2 writeback to 7760 ensure previous 7761 global/generic 7762 store/atomicrmw are 7763 visible at system scope. 7764 7765 2. s_waitcnt lgkmcnt(0) & 7766 vmcnt(0) 7767 7768 - If TgSplit execution mode, 7769 omit lgkmcnt(0). 7770 - If OpenCL, omit 7771 lgkmcnt(0). 7772 - Could be split into 7773 separate s_waitcnt 7774 vmcnt(0) and 7775 s_waitcnt 7776 lgkmcnt(0) to allow 7777 them to be 7778 independently moved 7779 according to the 7780 following rules. 7781 - s_waitcnt vmcnt(0) 7782 must happen after 7783 any preceding 7784 global/generic 7785 load/store/load 7786 atomic/store 7787 atomic/atomicrmw. 7788 - s_waitcnt lgkmcnt(0) 7789 must happen after 7790 any preceding 7791 local/generic 7792 load/store/load 7793 atomic/store 7794 atomic/atomicrmw. 7795 - Must happen before 7796 the following 7797 atomicrmw. 7798 - Ensures that all 7799 memory operations 7800 to global and L2 writeback 7801 have completed before 7802 performing the 7803 atomicrmw that is 7804 being released. 7805 7806 3. flat_atomic scc=1 7807 4. s_waitcnt vmcnt(0) & 7808 lgkmcnt(0) 7809 7810 - If TgSplit execution mode, 7811 omit lgkmcnt(0). 7812 - If OpenCL, omit 7813 lgkmcnt(0). 7814 - Must happen before 7815 following buffer_invl2 and 7816 buffer_wbinvl1_vol. 7817 - Ensures the 7818 atomicrmw has 7819 completed before 7820 invalidating the 7821 caches. 7822 7823 5. buffer_invl2; 7824 buffer_wbinvl1_vol 7825 7826 - Must happen before 7827 any following 7828 global/generic 7829 load/load 7830 atomic/atomicrmw. 7831 - Ensures that 7832 following loads 7833 will not see stale 7834 MTYPE NC global data. 7835 MTYPE RW and CC memory will 7836 never be stale due to the 7837 memory probes. 7838 7839 fence acq_rel - singlethread *none* *none* 7840 - wavefront 7841 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7842 7843 - Use lgkmcnt(0) if not 7844 TgSplit execution mode 7845 and vmcnt(0) if TgSplit 7846 execution mode. 7847 - If OpenCL and 7848 address space is 7849 not generic, omit 7850 lgkmcnt(0). 7851 - If OpenCL and 7852 address space is 7853 local, omit 7854 vmcnt(0). 7855 - However, 7856 since LLVM 7857 currently has no 7858 address space on 7859 the fence need to 7860 conservatively 7861 always generate 7862 (see comment for 7863 previous fence). 7864 - s_waitcnt vmcnt(0) 7865 must happen after 7866 any preceding 7867 global/generic 7868 load/store/ 7869 load atomic/store atomic/ 7870 atomicrmw. 7871 - s_waitcnt lgkmcnt(0) 7872 must happen after 7873 any preceding 7874 local/generic 7875 load/load 7876 atomic/store/store 7877 atomic/atomicrmw. 7878 - Must happen before 7879 any following 7880 global/generic 7881 load/load 7882 atomic/store/store 7883 atomic/atomicrmw. 7884 - Ensures that all 7885 memory operations 7886 have 7887 completed before 7888 performing any 7889 following global 7890 memory operations. 7891 - Ensures that the 7892 preceding 7893 local/generic load 7894 atomic/atomicrmw 7895 with an equal or 7896 wider sync scope 7897 and memory ordering 7898 stronger than 7899 unordered (this is 7900 termed the 7901 acquire-fence-paired-atomic) 7902 has completed 7903 before following 7904 global memory 7905 operations. This 7906 satisfies the 7907 requirements of 7908 acquire. 7909 - Ensures that all 7910 previous memory 7911 operations have 7912 completed before a 7913 following 7914 local/generic store 7915 atomic/atomicrmw 7916 with an equal or 7917 wider sync scope 7918 and memory ordering 7919 stronger than 7920 unordered (this is 7921 termed the 7922 release-fence-paired-atomic). 7923 This satisfies the 7924 requirements of 7925 release. 7926 - Must happen before 7927 the following 7928 buffer_wbinvl1_vol. 7929 - Ensures that the 7930 acquire-fence-paired 7931 atomic has completed 7932 before invalidating 7933 the 7934 cache. Therefore 7935 any following 7936 locations read must 7937 be no older than 7938 the value read by 7939 the 7940 acquire-fence-paired-atomic. 7941 7942 3. buffer_wbinvl1_vol 7943 7944 - If not TgSplit execution 7945 mode, omit. 7946 - Ensures that 7947 following 7948 loads will not see 7949 stale data. 7950 7951 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7952 vmcnt(0) 7953 7954 - If TgSplit execution mode, 7955 omit lgkmcnt(0). 7956 - If OpenCL and 7957 address space is 7958 not generic, omit 7959 lgkmcnt(0). 7960 - However, since LLVM 7961 currently has no 7962 address space on 7963 the fence need to 7964 conservatively 7965 always generate 7966 (see comment for 7967 previous fence). 7968 - Could be split into 7969 separate s_waitcnt 7970 vmcnt(0) and 7971 s_waitcnt 7972 lgkmcnt(0) to allow 7973 them to be 7974 independently moved 7975 according to the 7976 following rules. 7977 - s_waitcnt vmcnt(0) 7978 must happen after 7979 any preceding 7980 global/generic 7981 load/store/load 7982 atomic/store 7983 atomic/atomicrmw. 7984 - s_waitcnt lgkmcnt(0) 7985 must happen after 7986 any preceding 7987 local/generic 7988 load/store/load 7989 atomic/store 7990 atomic/atomicrmw. 7991 - Must happen before 7992 the following 7993 buffer_wbinvl1_vol. 7994 - Ensures that the 7995 preceding 7996 global/local/generic 7997 load 7998 atomic/atomicrmw 7999 with an equal or 8000 wider sync scope 8001 and memory ordering 8002 stronger than 8003 unordered (this is 8004 termed the 8005 acquire-fence-paired-atomic) 8006 has completed 8007 before invalidating 8008 the cache. This 8009 satisfies the 8010 requirements of 8011 acquire. 8012 - Ensures that all 8013 previous memory 8014 operations have 8015 completed before a 8016 following 8017 global/local/generic 8018 store 8019 atomic/atomicrmw 8020 with an equal or 8021 wider sync scope 8022 and memory ordering 8023 stronger than 8024 unordered (this is 8025 termed the 8026 release-fence-paired-atomic). 8027 This satisfies the 8028 requirements of 8029 release. 8030 8031 2. buffer_wbinvl1_vol 8032 8033 - Must happen before 8034 any following 8035 global/generic 8036 load/load 8037 atomic/store/store 8038 atomic/atomicrmw. 8039 - Ensures that 8040 following loads 8041 will not see stale 8042 global data. This 8043 satisfies the 8044 requirements of 8045 acquire. 8046 8047 fence acq_rel - system *none* 1. buffer_wbl2 8048 8049 - If OpenCL and 8050 address space is 8051 local, omit. 8052 - Must happen before 8053 following s_waitcnt. 8054 - Performs L2 writeback to 8055 ensure previous 8056 global/generic 8057 store/atomicrmw are 8058 visible at system scope. 8059 8060 2. s_waitcnt lgkmcnt(0) & 8061 vmcnt(0) 8062 8063 - If TgSplit execution mode, 8064 omit lgkmcnt(0). 8065 - If OpenCL and 8066 address space is 8067 not generic, omit 8068 lgkmcnt(0). 8069 - However, since LLVM 8070 currently has no 8071 address space on 8072 the fence need to 8073 conservatively 8074 always generate 8075 (see comment for 8076 previous fence). 8077 - Could be split into 8078 separate s_waitcnt 8079 vmcnt(0) and 8080 s_waitcnt 8081 lgkmcnt(0) to allow 8082 them to be 8083 independently moved 8084 according to the 8085 following rules. 8086 - s_waitcnt vmcnt(0) 8087 must happen after 8088 any preceding 8089 global/generic 8090 load/store/load 8091 atomic/store 8092 atomic/atomicrmw. 8093 - s_waitcnt lgkmcnt(0) 8094 must happen after 8095 any preceding 8096 local/generic 8097 load/store/load 8098 atomic/store 8099 atomic/atomicrmw. 8100 - Must happen before 8101 the following buffer_invl2 and 8102 buffer_wbinvl1_vol. 8103 - Ensures that the 8104 preceding 8105 global/local/generic 8106 load 8107 atomic/atomicrmw 8108 with an equal or 8109 wider sync scope 8110 and memory ordering 8111 stronger than 8112 unordered (this is 8113 termed the 8114 acquire-fence-paired-atomic) 8115 has completed 8116 before invalidating 8117 the cache. This 8118 satisfies the 8119 requirements of 8120 acquire. 8121 - Ensures that all 8122 previous memory 8123 operations have 8124 completed before a 8125 following 8126 global/local/generic 8127 store 8128 atomic/atomicrmw 8129 with an equal or 8130 wider sync scope 8131 and memory ordering 8132 stronger than 8133 unordered (this is 8134 termed the 8135 release-fence-paired-atomic). 8136 This satisfies the 8137 requirements of 8138 release. 8139 8140 3. buffer_invl2; 8141 buffer_wbinvl1_vol 8142 8143 - Must happen before 8144 any following 8145 global/generic 8146 load/load 8147 atomic/store/store 8148 atomic/atomicrmw. 8149 - Ensures that 8150 following loads 8151 will not see stale 8152 MTYPE NC global data. 8153 MTYPE RW and CC memory will 8154 never be stale due to the 8155 memory probes. 8156 8157 **Sequential Consistent Atomic** 8158 ------------------------------------------------------------------------------------ 8159 load atomic seq_cst - singlethread - global *Same as corresponding 8160 - wavefront - local load atomic acquire, 8161 - generic except must generated 8162 all instructions even 8163 for OpenCL.* 8164 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8165 - generic 8166 - Use lgkmcnt(0) if not 8167 TgSplit execution mode 8168 and vmcnt(0) if TgSplit 8169 execution mode. 8170 - s_waitcnt lgkmcnt(0) must 8171 happen after 8172 preceding 8173 local/generic load 8174 atomic/store 8175 atomic/atomicrmw 8176 with memory 8177 ordering of seq_cst 8178 and with equal or 8179 wider sync scope. 8180 (Note that seq_cst 8181 fences have their 8182 own s_waitcnt 8183 lgkmcnt(0) and so do 8184 not need to be 8185 considered.) 8186 - s_waitcnt vmcnt(0) 8187 must happen after 8188 preceding 8189 global/generic load 8190 atomic/store 8191 atomic/atomicrmw 8192 with memory 8193 ordering of seq_cst 8194 and with equal or 8195 wider sync scope. 8196 (Note that seq_cst 8197 fences have their 8198 own s_waitcnt 8199 vmcnt(0) and so do 8200 not need to be 8201 considered.) 8202 - Ensures any 8203 preceding 8204 sequential 8205 consistent global/local 8206 memory instructions 8207 have completed 8208 before executing 8209 this sequentially 8210 consistent 8211 instruction. This 8212 prevents reordering 8213 a seq_cst store 8214 followed by a 8215 seq_cst load. (Note 8216 that seq_cst is 8217 stronger than 8218 acquire/release as 8219 the reordering of 8220 load acquire 8221 followed by a store 8222 release is 8223 prevented by the 8224 s_waitcnt of 8225 the release, but 8226 there is nothing 8227 preventing a store 8228 release followed by 8229 load acquire from 8230 completing out of 8231 order. The s_waitcnt 8232 could be placed after 8233 seq_store or before 8234 the seq_load. We 8235 choose the load to 8236 make the s_waitcnt be 8237 as late as possible 8238 so that the store 8239 may have already 8240 completed.) 8241 8242 2. *Following 8243 instructions same as 8244 corresponding load 8245 atomic acquire, 8246 except must generated 8247 all instructions even 8248 for OpenCL.* 8249 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8250 local address space cannot 8251 be used.* 8252 8253 *Same as corresponding 8254 load atomic acquire, 8255 except must generated 8256 all instructions even 8257 for OpenCL.* 8258 8259 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8260 - system - generic vmcnt(0) 8261 8262 - If TgSplit execution mode, 8263 omit lgkmcnt(0). 8264 - Could be split into 8265 separate s_waitcnt 8266 vmcnt(0) 8267 and s_waitcnt 8268 lgkmcnt(0) to allow 8269 them to be 8270 independently moved 8271 according to the 8272 following rules. 8273 - s_waitcnt lgkmcnt(0) 8274 must happen after 8275 preceding 8276 global/generic load 8277 atomic/store 8278 atomic/atomicrmw 8279 with memory 8280 ordering of seq_cst 8281 and with equal or 8282 wider sync scope. 8283 (Note that seq_cst 8284 fences have their 8285 own s_waitcnt 8286 lgkmcnt(0) and so do 8287 not need to be 8288 considered.) 8289 - s_waitcnt vmcnt(0) 8290 must happen after 8291 preceding 8292 global/generic load 8293 atomic/store 8294 atomic/atomicrmw 8295 with memory 8296 ordering of seq_cst 8297 and with equal or 8298 wider sync scope. 8299 (Note that seq_cst 8300 fences have their 8301 own s_waitcnt 8302 vmcnt(0) and so do 8303 not need to be 8304 considered.) 8305 - Ensures any 8306 preceding 8307 sequential 8308 consistent global 8309 memory instructions 8310 have completed 8311 before executing 8312 this sequentially 8313 consistent 8314 instruction. This 8315 prevents reordering 8316 a seq_cst store 8317 followed by a 8318 seq_cst load. (Note 8319 that seq_cst is 8320 stronger than 8321 acquire/release as 8322 the reordering of 8323 load acquire 8324 followed by a store 8325 release is 8326 prevented by the 8327 s_waitcnt of 8328 the release, but 8329 there is nothing 8330 preventing a store 8331 release followed by 8332 load acquire from 8333 completing out of 8334 order. The s_waitcnt 8335 could be placed after 8336 seq_store or before 8337 the seq_load. We 8338 choose the load to 8339 make the s_waitcnt be 8340 as late as possible 8341 so that the store 8342 may have already 8343 completed.) 8344 8345 2. *Following 8346 instructions same as 8347 corresponding load 8348 atomic acquire, 8349 except must generated 8350 all instructions even 8351 for OpenCL.* 8352 store atomic seq_cst - singlethread - global *Same as corresponding 8353 - wavefront - local store atomic release, 8354 - workgroup - generic except must generated 8355 - agent all instructions even 8356 - system for OpenCL.* 8357 atomicrmw seq_cst - singlethread - global *Same as corresponding 8358 - wavefront - local atomicrmw acq_rel, 8359 - workgroup - generic except must generated 8360 - agent all instructions even 8361 - system for OpenCL.* 8362 fence seq_cst - singlethread *none* *Same as corresponding 8363 - wavefront fence acq_rel, 8364 - workgroup except must generated 8365 - agent all instructions even 8366 - system for OpenCL.* 8367 ============ ============ ============== ========== ================================ 8368 8369.. _amdgpu-amdhsa-memory-model-gfx10: 8370 8371Memory Model GFX10 8372++++++++++++++++++ 8373 8374For GFX10: 8375 8376* Each agent has multiple shader arrays (SA). 8377* Each SA has multiple work-group processors (WGP). 8378* Each WGP has multiple compute units (CU). 8379* Each CU has multiple SIMDs that execute wavefronts. 8380* The wavefronts for a single work-group are executed in the same 8381 WGP. In CU wavefront execution mode the wavefronts may be executed by 8382 different SIMDs in the same CU. In WGP wavefront execution mode the 8383 wavefronts may be executed by different SIMDs in different CUs in the same 8384 WGP. 8385* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8386 executing on it. 8387* All LDS operations of a WGP are performed as wavefront wide operations in a 8388 global order and involve no caching. Completion is reported to a wavefront in 8389 execution order. 8390* The LDS memory has multiple request queues shared by the SIMDs of a 8391 WGP. Therefore, the LDS operations performed by different wavefronts of a 8392 work-group can be reordered relative to each other, which can result in 8393 reordering the visibility of vector memory operations with respect to LDS 8394 operations of other wavefronts in the same work-group. A ``s_waitcnt 8395 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8396 vector memory operations between wavefronts of a work-group, but not between 8397 operations performed by the same wavefront. 8398* The vector memory operations are performed as wavefront wide operations. 8399 Completion of load/store/sample operations are reported to a wavefront in 8400 execution order of other load/store/sample operations performed by that 8401 wavefront. 8402* The vector memory operations access a vector L0 cache. There is a single L0 8403 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8404 special action is required for coherence between the lanes of a single 8405 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8406 wavefronts executing in the same work-group as they may be executing on SIMDs 8407 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8408 required for coherence between wavefronts executing in different work-groups 8409 as they may be executing on different WGPs. 8410* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8411 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8412 operations are used in a restricted way so do not impact the memory model. See 8413 :ref:`amdgpu-amdhsa-memory-spaces`. 8414* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8415 the same SA. Therefore, no special action is required for coherence between 8416 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8417 required for coherence between wavefronts executing in different work-groups 8418 as they may be executing on different SAs that access different L1s. 8419* The L1 caches have independent quadrants to service disjoint ranges of virtual 8420 addresses. 8421* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8422 vector and scalar memory operations performed by different wavefronts, whether 8423 executing in the same or different work-groups (which may be executing on 8424 different CUs accessing different L0s), can be reordered relative to each 8425 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8426 synchronization between vector memory operations of different wavefronts. It 8427 ensures a previous vector memory operation has completed before executing a 8428 subsequent vector memory or LDS operation and so can be used to meet the 8429 requirements of acquire, release and sequential consistency. 8430* The L1 caches use an L2 cache shared by all SAs on the same agent. 8431* The L2 cache has independent channels to service disjoint ranges of virtual 8432 addresses. 8433* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8434 quadrant has a separate request queue per L2 channel. Therefore, the vector 8435 and scalar memory operations performed by wavefronts executing in different 8436 work-groups (which may be executing on different SAs) of an agent can be 8437 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8438 required to ensure synchronization between vector memory operations of 8439 different SAs. It ensures a previous vector memory operation has completed 8440 before executing a subsequent vector memory and so can be used to meet the 8441 requirements of acquire, release and sequential consistency. 8442* The L2 cache can be kept coherent with other agents on some targets, or ranges 8443 of virtual addresses can be set up to bypass it to ensure system coherence. 8444 8445Scalar memory operations are only used to access memory that is proven to not 8446change during the execution of the kernel dispatch. This includes constant 8447address space and global address space for program scope ``const`` variables. 8448Therefore, the kernel machine code does not have to maintain the scalar cache to 8449ensure it is coherent with the vector caches. The scalar and vector caches are 8450invalidated between kernel dispatches by CP since constant address space data 8451may change between kernel dispatch executions. See 8452:ref:`amdgpu-amdhsa-memory-spaces`. 8453 8454The one exception is if scalar writes are used to spill SGPR registers. In this 8455case the AMDGPU backend ensures the memory location used to spill is never 8456accessed by vector memory operations at the same time. If scalar writes are used 8457then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8458return since the locations may be used for vector memory instructions by a 8459future wavefront that uses the same scratch area, or a function call that 8460creates a frame at the same address, respectively. There is no need for a 8461``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8462 8463For kernarg backing memory: 8464 8465* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8466* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8467 needing to invalidate the L2 cache. 8468* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8469 so the L2 cache will be coherent with the CPU and other agents. 8470 8471Scratch backing memory (which is used for the private address space) is accessed 8472with MTYPE NC (non-coherent). Since the private address space is only accessed 8473by a single thread, and is always write-before-read, there is never a need to 8474invalidate these entries from the L0 or L1 caches. 8475 8476Wavefronts are executed in native mode with in-order reporting of loads and 8477sample instructions. In this mode vmcnt reports completion of load, atomic with 8478return and sample instructions in order, and the vscnt reports the completion of 8479store and atomic without return in order. See ``MEM_ORDERED`` field in 8480:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8481 8482Wavefronts can be executed in WGP or CU wavefront execution mode: 8483 8484* In WGP wavefront execution mode the wavefronts of a work-group are executed 8485 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8486 CU L0 caches is required for work-group synchronization. Also accesses to L1 8487 at work-group scope need to be explicitly ordered as the accesses from 8488 different CUs are not ordered. 8489* In CU wavefront execution mode the wavefronts of a work-group are executed on 8490 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8491 the work-group access the same L0 which in turn ensures L1 accesses are 8492 ordered and so do not require explicit management of the caches for 8493 work-group synchronization. 8494 8495See ``WGP_MODE`` field in 8496:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8497:ref:`amdgpu-target-features`. 8498 8499The code sequences used to implement the memory model for GFX10 are defined in 8500table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8501 8502 .. table:: AMDHSA Memory Model Code Sequences GFX10 8503 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8504 8505 ============ ============ ============== ========== ================================ 8506 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8507 Ordering Sync Scope Address GFX10 8508 Space 8509 ============ ============ ============== ========== ================================ 8510 **Non-Atomic** 8511 ------------------------------------------------------------------------------------ 8512 load *none* *none* - global - !volatile & !nontemporal 8513 - generic 8514 - private 1. buffer/global/flat_load 8515 - constant 8516 - !volatile & nontemporal 8517 8518 1. buffer/global/flat_load 8519 slc=1 8520 8521 - volatile 8522 8523 1. buffer/global/flat_load 8524 glc=1 dlc=1 8525 2. s_waitcnt vmcnt(0) 8526 8527 - Must happen before 8528 any following volatile 8529 global/generic 8530 load/store. 8531 - Ensures that 8532 volatile 8533 operations to 8534 different 8535 addresses will not 8536 be reordered by 8537 hardware. 8538 8539 load *none* *none* - local 1. ds_load 8540 store *none* *none* - global - !volatile & !nontemporal 8541 - generic 8542 - private 1. buffer/global/flat_store 8543 - constant 8544 - !volatile & nontemporal 8545 8546 1. buffer/global/flat_store 8547 slc=1 8548 8549 - volatile 8550 8551 1. buffer/global/flat_store 8552 2. s_waitcnt vscnt(0) 8553 8554 - Must happen before 8555 any following volatile 8556 global/generic 8557 load/store. 8558 - Ensures that 8559 volatile 8560 operations to 8561 different 8562 addresses will not 8563 be reordered by 8564 hardware. 8565 8566 store *none* *none* - local 1. ds_store 8567 **Unordered Atomic** 8568 ------------------------------------------------------------------------------------ 8569 load atomic unordered *any* *any* *Same as non-atomic*. 8570 store atomic unordered *any* *any* *Same as non-atomic*. 8571 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8572 **Monotonic Atomic** 8573 ------------------------------------------------------------------------------------ 8574 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8575 - wavefront - generic 8576 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8577 - generic glc=1 8578 8579 - If CU wavefront execution 8580 mode, omit glc=1. 8581 8582 load atomic monotonic - singlethread - local 1. ds_load 8583 - wavefront 8584 - workgroup 8585 load atomic monotonic - agent - global 1. buffer/global/flat_load 8586 - system - generic glc=1 dlc=1 8587 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8588 - wavefront - generic 8589 - workgroup 8590 - agent 8591 - system 8592 store atomic monotonic - singlethread - local 1. ds_store 8593 - wavefront 8594 - workgroup 8595 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8596 - wavefront - generic 8597 - workgroup 8598 - agent 8599 - system 8600 atomicrmw monotonic - singlethread - local 1. ds_atomic 8601 - wavefront 8602 - workgroup 8603 **Acquire Atomic** 8604 ------------------------------------------------------------------------------------ 8605 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8606 - wavefront - local 8607 - generic 8608 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8609 8610 - If CU wavefront execution 8611 mode, omit glc=1. 8612 8613 2. s_waitcnt vmcnt(0) 8614 8615 - If CU wavefront execution 8616 mode, omit. 8617 - Must happen before 8618 the following buffer_gl0_inv 8619 and before any following 8620 global/generic 8621 load/load 8622 atomic/store/store 8623 atomic/atomicrmw. 8624 8625 3. buffer_gl0_inv 8626 8627 - If CU wavefront execution 8628 mode, omit. 8629 - Ensures that 8630 following 8631 loads will not see 8632 stale data. 8633 8634 load atomic acquire - workgroup - local 1. ds_load 8635 2. s_waitcnt lgkmcnt(0) 8636 8637 - If OpenCL, omit. 8638 - Must happen before 8639 the following buffer_gl0_inv 8640 and before any following 8641 global/generic load/load 8642 atomic/store/store 8643 atomic/atomicrmw. 8644 - Ensures any 8645 following global 8646 data read is no 8647 older than the local load 8648 atomic value being 8649 acquired. 8650 8651 3. buffer_gl0_inv 8652 8653 - If CU wavefront execution 8654 mode, omit. 8655 - If OpenCL, omit. 8656 - Ensures that 8657 following 8658 loads will not see 8659 stale data. 8660 8661 load atomic acquire - workgroup - generic 1. flat_load glc=1 8662 8663 - If CU wavefront execution 8664 mode, omit glc=1. 8665 8666 2. s_waitcnt lgkmcnt(0) & 8667 vmcnt(0) 8668 8669 - If CU wavefront execution 8670 mode, omit vmcnt(0). 8671 - If OpenCL, omit 8672 lgkmcnt(0). 8673 - Must happen before 8674 the following 8675 buffer_gl0_inv and any 8676 following global/generic 8677 load/load 8678 atomic/store/store 8679 atomic/atomicrmw. 8680 - Ensures any 8681 following global 8682 data read is no 8683 older than a local load 8684 atomic value being 8685 acquired. 8686 8687 3. buffer_gl0_inv 8688 8689 - If CU wavefront execution 8690 mode, omit. 8691 - Ensures that 8692 following 8693 loads will not see 8694 stale data. 8695 8696 load atomic acquire - agent - global 1. buffer/global_load 8697 - system glc=1 dlc=1 8698 2. s_waitcnt vmcnt(0) 8699 8700 - Must happen before 8701 following 8702 buffer_gl*_inv. 8703 - Ensures the load 8704 has completed 8705 before invalidating 8706 the caches. 8707 8708 3. buffer_gl0_inv; 8709 buffer_gl1_inv 8710 8711 - Must happen before 8712 any following 8713 global/generic 8714 load/load 8715 atomic/atomicrmw. 8716 - Ensures that 8717 following 8718 loads will not see 8719 stale global data. 8720 8721 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8722 - system 2. s_waitcnt vmcnt(0) & 8723 lgkmcnt(0) 8724 8725 - If OpenCL omit 8726 lgkmcnt(0). 8727 - Must happen before 8728 following 8729 buffer_gl*_invl. 8730 - Ensures the flat_load 8731 has completed 8732 before invalidating 8733 the caches. 8734 8735 3. buffer_gl0_inv; 8736 buffer_gl1_inv 8737 8738 - Must happen before 8739 any following 8740 global/generic 8741 load/load 8742 atomic/atomicrmw. 8743 - Ensures that 8744 following loads 8745 will not see stale 8746 global data. 8747 8748 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8749 - wavefront - local 8750 - generic 8751 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8752 2. s_waitcnt vm/vscnt(0) 8753 8754 - If CU wavefront execution 8755 mode, omit. 8756 - Use vmcnt(0) if atomic with 8757 return and vscnt(0) if 8758 atomic with no-return. 8759 - Must happen before 8760 the following buffer_gl0_inv 8761 and before any following 8762 global/generic 8763 load/load 8764 atomic/store/store 8765 atomic/atomicrmw. 8766 8767 3. buffer_gl0_inv 8768 8769 - If CU wavefront execution 8770 mode, omit. 8771 - Ensures that 8772 following 8773 loads will not see 8774 stale data. 8775 8776 atomicrmw acquire - workgroup - local 1. ds_atomic 8777 2. s_waitcnt lgkmcnt(0) 8778 8779 - If OpenCL, omit. 8780 - Must happen before 8781 the following 8782 buffer_gl0_inv. 8783 - Ensures any 8784 following global 8785 data read is no 8786 older than the local 8787 atomicrmw value 8788 being acquired. 8789 8790 3. buffer_gl0_inv 8791 8792 - If OpenCL omit. 8793 - Ensures that 8794 following 8795 loads will not see 8796 stale data. 8797 8798 atomicrmw acquire - workgroup - generic 1. flat_atomic 8799 2. s_waitcnt lgkmcnt(0) & 8800 vm/vscnt(0) 8801 8802 - If CU wavefront execution 8803 mode, omit vm/vscnt(0). 8804 - If OpenCL, omit lgkmcnt(0). 8805 - Use vmcnt(0) if atomic with 8806 return and vscnt(0) if 8807 atomic with no-return. 8808 - Must happen before 8809 the following 8810 buffer_gl0_inv. 8811 - Ensures any 8812 following global 8813 data read is no 8814 older than a local 8815 atomicrmw value 8816 being acquired. 8817 8818 3. buffer_gl0_inv 8819 8820 - If CU wavefront execution 8821 mode, omit. 8822 - Ensures that 8823 following 8824 loads will not see 8825 stale data. 8826 8827 atomicrmw acquire - agent - global 1. buffer/global_atomic 8828 - system 2. s_waitcnt vm/vscnt(0) 8829 8830 - Use vmcnt(0) if atomic with 8831 return and vscnt(0) if 8832 atomic with no-return. 8833 - Must happen before 8834 following 8835 buffer_gl*_inv. 8836 - Ensures the 8837 atomicrmw has 8838 completed before 8839 invalidating the 8840 caches. 8841 8842 3. buffer_gl0_inv; 8843 buffer_gl1_inv 8844 8845 - Must happen before 8846 any following 8847 global/generic 8848 load/load 8849 atomic/atomicrmw. 8850 - Ensures that 8851 following loads 8852 will not see stale 8853 global data. 8854 8855 atomicrmw acquire - agent - generic 1. flat_atomic 8856 - system 2. s_waitcnt vm/vscnt(0) & 8857 lgkmcnt(0) 8858 8859 - If OpenCL, omit 8860 lgkmcnt(0). 8861 - Use vmcnt(0) if atomic with 8862 return and vscnt(0) if 8863 atomic with no-return. 8864 - Must happen before 8865 following 8866 buffer_gl*_inv. 8867 - Ensures the 8868 atomicrmw has 8869 completed before 8870 invalidating the 8871 caches. 8872 8873 3. buffer_gl0_inv; 8874 buffer_gl1_inv 8875 8876 - Must happen before 8877 any following 8878 global/generic 8879 load/load 8880 atomic/atomicrmw. 8881 - Ensures that 8882 following loads 8883 will not see stale 8884 global data. 8885 8886 fence acquire - singlethread *none* *none* 8887 - wavefront 8888 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 8889 vmcnt(0) & vscnt(0) 8890 8891 - If CU wavefront execution 8892 mode, omit vmcnt(0) and 8893 vscnt(0). 8894 - If OpenCL and 8895 address space is 8896 not generic, omit 8897 lgkmcnt(0). 8898 - If OpenCL and 8899 address space is 8900 local, omit 8901 vmcnt(0) and vscnt(0). 8902 - However, since LLVM 8903 currently has no 8904 address space on 8905 the fence need to 8906 conservatively 8907 always generate. If 8908 fence had an 8909 address space then 8910 set to address 8911 space of OpenCL 8912 fence flag, or to 8913 generic if both 8914 local and global 8915 flags are 8916 specified. 8917 - Could be split into 8918 separate s_waitcnt 8919 vmcnt(0), s_waitcnt 8920 vscnt(0) and s_waitcnt 8921 lgkmcnt(0) to allow 8922 them to be 8923 independently moved 8924 according to the 8925 following rules. 8926 - s_waitcnt vmcnt(0) 8927 must happen after 8928 any preceding 8929 global/generic load 8930 atomic/ 8931 atomicrmw-with-return-value 8932 with an equal or 8933 wider sync scope 8934 and memory ordering 8935 stronger than 8936 unordered (this is 8937 termed the 8938 fence-paired-atomic). 8939 - s_waitcnt vscnt(0) 8940 must happen after 8941 any preceding 8942 global/generic 8943 atomicrmw-no-return-value 8944 with an equal or 8945 wider sync scope 8946 and memory ordering 8947 stronger than 8948 unordered (this is 8949 termed the 8950 fence-paired-atomic). 8951 - s_waitcnt lgkmcnt(0) 8952 must happen after 8953 any preceding 8954 local/generic load 8955 atomic/atomicrmw 8956 with an equal or 8957 wider sync scope 8958 and memory ordering 8959 stronger than 8960 unordered (this is 8961 termed the 8962 fence-paired-atomic). 8963 - Must happen before 8964 the following 8965 buffer_gl0_inv. 8966 - Ensures that the 8967 fence-paired atomic 8968 has completed 8969 before invalidating 8970 the 8971 cache. Therefore 8972 any following 8973 locations read must 8974 be no older than 8975 the value read by 8976 the 8977 fence-paired-atomic. 8978 8979 3. buffer_gl0_inv 8980 8981 - If CU wavefront execution 8982 mode, omit. 8983 - Ensures that 8984 following 8985 loads will not see 8986 stale data. 8987 8988 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 8989 - system vmcnt(0) & vscnt(0) 8990 8991 - If OpenCL and 8992 address space is 8993 not generic, omit 8994 lgkmcnt(0). 8995 - If OpenCL and 8996 address space is 8997 local, omit 8998 vmcnt(0) and vscnt(0). 8999 - However, since LLVM 9000 currently has no 9001 address space on 9002 the fence need to 9003 conservatively 9004 always generate 9005 (see comment for 9006 previous fence). 9007 - Could be split into 9008 separate s_waitcnt 9009 vmcnt(0), s_waitcnt 9010 vscnt(0) and s_waitcnt 9011 lgkmcnt(0) to allow 9012 them to be 9013 independently moved 9014 according to the 9015 following rules. 9016 - s_waitcnt vmcnt(0) 9017 must happen after 9018 any preceding 9019 global/generic load 9020 atomic/ 9021 atomicrmw-with-return-value 9022 with an equal or 9023 wider sync scope 9024 and memory ordering 9025 stronger than 9026 unordered (this is 9027 termed the 9028 fence-paired-atomic). 9029 - s_waitcnt vscnt(0) 9030 must happen after 9031 any preceding 9032 global/generic 9033 atomicrmw-no-return-value 9034 with an equal or 9035 wider sync scope 9036 and memory ordering 9037 stronger than 9038 unordered (this is 9039 termed the 9040 fence-paired-atomic). 9041 - s_waitcnt lgkmcnt(0) 9042 must happen after 9043 any preceding 9044 local/generic load 9045 atomic/atomicrmw 9046 with an equal or 9047 wider sync scope 9048 and memory ordering 9049 stronger than 9050 unordered (this is 9051 termed the 9052 fence-paired-atomic). 9053 - Must happen before 9054 the following 9055 buffer_gl*_inv. 9056 - Ensures that the 9057 fence-paired atomic 9058 has completed 9059 before invalidating 9060 the 9061 caches. Therefore 9062 any following 9063 locations read must 9064 be no older than 9065 the value read by 9066 the 9067 fence-paired-atomic. 9068 9069 2. buffer_gl0_inv; 9070 buffer_gl1_inv 9071 9072 - Must happen before any 9073 following global/generic 9074 load/load 9075 atomic/store/store 9076 atomic/atomicrmw. 9077 - Ensures that 9078 following loads 9079 will not see stale 9080 global data. 9081 9082 **Release Atomic** 9083 ------------------------------------------------------------------------------------ 9084 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9085 - wavefront - local 9086 - generic 9087 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9088 - generic vmcnt(0) & vscnt(0) 9089 9090 - If CU wavefront execution 9091 mode, omit vmcnt(0) and 9092 vscnt(0). 9093 - If OpenCL, omit 9094 lgkmcnt(0). 9095 - Could be split into 9096 separate s_waitcnt 9097 vmcnt(0), s_waitcnt 9098 vscnt(0) and s_waitcnt 9099 lgkmcnt(0) to allow 9100 them to be 9101 independently moved 9102 according to the 9103 following rules. 9104 - s_waitcnt vmcnt(0) 9105 must happen after 9106 any preceding 9107 global/generic load/load 9108 atomic/ 9109 atomicrmw-with-return-value. 9110 - s_waitcnt vscnt(0) 9111 must happen after 9112 any preceding 9113 global/generic 9114 store/store 9115 atomic/ 9116 atomicrmw-no-return-value. 9117 - s_waitcnt lgkmcnt(0) 9118 must happen after 9119 any preceding 9120 local/generic 9121 load/store/load 9122 atomic/store 9123 atomic/atomicrmw. 9124 - Must happen before 9125 the following 9126 store. 9127 - Ensures that all 9128 memory operations 9129 have 9130 completed before 9131 performing the 9132 store that is being 9133 released. 9134 9135 2. buffer/global/flat_store 9136 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9137 9138 - If CU wavefront execution 9139 mode, omit. 9140 - If OpenCL, omit. 9141 - Could be split into 9142 separate s_waitcnt 9143 vmcnt(0) and s_waitcnt 9144 vscnt(0) to allow 9145 them to be 9146 independently moved 9147 according to the 9148 following rules. 9149 - s_waitcnt vmcnt(0) 9150 must happen after 9151 any preceding 9152 global/generic load/load 9153 atomic/ 9154 atomicrmw-with-return-value. 9155 - s_waitcnt vscnt(0) 9156 must happen after 9157 any preceding 9158 global/generic 9159 store/store atomic/ 9160 atomicrmw-no-return-value. 9161 - Must happen before 9162 the following 9163 store. 9164 - Ensures that all 9165 global memory 9166 operations have 9167 completed before 9168 performing the 9169 store that is being 9170 released. 9171 9172 2. ds_store 9173 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9174 - system - generic vmcnt(0) & vscnt(0) 9175 9176 - If OpenCL and 9177 address space is 9178 not generic, omit 9179 lgkmcnt(0). 9180 - Could be split into 9181 separate s_waitcnt 9182 vmcnt(0), s_waitcnt vscnt(0) 9183 and s_waitcnt 9184 lgkmcnt(0) to allow 9185 them to be 9186 independently moved 9187 according to the 9188 following rules. 9189 - s_waitcnt vmcnt(0) 9190 must happen after 9191 any preceding 9192 global/generic 9193 load/load 9194 atomic/ 9195 atomicrmw-with-return-value. 9196 - s_waitcnt vscnt(0) 9197 must happen after 9198 any preceding 9199 global/generic 9200 store/store atomic/ 9201 atomicrmw-no-return-value. 9202 - s_waitcnt lgkmcnt(0) 9203 must happen after 9204 any preceding 9205 local/generic 9206 load/store/load 9207 atomic/store 9208 atomic/atomicrmw. 9209 - Must happen before 9210 the following 9211 store. 9212 - Ensures that all 9213 memory operations 9214 have 9215 completed before 9216 performing the 9217 store that is being 9218 released. 9219 9220 2. buffer/global/flat_store 9221 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9222 - wavefront - local 9223 - generic 9224 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9225 - generic vmcnt(0) & vscnt(0) 9226 9227 - If CU wavefront execution 9228 mode, omit vmcnt(0) and 9229 vscnt(0). 9230 - If OpenCL, omit lgkmcnt(0). 9231 - Could be split into 9232 separate s_waitcnt 9233 vmcnt(0), s_waitcnt 9234 vscnt(0) and s_waitcnt 9235 lgkmcnt(0) to allow 9236 them to be 9237 independently moved 9238 according to the 9239 following rules. 9240 - s_waitcnt vmcnt(0) 9241 must happen after 9242 any preceding 9243 global/generic load/load 9244 atomic/ 9245 atomicrmw-with-return-value. 9246 - s_waitcnt vscnt(0) 9247 must happen after 9248 any preceding 9249 global/generic 9250 store/store 9251 atomic/ 9252 atomicrmw-no-return-value. 9253 - s_waitcnt lgkmcnt(0) 9254 must happen after 9255 any preceding 9256 local/generic 9257 load/store/load 9258 atomic/store 9259 atomic/atomicrmw. 9260 - Must happen before 9261 the following 9262 atomicrmw. 9263 - Ensures that all 9264 memory operations 9265 have 9266 completed before 9267 performing the 9268 atomicrmw that is 9269 being released. 9270 9271 2. buffer/global/flat_atomic 9272 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9273 9274 - If CU wavefront execution 9275 mode, omit. 9276 - If OpenCL, omit. 9277 - Could be split into 9278 separate s_waitcnt 9279 vmcnt(0) and s_waitcnt 9280 vscnt(0) to allow 9281 them to be 9282 independently moved 9283 according to the 9284 following rules. 9285 - s_waitcnt vmcnt(0) 9286 must happen after 9287 any preceding 9288 global/generic load/load 9289 atomic/ 9290 atomicrmw-with-return-value. 9291 - s_waitcnt vscnt(0) 9292 must happen after 9293 any preceding 9294 global/generic 9295 store/store atomic/ 9296 atomicrmw-no-return-value. 9297 - Must happen before 9298 the following 9299 store. 9300 - Ensures that all 9301 global memory 9302 operations have 9303 completed before 9304 performing the 9305 store that is being 9306 released. 9307 9308 2. ds_atomic 9309 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9310 - system - generic vmcnt(0) & vscnt(0) 9311 9312 - If OpenCL, omit 9313 lgkmcnt(0). 9314 - Could be split into 9315 separate s_waitcnt 9316 vmcnt(0), s_waitcnt 9317 vscnt(0) and s_waitcnt 9318 lgkmcnt(0) to allow 9319 them to be 9320 independently moved 9321 according to the 9322 following rules. 9323 - s_waitcnt vmcnt(0) 9324 must happen after 9325 any preceding 9326 global/generic 9327 load/load atomic/ 9328 atomicrmw-with-return-value. 9329 - s_waitcnt vscnt(0) 9330 must happen after 9331 any preceding 9332 global/generic 9333 store/store atomic/ 9334 atomicrmw-no-return-value. 9335 - s_waitcnt lgkmcnt(0) 9336 must happen after 9337 any preceding 9338 local/generic 9339 load/store/load 9340 atomic/store 9341 atomic/atomicrmw. 9342 - Must happen before 9343 the following 9344 atomicrmw. 9345 - Ensures that all 9346 memory operations 9347 to global and local 9348 have completed 9349 before performing 9350 the atomicrmw that 9351 is being released. 9352 9353 2. buffer/global/flat_atomic 9354 fence release - singlethread *none* *none* 9355 - wavefront 9356 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9357 vmcnt(0) & vscnt(0) 9358 9359 - If CU wavefront execution 9360 mode, omit vmcnt(0) and 9361 vscnt(0). 9362 - If OpenCL and 9363 address space is 9364 not generic, omit 9365 lgkmcnt(0). 9366 - If OpenCL and 9367 address space is 9368 local, omit 9369 vmcnt(0) and vscnt(0). 9370 - However, since LLVM 9371 currently has no 9372 address space on 9373 the fence need to 9374 conservatively 9375 always generate. If 9376 fence had an 9377 address space then 9378 set to address 9379 space of OpenCL 9380 fence flag, or to 9381 generic if both 9382 local and global 9383 flags are 9384 specified. 9385 - Could be split into 9386 separate s_waitcnt 9387 vmcnt(0), s_waitcnt 9388 vscnt(0) and s_waitcnt 9389 lgkmcnt(0) to allow 9390 them to be 9391 independently moved 9392 according to the 9393 following rules. 9394 - s_waitcnt vmcnt(0) 9395 must happen after 9396 any preceding 9397 global/generic 9398 load/load 9399 atomic/ 9400 atomicrmw-with-return-value. 9401 - s_waitcnt vscnt(0) 9402 must happen after 9403 any preceding 9404 global/generic 9405 store/store atomic/ 9406 atomicrmw-no-return-value. 9407 - s_waitcnt lgkmcnt(0) 9408 must happen after 9409 any preceding 9410 local/generic 9411 load/store/load 9412 atomic/store atomic/ 9413 atomicrmw. 9414 - Must happen before 9415 any following store 9416 atomic/atomicrmw 9417 with an equal or 9418 wider sync scope 9419 and memory ordering 9420 stronger than 9421 unordered (this is 9422 termed the 9423 fence-paired-atomic). 9424 - Ensures that all 9425 memory operations 9426 have 9427 completed before 9428 performing the 9429 following 9430 fence-paired-atomic. 9431 9432 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9433 - system vmcnt(0) & vscnt(0) 9434 9435 - If OpenCL and 9436 address space is 9437 not generic, omit 9438 lgkmcnt(0). 9439 - If OpenCL and 9440 address space is 9441 local, omit 9442 vmcnt(0) and vscnt(0). 9443 - However, since LLVM 9444 currently has no 9445 address space on 9446 the fence need to 9447 conservatively 9448 always generate. If 9449 fence had an 9450 address space then 9451 set to address 9452 space of OpenCL 9453 fence flag, or to 9454 generic if both 9455 local and global 9456 flags are 9457 specified. 9458 - Could be split into 9459 separate s_waitcnt 9460 vmcnt(0), s_waitcnt 9461 vscnt(0) and s_waitcnt 9462 lgkmcnt(0) to allow 9463 them to be 9464 independently moved 9465 according to the 9466 following rules. 9467 - s_waitcnt vmcnt(0) 9468 must happen after 9469 any preceding 9470 global/generic 9471 load/load atomic/ 9472 atomicrmw-with-return-value. 9473 - s_waitcnt vscnt(0) 9474 must happen after 9475 any preceding 9476 global/generic 9477 store/store atomic/ 9478 atomicrmw-no-return-value. 9479 - s_waitcnt lgkmcnt(0) 9480 must happen after 9481 any preceding 9482 local/generic 9483 load/store/load 9484 atomic/store 9485 atomic/atomicrmw. 9486 - Must happen before 9487 any following store 9488 atomic/atomicrmw 9489 with an equal or 9490 wider sync scope 9491 and memory ordering 9492 stronger than 9493 unordered (this is 9494 termed the 9495 fence-paired-atomic). 9496 - Ensures that all 9497 memory operations 9498 have 9499 completed before 9500 performing the 9501 following 9502 fence-paired-atomic. 9503 9504 **Acquire-Release Atomic** 9505 ------------------------------------------------------------------------------------ 9506 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9507 - wavefront - local 9508 - generic 9509 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9510 vmcnt(0) & vscnt(0) 9511 9512 - If CU wavefront execution 9513 mode, omit vmcnt(0) and 9514 vscnt(0). 9515 - If OpenCL, omit 9516 lgkmcnt(0). 9517 - Must happen after 9518 any preceding 9519 local/generic 9520 load/store/load 9521 atomic/store 9522 atomic/atomicrmw. 9523 - Could be split into 9524 separate s_waitcnt 9525 vmcnt(0), s_waitcnt 9526 vscnt(0), and s_waitcnt 9527 lgkmcnt(0) to allow 9528 them to be 9529 independently moved 9530 according to the 9531 following rules. 9532 - s_waitcnt vmcnt(0) 9533 must happen after 9534 any preceding 9535 global/generic load/load 9536 atomic/ 9537 atomicrmw-with-return-value. 9538 - s_waitcnt vscnt(0) 9539 must happen after 9540 any preceding 9541 global/generic 9542 store/store 9543 atomic/ 9544 atomicrmw-no-return-value. 9545 - s_waitcnt lgkmcnt(0) 9546 must happen after 9547 any preceding 9548 local/generic 9549 load/store/load 9550 atomic/store 9551 atomic/atomicrmw. 9552 - Must happen before 9553 the following 9554 atomicrmw. 9555 - Ensures that all 9556 memory operations 9557 have 9558 completed before 9559 performing the 9560 atomicrmw that is 9561 being released. 9562 9563 2. buffer/global_atomic 9564 3. s_waitcnt vm/vscnt(0) 9565 9566 - If CU wavefront execution 9567 mode, omit. 9568 - Use vmcnt(0) if atomic with 9569 return and vscnt(0) if 9570 atomic with no-return. 9571 - Must happen before 9572 the following 9573 buffer_gl0_inv. 9574 - Ensures any 9575 following global 9576 data read is no 9577 older than the 9578 atomicrmw value 9579 being acquired. 9580 9581 4. buffer_gl0_inv 9582 9583 - If CU wavefront execution 9584 mode, omit. 9585 - Ensures that 9586 following 9587 loads will not see 9588 stale data. 9589 9590 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9591 9592 - If CU wavefront execution 9593 mode, omit. 9594 - If OpenCL, omit. 9595 - Could be split into 9596 separate s_waitcnt 9597 vmcnt(0) and s_waitcnt 9598 vscnt(0) to allow 9599 them to be 9600 independently moved 9601 according to the 9602 following rules. 9603 - s_waitcnt vmcnt(0) 9604 must happen after 9605 any preceding 9606 global/generic load/load 9607 atomic/ 9608 atomicrmw-with-return-value. 9609 - s_waitcnt vscnt(0) 9610 must happen after 9611 any preceding 9612 global/generic 9613 store/store atomic/ 9614 atomicrmw-no-return-value. 9615 - Must happen before 9616 the following 9617 store. 9618 - Ensures that all 9619 global memory 9620 operations have 9621 completed before 9622 performing the 9623 store that is being 9624 released. 9625 9626 2. ds_atomic 9627 3. s_waitcnt lgkmcnt(0) 9628 9629 - If OpenCL, omit. 9630 - Must happen before 9631 the following 9632 buffer_gl0_inv. 9633 - Ensures any 9634 following global 9635 data read is no 9636 older than the local load 9637 atomic value being 9638 acquired. 9639 9640 4. buffer_gl0_inv 9641 9642 - If CU wavefront execution 9643 mode, omit. 9644 - If OpenCL omit. 9645 - Ensures that 9646 following 9647 loads will not see 9648 stale data. 9649 9650 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9651 vmcnt(0) & vscnt(0) 9652 9653 - If CU wavefront execution 9654 mode, omit vmcnt(0) and 9655 vscnt(0). 9656 - If OpenCL, omit lgkmcnt(0). 9657 - Could be split into 9658 separate s_waitcnt 9659 vmcnt(0), s_waitcnt 9660 vscnt(0) and s_waitcnt 9661 lgkmcnt(0) to allow 9662 them to be 9663 independently moved 9664 according to the 9665 following rules. 9666 - s_waitcnt vmcnt(0) 9667 must happen after 9668 any preceding 9669 global/generic load/load 9670 atomic/ 9671 atomicrmw-with-return-value. 9672 - s_waitcnt vscnt(0) 9673 must happen after 9674 any preceding 9675 global/generic 9676 store/store 9677 atomic/ 9678 atomicrmw-no-return-value. 9679 - s_waitcnt lgkmcnt(0) 9680 must happen after 9681 any preceding 9682 local/generic 9683 load/store/load 9684 atomic/store 9685 atomic/atomicrmw. 9686 - Must happen before 9687 the following 9688 atomicrmw. 9689 - Ensures that all 9690 memory operations 9691 have 9692 completed before 9693 performing the 9694 atomicrmw that is 9695 being released. 9696 9697 2. flat_atomic 9698 3. s_waitcnt lgkmcnt(0) & 9699 vmcnt(0) & vscnt(0) 9700 9701 - If CU wavefront execution 9702 mode, omit vmcnt(0) and 9703 vscnt(0). 9704 - If OpenCL, omit lgkmcnt(0). 9705 - Must happen before 9706 the following 9707 buffer_gl0_inv. 9708 - Ensures any 9709 following global 9710 data read is no 9711 older than the load 9712 atomic value being 9713 acquired. 9714 9715 3. buffer_gl0_inv 9716 9717 - If CU wavefront execution 9718 mode, omit. 9719 - Ensures that 9720 following 9721 loads will not see 9722 stale data. 9723 9724 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9725 - system vmcnt(0) & vscnt(0) 9726 9727 - If OpenCL, omit 9728 lgkmcnt(0). 9729 - Could be split into 9730 separate s_waitcnt 9731 vmcnt(0), s_waitcnt 9732 vscnt(0) and s_waitcnt 9733 lgkmcnt(0) to allow 9734 them to be 9735 independently moved 9736 according to the 9737 following rules. 9738 - s_waitcnt vmcnt(0) 9739 must happen after 9740 any preceding 9741 global/generic 9742 load/load atomic/ 9743 atomicrmw-with-return-value. 9744 - s_waitcnt vscnt(0) 9745 must happen after 9746 any preceding 9747 global/generic 9748 store/store atomic/ 9749 atomicrmw-no-return-value. 9750 - s_waitcnt lgkmcnt(0) 9751 must happen after 9752 any preceding 9753 local/generic 9754 load/store/load 9755 atomic/store 9756 atomic/atomicrmw. 9757 - Must happen before 9758 the following 9759 atomicrmw. 9760 - Ensures that all 9761 memory operations 9762 to global have 9763 completed before 9764 performing the 9765 atomicrmw that is 9766 being released. 9767 9768 2. buffer/global_atomic 9769 3. s_waitcnt vm/vscnt(0) 9770 9771 - Use vmcnt(0) if atomic with 9772 return and vscnt(0) if 9773 atomic with no-return. 9774 - Must happen before 9775 following 9776 buffer_gl*_inv. 9777 - Ensures the 9778 atomicrmw has 9779 completed before 9780 invalidating the 9781 caches. 9782 9783 4. buffer_gl0_inv; 9784 buffer_gl1_inv 9785 9786 - Must happen before 9787 any following 9788 global/generic 9789 load/load 9790 atomic/atomicrmw. 9791 - Ensures that 9792 following loads 9793 will not see stale 9794 global data. 9795 9796 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9797 - system vmcnt(0) & vscnt(0) 9798 9799 - If OpenCL, omit 9800 lgkmcnt(0). 9801 - Could be split into 9802 separate s_waitcnt 9803 vmcnt(0), s_waitcnt 9804 vscnt(0), and s_waitcnt 9805 lgkmcnt(0) to allow 9806 them to be 9807 independently moved 9808 according to the 9809 following rules. 9810 - s_waitcnt vmcnt(0) 9811 must happen after 9812 any preceding 9813 global/generic 9814 load/load atomic 9815 atomicrmw-with-return-value. 9816 - s_waitcnt vscnt(0) 9817 must happen after 9818 any preceding 9819 global/generic 9820 store/store atomic/ 9821 atomicrmw-no-return-value. 9822 - s_waitcnt lgkmcnt(0) 9823 must happen after 9824 any preceding 9825 local/generic 9826 load/store/load 9827 atomic/store 9828 atomic/atomicrmw. 9829 - Must happen before 9830 the following 9831 atomicrmw. 9832 - Ensures that all 9833 memory operations 9834 have 9835 completed before 9836 performing the 9837 atomicrmw that is 9838 being released. 9839 9840 2. flat_atomic 9841 3. s_waitcnt vm/vscnt(0) & 9842 lgkmcnt(0) 9843 9844 - If OpenCL, omit 9845 lgkmcnt(0). 9846 - Use vmcnt(0) if atomic with 9847 return and vscnt(0) if 9848 atomic with no-return. 9849 - Must happen before 9850 following 9851 buffer_gl*_inv. 9852 - Ensures the 9853 atomicrmw has 9854 completed before 9855 invalidating the 9856 caches. 9857 9858 4. buffer_gl0_inv; 9859 buffer_gl1_inv 9860 9861 - Must happen before 9862 any following 9863 global/generic 9864 load/load 9865 atomic/atomicrmw. 9866 - Ensures that 9867 following loads 9868 will not see stale 9869 global data. 9870 9871 fence acq_rel - singlethread *none* *none* 9872 - wavefront 9873 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9874 vmcnt(0) & vscnt(0) 9875 9876 - If CU wavefront execution 9877 mode, omit vmcnt(0) and 9878 vscnt(0). 9879 - If OpenCL and 9880 address space is 9881 not generic, omit 9882 lgkmcnt(0). 9883 - If OpenCL and 9884 address space is 9885 local, omit 9886 vmcnt(0) and vscnt(0). 9887 - However, 9888 since LLVM 9889 currently has no 9890 address space on 9891 the fence need to 9892 conservatively 9893 always generate 9894 (see comment for 9895 previous fence). 9896 - Could be split into 9897 separate s_waitcnt 9898 vmcnt(0), s_waitcnt 9899 vscnt(0) and s_waitcnt 9900 lgkmcnt(0) to allow 9901 them to be 9902 independently moved 9903 according to the 9904 following rules. 9905 - s_waitcnt vmcnt(0) 9906 must happen after 9907 any preceding 9908 global/generic 9909 load/load 9910 atomic/ 9911 atomicrmw-with-return-value. 9912 - s_waitcnt vscnt(0) 9913 must happen after 9914 any preceding 9915 global/generic 9916 store/store atomic/ 9917 atomicrmw-no-return-value. 9918 - s_waitcnt lgkmcnt(0) 9919 must happen after 9920 any preceding 9921 local/generic 9922 load/store/load 9923 atomic/store atomic/ 9924 atomicrmw. 9925 - Must happen before 9926 any following 9927 global/generic 9928 load/load 9929 atomic/store/store 9930 atomic/atomicrmw. 9931 - Ensures that all 9932 memory operations 9933 have 9934 completed before 9935 performing any 9936 following global 9937 memory operations. 9938 - Ensures that the 9939 preceding 9940 local/generic load 9941 atomic/atomicrmw 9942 with an equal or 9943 wider sync scope 9944 and memory ordering 9945 stronger than 9946 unordered (this is 9947 termed the 9948 acquire-fence-paired-atomic) 9949 has completed 9950 before following 9951 global memory 9952 operations. This 9953 satisfies the 9954 requirements of 9955 acquire. 9956 - Ensures that all 9957 previous memory 9958 operations have 9959 completed before a 9960 following 9961 local/generic store 9962 atomic/atomicrmw 9963 with an equal or 9964 wider sync scope 9965 and memory ordering 9966 stronger than 9967 unordered (this is 9968 termed the 9969 release-fence-paired-atomic). 9970 This satisfies the 9971 requirements of 9972 release. 9973 - Must happen before 9974 the following 9975 buffer_gl0_inv. 9976 - Ensures that the 9977 acquire-fence-paired 9978 atomic has completed 9979 before invalidating 9980 the 9981 cache. Therefore 9982 any following 9983 locations read must 9984 be no older than 9985 the value read by 9986 the 9987 acquire-fence-paired-atomic. 9988 9989 3. buffer_gl0_inv 9990 9991 - If CU wavefront execution 9992 mode, omit. 9993 - Ensures that 9994 following 9995 loads will not see 9996 stale data. 9997 9998 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 9999 - system vmcnt(0) & vscnt(0) 10000 10001 - If OpenCL and 10002 address space is 10003 not generic, omit 10004 lgkmcnt(0). 10005 - If OpenCL and 10006 address space is 10007 local, omit 10008 vmcnt(0) and vscnt(0). 10009 - However, since LLVM 10010 currently has no 10011 address space on 10012 the fence need to 10013 conservatively 10014 always generate 10015 (see comment for 10016 previous fence). 10017 - Could be split into 10018 separate s_waitcnt 10019 vmcnt(0), s_waitcnt 10020 vscnt(0) and s_waitcnt 10021 lgkmcnt(0) to allow 10022 them to be 10023 independently moved 10024 according to the 10025 following rules. 10026 - s_waitcnt vmcnt(0) 10027 must happen after 10028 any preceding 10029 global/generic 10030 load/load 10031 atomic/ 10032 atomicrmw-with-return-value. 10033 - s_waitcnt vscnt(0) 10034 must happen after 10035 any preceding 10036 global/generic 10037 store/store atomic/ 10038 atomicrmw-no-return-value. 10039 - s_waitcnt lgkmcnt(0) 10040 must happen after 10041 any preceding 10042 local/generic 10043 load/store/load 10044 atomic/store 10045 atomic/atomicrmw. 10046 - Must happen before 10047 the following 10048 buffer_gl*_inv. 10049 - Ensures that the 10050 preceding 10051 global/local/generic 10052 load 10053 atomic/atomicrmw 10054 with an equal or 10055 wider sync scope 10056 and memory ordering 10057 stronger than 10058 unordered (this is 10059 termed the 10060 acquire-fence-paired-atomic) 10061 has completed 10062 before invalidating 10063 the caches. This 10064 satisfies the 10065 requirements of 10066 acquire. 10067 - Ensures that all 10068 previous memory 10069 operations have 10070 completed before a 10071 following 10072 global/local/generic 10073 store 10074 atomic/atomicrmw 10075 with an equal or 10076 wider sync scope 10077 and memory ordering 10078 stronger than 10079 unordered (this is 10080 termed the 10081 release-fence-paired-atomic). 10082 This satisfies the 10083 requirements of 10084 release. 10085 10086 2. buffer_gl0_inv; 10087 buffer_gl1_inv 10088 10089 - Must happen before 10090 any following 10091 global/generic 10092 load/load 10093 atomic/store/store 10094 atomic/atomicrmw. 10095 - Ensures that 10096 following loads 10097 will not see stale 10098 global data. This 10099 satisfies the 10100 requirements of 10101 acquire. 10102 10103 **Sequential Consistent Atomic** 10104 ------------------------------------------------------------------------------------ 10105 load atomic seq_cst - singlethread - global *Same as corresponding 10106 - wavefront - local load atomic acquire, 10107 - generic except must generated 10108 all instructions even 10109 for OpenCL.* 10110 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10111 - generic vmcnt(0) & vscnt(0) 10112 10113 - If CU wavefront execution 10114 mode, omit vmcnt(0) and 10115 vscnt(0). 10116 - Could be split into 10117 separate s_waitcnt 10118 vmcnt(0), s_waitcnt 10119 vscnt(0), and s_waitcnt 10120 lgkmcnt(0) to allow 10121 them to be 10122 independently moved 10123 according to the 10124 following rules. 10125 - s_waitcnt lgkmcnt(0) must 10126 happen after 10127 preceding 10128 local/generic load 10129 atomic/store 10130 atomic/atomicrmw 10131 with memory 10132 ordering of seq_cst 10133 and with equal or 10134 wider sync scope. 10135 (Note that seq_cst 10136 fences have their 10137 own s_waitcnt 10138 lgkmcnt(0) and so do 10139 not need to be 10140 considered.) 10141 - s_waitcnt vmcnt(0) 10142 must happen after 10143 preceding 10144 global/generic load 10145 atomic/ 10146 atomicrmw-with-return-value 10147 with memory 10148 ordering of seq_cst 10149 and with equal or 10150 wider sync scope. 10151 (Note that seq_cst 10152 fences have their 10153 own s_waitcnt 10154 vmcnt(0) and so do 10155 not need to be 10156 considered.) 10157 - s_waitcnt vscnt(0) 10158 Must happen after 10159 preceding 10160 global/generic store 10161 atomic/ 10162 atomicrmw-no-return-value 10163 with memory 10164 ordering of seq_cst 10165 and with equal or 10166 wider sync scope. 10167 (Note that seq_cst 10168 fences have their 10169 own s_waitcnt 10170 vscnt(0) and so do 10171 not need to be 10172 considered.) 10173 - Ensures any 10174 preceding 10175 sequential 10176 consistent global/local 10177 memory instructions 10178 have completed 10179 before executing 10180 this sequentially 10181 consistent 10182 instruction. This 10183 prevents reordering 10184 a seq_cst store 10185 followed by a 10186 seq_cst load. (Note 10187 that seq_cst is 10188 stronger than 10189 acquire/release as 10190 the reordering of 10191 load acquire 10192 followed by a store 10193 release is 10194 prevented by the 10195 s_waitcnt of 10196 the release, but 10197 there is nothing 10198 preventing a store 10199 release followed by 10200 load acquire from 10201 completing out of 10202 order. The s_waitcnt 10203 could be placed after 10204 seq_store or before 10205 the seq_load. We 10206 choose the load to 10207 make the s_waitcnt be 10208 as late as possible 10209 so that the store 10210 may have already 10211 completed.) 10212 10213 2. *Following 10214 instructions same as 10215 corresponding load 10216 atomic acquire, 10217 except must generated 10218 all instructions even 10219 for OpenCL.* 10220 load atomic seq_cst - workgroup - local 10221 10222 1. s_waitcnt vmcnt(0) & vscnt(0) 10223 10224 - If CU wavefront execution 10225 mode, omit. 10226 - Could be split into 10227 separate s_waitcnt 10228 vmcnt(0) and s_waitcnt 10229 vscnt(0) to allow 10230 them to be 10231 independently moved 10232 according to the 10233 following rules. 10234 - s_waitcnt vmcnt(0) 10235 Must happen after 10236 preceding 10237 global/generic load 10238 atomic/ 10239 atomicrmw-with-return-value 10240 with memory 10241 ordering of seq_cst 10242 and with equal or 10243 wider sync scope. 10244 (Note that seq_cst 10245 fences have their 10246 own s_waitcnt 10247 vmcnt(0) and so do 10248 not need to be 10249 considered.) 10250 - s_waitcnt vscnt(0) 10251 Must happen after 10252 preceding 10253 global/generic store 10254 atomic/ 10255 atomicrmw-no-return-value 10256 with memory 10257 ordering of seq_cst 10258 and with equal or 10259 wider sync scope. 10260 (Note that seq_cst 10261 fences have their 10262 own s_waitcnt 10263 vscnt(0) and so do 10264 not need to be 10265 considered.) 10266 - Ensures any 10267 preceding 10268 sequential 10269 consistent global 10270 memory instructions 10271 have completed 10272 before executing 10273 this sequentially 10274 consistent 10275 instruction. This 10276 prevents reordering 10277 a seq_cst store 10278 followed by a 10279 seq_cst load. (Note 10280 that seq_cst is 10281 stronger than 10282 acquire/release as 10283 the reordering of 10284 load acquire 10285 followed by a store 10286 release is 10287 prevented by the 10288 s_waitcnt of 10289 the release, but 10290 there is nothing 10291 preventing a store 10292 release followed by 10293 load acquire from 10294 completing out of 10295 order. The s_waitcnt 10296 could be placed after 10297 seq_store or before 10298 the seq_load. We 10299 choose the load to 10300 make the s_waitcnt be 10301 as late as possible 10302 so that the store 10303 may have already 10304 completed.) 10305 10306 2. *Following 10307 instructions same as 10308 corresponding load 10309 atomic acquire, 10310 except must generated 10311 all instructions even 10312 for OpenCL.* 10313 10314 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10315 - system - generic vmcnt(0) & vscnt(0) 10316 10317 - Could be split into 10318 separate s_waitcnt 10319 vmcnt(0), s_waitcnt 10320 vscnt(0) and s_waitcnt 10321 lgkmcnt(0) to allow 10322 them to be 10323 independently moved 10324 according to the 10325 following rules. 10326 - s_waitcnt lgkmcnt(0) 10327 must happen after 10328 preceding 10329 local load 10330 atomic/store 10331 atomic/atomicrmw 10332 with memory 10333 ordering of seq_cst 10334 and with equal or 10335 wider sync scope. 10336 (Note that seq_cst 10337 fences have their 10338 own s_waitcnt 10339 lgkmcnt(0) and so do 10340 not need to be 10341 considered.) 10342 - s_waitcnt vmcnt(0) 10343 must happen after 10344 preceding 10345 global/generic load 10346 atomic/ 10347 atomicrmw-with-return-value 10348 with memory 10349 ordering of seq_cst 10350 and with equal or 10351 wider sync scope. 10352 (Note that seq_cst 10353 fences have their 10354 own s_waitcnt 10355 vmcnt(0) and so do 10356 not need to be 10357 considered.) 10358 - s_waitcnt vscnt(0) 10359 Must happen after 10360 preceding 10361 global/generic store 10362 atomic/ 10363 atomicrmw-no-return-value 10364 with memory 10365 ordering of seq_cst 10366 and with equal or 10367 wider sync scope. 10368 (Note that seq_cst 10369 fences have their 10370 own s_waitcnt 10371 vscnt(0) and so do 10372 not need to be 10373 considered.) 10374 - Ensures any 10375 preceding 10376 sequential 10377 consistent global 10378 memory instructions 10379 have completed 10380 before executing 10381 this sequentially 10382 consistent 10383 instruction. This 10384 prevents reordering 10385 a seq_cst store 10386 followed by a 10387 seq_cst load. (Note 10388 that seq_cst is 10389 stronger than 10390 acquire/release as 10391 the reordering of 10392 load acquire 10393 followed by a store 10394 release is 10395 prevented by the 10396 s_waitcnt of 10397 the release, but 10398 there is nothing 10399 preventing a store 10400 release followed by 10401 load acquire from 10402 completing out of 10403 order. The s_waitcnt 10404 could be placed after 10405 seq_store or before 10406 the seq_load. We 10407 choose the load to 10408 make the s_waitcnt be 10409 as late as possible 10410 so that the store 10411 may have already 10412 completed.) 10413 10414 2. *Following 10415 instructions same as 10416 corresponding load 10417 atomic acquire, 10418 except must generated 10419 all instructions even 10420 for OpenCL.* 10421 store atomic seq_cst - singlethread - global *Same as corresponding 10422 - wavefront - local store atomic release, 10423 - workgroup - generic except must generated 10424 - agent all instructions even 10425 - system for OpenCL.* 10426 atomicrmw seq_cst - singlethread - global *Same as corresponding 10427 - wavefront - local atomicrmw acq_rel, 10428 - workgroup - generic except must generated 10429 - agent all instructions even 10430 - system for OpenCL.* 10431 fence seq_cst - singlethread *none* *Same as corresponding 10432 - wavefront fence acq_rel, 10433 - workgroup except must generated 10434 - agent all instructions even 10435 - system for OpenCL.* 10436 ============ ============ ============== ========== ================================ 10437 10438Trap Handler ABI 10439~~~~~~~~~~~~~~~~ 10440 10441For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10442runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10443supports the ``s_trap`` instruction. For usage see: 10444 10445- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10446- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10447- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10448 10449 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10450 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10451 10452 =================== =============== =============== ======================================= 10453 Usage Code Sequence Trap Handler Description 10454 Inputs 10455 =================== =============== =============== ======================================= 10456 reserved ``s_trap 0x00`` Reserved by hardware. 10457 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10458 ``queue_ptr`` intrinsic (not implemented). 10459 ``VGPR0``: 10460 ``arg`` 10461 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10462 ``queue_ptr`` the trap instruction. The associated 10463 queue is signalled to put it into the 10464 error state. When the queue is put in 10465 the error state, the waves executing 10466 dispatches on the queue will be 10467 terminated. 10468 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10469 as a no-operation. The trap handler 10470 is entered and immediately returns to 10471 continue execution of the wavefront. 10472 - If the debugger is enabled, causes 10473 the debug trap to be reported by the 10474 debugger and the wavefront is put in 10475 the halt state with the PC at the 10476 instruction. The debugger must 10477 increment the PC and resume the wave. 10478 reserved ``s_trap 0x04`` Reserved. 10479 reserved ``s_trap 0x05`` Reserved. 10480 reserved ``s_trap 0x06`` Reserved. 10481 reserved ``s_trap 0x07`` Reserved. 10482 reserved ``s_trap 0x08`` Reserved. 10483 reserved ``s_trap 0xfe`` Reserved. 10484 reserved ``s_trap 0xff`` Reserved. 10485 =================== =============== =============== ======================================= 10486 10487.. 10488 10489 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10490 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10491 10492 =================== =============== =============== ======================================= 10493 Usage Code Sequence Trap Handler Description 10494 Inputs 10495 =================== =============== =============== ======================================= 10496 reserved ``s_trap 0x00`` Reserved by hardware. 10497 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10498 breakpoints. Causes wave to be halted 10499 with the PC at the trap instruction. 10500 The debugger is responsible to resume 10501 the wave, including the instruction 10502 that the breakpoint overwrote. 10503 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10504 ``queue_ptr`` the trap instruction. The associated 10505 queue is signalled to put it into the 10506 error state. When the queue is put in 10507 the error state, the waves executing 10508 dispatches on the queue will be 10509 terminated. 10510 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10511 as a no-operation. The trap handler 10512 is entered and immediately returns to 10513 continue execution of the wavefront. 10514 - If the debugger is enabled, causes 10515 the debug trap to be reported by the 10516 debugger and the wavefront is put in 10517 the halt state with the PC at the 10518 instruction. The debugger must 10519 increment the PC and resume the wave. 10520 reserved ``s_trap 0x04`` Reserved. 10521 reserved ``s_trap 0x05`` Reserved. 10522 reserved ``s_trap 0x06`` Reserved. 10523 reserved ``s_trap 0x07`` Reserved. 10524 reserved ``s_trap 0x08`` Reserved. 10525 reserved ``s_trap 0xfe`` Reserved. 10526 reserved ``s_trap 0xff`` Reserved. 10527 =================== =============== =============== ======================================= 10528 10529.. 10530 10531 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10532 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10533 10534 =================== =============== ================ ================= ======================================= 10535 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10536 =================== =============== ================ ================= ======================================= 10537 reserved ``s_trap 0x00`` Reserved by hardware. 10538 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10539 breakpoints. Causes wave to be halted 10540 with the PC at the trap instruction. 10541 The debugger is responsible to resume 10542 the wave, including the instruction 10543 that the breakpoint overwrote. 10544 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10545 ``queue_ptr`` the trap instruction. The associated 10546 queue is signalled to put it into the 10547 error state. When the queue is put in 10548 the error state, the waves executing 10549 dispatches on the queue will be 10550 terminated. 10551 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10552 as a no-operation. The trap handler 10553 is entered and immediately returns to 10554 continue execution of the wavefront. 10555 - If the debugger is enabled, causes 10556 the debug trap to be reported by the 10557 debugger and the wavefront is put in 10558 the halt state with the PC at the 10559 instruction. The debugger must 10560 increment the PC and resume the wave. 10561 reserved ``s_trap 0x04`` Reserved. 10562 reserved ``s_trap 0x05`` Reserved. 10563 reserved ``s_trap 0x06`` Reserved. 10564 reserved ``s_trap 0x07`` Reserved. 10565 reserved ``s_trap 0x08`` Reserved. 10566 reserved ``s_trap 0xfe`` Reserved. 10567 reserved ``s_trap 0xff`` Reserved. 10568 =================== =============== ================ ================= ======================================= 10569 10570.. _amdgpu-amdhsa-function-call-convention: 10571 10572Call Convention 10573~~~~~~~~~~~~~~~ 10574 10575.. note:: 10576 10577 This section is currently incomplete and has inaccuracies. It is WIP that will 10578 be updated as information is determined. 10579 10580See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10581addresses. Unswizzled addresses are normal linear addresses. 10582 10583.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10584 10585Kernel Functions 10586++++++++++++++++ 10587 10588This section describes the call convention ABI for the outer kernel function. 10589 10590See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10591convention. 10592 10593The following is not part of the AMDGPU kernel calling convention but describes 10594how the AMDGPU implements function calls: 10595 105961. Clang decides the kernarg layout to match the *HSA Programmer's Language 10597 Reference* [HSA]_. 10598 10599 - All structs are passed directly. 10600 - Lambda values are passed *TBA*. 10601 10602 .. TODO:: 10603 10604 - Does this really follow HSA rules? Or are structs >16 bytes passed 10605 by-value struct? 10606 - What is ABI for lambda values? 10607 106084. The kernel performs certain setup in its prolog, as described in 10609 :ref:`amdgpu-amdhsa-kernel-prolog`. 10610 10611.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10612 10613Non-Kernel Functions 10614++++++++++++++++++++ 10615 10616This section describes the call convention ABI for functions other than the 10617outer kernel function. 10618 10619If a kernel has function calls then scratch is always allocated and used for 10620the call stack which grows from low address to high address using the swizzled 10621scratch address space. 10622 10623On entry to a function: 10624 106251. SGPR0-3 contain a V# with the following properties (see 10626 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10627 10628 * Base address pointing to the beginning of the wavefront scratch backing 10629 memory. 10630 * Swizzled with dword element size and stride of wavefront size elements. 10631 106322. The FLAT_SCRATCH register pair is setup. See 10633 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 106343. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10635 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 106364. The EXEC register is set to the lanes active on entry to the function. 106375. MODE register: *TBD* 106386. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10639 below. 106407. SGPR30-31 return address (RA). The code address that the function must 10641 return to when it completes. The value is undefined if the function is *no 10642 return*. 106438. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10644 offset relative to the beginning of the wavefront scratch backing memory. 10645 10646 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10647 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10648 manner. 10649 10650 The unswizzled SP value can be converted into the swizzled SP value by: 10651 10652 | swizzled SP = unswizzled SP / wavefront size 10653 10654 This may be used to obtain the private address space address of stack 10655 objects and to convert this address to a flat address by adding the flat 10656 scratch aperture base address. 10657 10658 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10659 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10660 10661 .. note:: 10662 10663 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10664 OpenCL language which has the largest base type defined as 16 bytes. 10665 10666 On entry, the swizzled SP value is the address of the first function 10667 argument passed on the stack. Other stack passed arguments are positive 10668 offsets from the entry swizzled SP value. 10669 10670 The function may use positive offsets beyond the last stack passed argument 10671 for stack allocated local variables and register spill slots. If necessary, 10672 the function may align these to greater alignment than 16 bytes. After these 10673 the function may dynamically allocate space for such things as runtime sized 10674 ``alloca`` local allocations. 10675 10676 If the function calls another function, it will place any stack allocated 10677 arguments after the last local allocation and adjust SGPR32 to the address 10678 after the last local allocation. 10679 106809. All other registers are unspecified. 1068110. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10682 to the function. 10683 10684On exit from a function: 10685 106861. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10687 described below. Any registers used are considered clobbered registers. 106882. The following registers are preserved and have the same value as on entry: 10689 10690 * FLAT_SCRATCH 10691 * EXEC 10692 * GFX6-GFX8: M0 10693 * All SGPR registers except the clobbered registers of SGPR4-31. 10694 * VGPR40-47 10695 * VGPR56-63 10696 * VGPR72-79 10697 * VGPR88-95 10698 * VGPR104-111 10699 * VGPR120-127 10700 * VGPR136-143 10701 * VGPR152-159 10702 * VGPR168-175 10703 * VGPR184-191 10704 * VGPR200-207 10705 * VGPR216-223 10706 * VGPR232-239 10707 * VGPR248-255 10708 10709 .. note:: 10710 10711 Except the argument registers, the VGPRs clobbered and the preserved 10712 registers are intermixed at regular intervals in order to keep a 10713 similar ratio independent of the number of allocated VGPRs. 10714 10715 * Lanes of all VGPRs that are inactive at the call site. 10716 10717 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10718 optimization may mark some of clobbered SGPR and VGPR registers as 10719 preserved if it can be determined that the called function does not change 10720 their value. 10721 107222. The PC is set to the RA provided on entry. 107233. MODE register: *TBD*. 107244. All other registers are clobbered. 107255. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10726 function is available to the caller. 10727 10728.. TODO:: 10729 10730 - On gfx908 are all ACC registers clobbered? 10731 10732 - How are function results returned? The address of structured types is passed 10733 by reference, but what about other types? 10734 10735The function input arguments are made up of the formal arguments explicitly 10736declared by the source language function plus the implicit input arguments used 10737by the implementation. 10738 10739The source language input arguments are: 10740 107411. Any source language implicit ``this`` or ``self`` argument comes first as a 10742 pointer type. 107432. Followed by the function formal arguments in left to right source order. 10744 10745The source language result arguments are: 10746 107471. The function result argument. 10748 10749The source language input or result struct type arguments that are less than or 10750equal to 16 bytes, are decomposed recursively into their base type fields, and 10751each field is passed as if a separate argument. For input arguments, if the 10752called function requires the struct to be in memory, for example because its 10753address is taken, then the function body is responsible for allocating a stack 10754location and copying the field arguments into it. Clang terms this *direct 10755struct*. 10756 10757The source language input struct type arguments that are greater than 16 bytes, 10758are passed by reference. The caller is responsible for allocating a stack 10759location to make a copy of the struct value and pass the address as the input 10760argument. The called function is responsible to perform the dereference when 10761accessing the input argument. Clang terms this *by-value struct*. 10762 10763A source language result struct type argument that is greater than 16 bytes, is 10764returned by reference. The caller is responsible for allocating a stack location 10765to hold the result value and passes the address as the last input argument 10766(before the implicit input arguments). In this case there are no result 10767arguments. The called function is responsible to perform the dereference when 10768storing the result value. Clang terms this *structured return (sret)*. 10769 10770*TODO: correct the ``sret`` definition.* 10771 10772.. TODO:: 10773 10774 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10775 pass as non-decomposed struct as stack argument? Or something else? Is the 10776 memory location in the caller stack frame, or a stack memory argument and so 10777 no address is passed as the caller can directly write to the argument stack 10778 location? But then the stack location is still live after return. If an 10779 argument stack location is it the first stack argument or the last one? 10780 10781Lambda argument types are treated as struct types with an implementation defined 10782set of fields. 10783 10784.. TODO:: 10785 10786 Need to specify the ABI for lambda types for AMDGPU. 10787 10788For AMDGPU backend all source language arguments (including the decomposed 10789struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10790they are passed in SGPRs. 10791 10792The AMDGPU backend walks the function call graph from the leaves to determine 10793which implicit input arguments are used, propagating to each caller of the 10794function. The used implicit arguments are appended to the function arguments 10795after the source language arguments in the following order: 10796 10797.. TODO:: 10798 10799 Is recursion or external functions supported? 10800 108011. Work-Item ID (1 VGPR) 10802 10803 The X, Y and Z work-item ID are packed into a single VGRP with the following 10804 layout. Only fields actually used by the function are set. The other bits 10805 are undefined. 10806 10807 The values come from the initial kernel execution state. See 10808 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10809 10810 .. table:: Work-item implicit argument layout 10811 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10812 10813 ======= ======= ============== 10814 Bits Size Field Name 10815 ======= ======= ============== 10816 9:0 10 bits X Work-Item ID 10817 19:10 10 bits Y Work-Item ID 10818 29:20 10 bits Z Work-Item ID 10819 31:30 2 bits Unused 10820 ======= ======= ============== 10821 108222. Dispatch Ptr (2 SGPRs) 10823 10824 The value comes from the initial kernel execution state. See 10825 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10826 108273. Queue Ptr (2 SGPRs) 10828 10829 The value comes from the initial kernel execution state. See 10830 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10831 108324. Kernarg Segment Ptr (2 SGPRs) 10833 10834 The value comes from the initial kernel execution state. See 10835 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10836 108375. Dispatch id (2 SGPRs) 10838 10839 The value comes from the initial kernel execution state. See 10840 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10841 108426. Work-Group ID X (1 SGPR) 10843 10844 The value comes from the initial kernel execution state. See 10845 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10846 108477. Work-Group ID Y (1 SGPR) 10848 10849 The value comes from the initial kernel execution state. See 10850 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10851 108528. Work-Group ID Z (1 SGPR) 10853 10854 The value comes from the initial kernel execution state. See 10855 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10856 108579. Implicit Argument Ptr (2 SGPRs) 10858 10859 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10860 global address space pointer to the first kernarg implicit argument. 10861 10862The input and result arguments are assigned in order in the following manner: 10863 10864.. note:: 10865 10866 There are likely some errors and omissions in the following description that 10867 need correction. 10868 10869 .. TODO:: 10870 10871 Check the Clang source code to decipher how function arguments and return 10872 results are handled. Also see the AMDGPU specific values used. 10873 10874* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 10875 VGPR31. 10876 10877 If there are more arguments than will fit in these registers, the remaining 10878 arguments are allocated on the stack in order on naturally aligned 10879 addresses. 10880 10881 .. TODO:: 10882 10883 How are overly aligned structures allocated on the stack? 10884 10885* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 10886 SGPR29. 10887 10888 If there are more arguments than will fit in these registers, the remaining 10889 arguments are allocated on the stack in order on naturally aligned 10890 addresses. 10891 10892Note that decomposed struct type arguments may have some fields passed in 10893registers and some in memory. 10894 10895.. TODO:: 10896 10897 So, a struct which can pass some fields as decomposed register arguments, will 10898 pass the rest as decomposed stack elements? But an argument that will not start 10899 in registers will not be decomposed and will be passed as a non-decomposed 10900 stack value? 10901 10902The following is not part of the AMDGPU function calling convention but 10903describes how the AMDGPU implements function calls: 10904 109051. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 10906 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 10907 are used, or for the reasons defined in ``SIFrameLowering``. 109082. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 10909 to access the incoming stack arguments in the function. The BP is needed 10910 only when the function requires the runtime stack alignment. 10911 109123. Allocating SGPR arguments on the stack are not supported. 10913 109144. No CFI is currently generated. See 10915 :ref:`amdgpu-dwarf-call-frame-information`. 10916 10917 .. note:: 10918 10919 CFI will be generated that defines the CFA as the unswizzled address 10920 relative to the wave scratch base in the unswizzled private address space 10921 of the lowest address stack allocated local variable. 10922 10923 ``DW_AT_frame_base`` will be defined as the swizzled address in the 10924 swizzled private address space by dividing the CFA by the wavefront size 10925 (since CFA is always at least dword aligned which matches the scratch 10926 swizzle element size). 10927 10928 If no dynamic stack alignment was performed, the stack allocated arguments 10929 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 10930 local variables and register spill slots are accessed as positive offsets 10931 relative to ``DW_AT_frame_base``. 10932 109335. Function argument passing is implemented by copying the input physical 10934 registers to virtual registers on entry. The register allocator can spill if 10935 necessary. These are copied back to physical registers at call sites. The 10936 net effect is that each function call can have these values in entirely 10937 distinct locations. The IPRA can help avoid shuffling argument registers. 109386. Call sites are implemented by setting up the arguments at positive offsets 10939 from SP. Then SP is incremented to account for the known frame size before 10940 the call and decremented after the call. 10941 10942 .. note:: 10943 10944 The CFI will reflect the changed calculation needed to compute the CFA 10945 from SP. 10946 109477. 4 byte spill slots are used in the stack frame. One slot is allocated for an 10948 emergency spill slot. Buffer instructions are used for stack accesses and 10949 not the ``flat_scratch`` instruction. 10950 10951 .. TODO:: 10952 10953 Explain when the emergency spill slot is used. 10954 10955.. TODO:: 10956 10957 Possible broken issues: 10958 10959 - Stack arguments must be aligned to required alignment. 10960 - Stack is aligned to max(16, max formal argument alignment) 10961 - Direct argument < 64 bits should check register budget. 10962 - Register budget calculation should respect ``inreg`` for SGPR. 10963 - SGPR overflow is not handled. 10964 - struct with 1 member unpeeling is not checking size of member. 10965 - ``sret`` is after ``this`` pointer. 10966 - Caller is not implementing stack realignment: need an extra pointer. 10967 - Should say AMDGPU passes FP rather than SP. 10968 - Should CFI define CFA as address of locals or arguments. Difference is 10969 apparent when have implemented dynamic alignment. 10970 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 10971 highest address of stack frame and use negative offset for locals. Would 10972 allow SP to be the same as FP and could support signal-handler-like as now 10973 have a real SP for the top of the stack. 10974 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 10975 arguments? 10976 10977AMDPAL 10978------ 10979 10980This section provides code conventions used when the target triple OS is 10981``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 10982from the application/runtime to each invocation of a hardware shader. These 10983parameters include both generic, application-controlled parameters called 10984*user data* as well as system-generated parameters that are a product of the 10985draw or dispatch execution. 10986 10987User Data 10988~~~~~~~~~ 10989 10990Each hardware stage has a set of 32-bit *user data registers* which can be 10991written from a command buffer and then loaded into SGPRs when waves are launched 10992via a subsequent dispatch or draw operation. This is the way most arguments are 10993passed from the application/runtime to a hardware shader. 10994 10995Compute User Data 10996~~~~~~~~~~~~~~~~~ 10997 10998Compute shader user data mappings are simpler than graphics shaders and have a 10999fixed mapping. 11000 11001Note that there are always 10 available *user data entries* in registers - 11002entries beyond that limit must be fetched from memory (via the spill table 11003pointer) by the shader. 11004 11005 .. table:: PAL Compute Shader User Data Registers 11006 :name: pal-compute-user-data-registers 11007 11008 ============= ================================ 11009 User Register Description 11010 ============= ================================ 11011 0 Global Internal Table (32-bit pointer) 11012 1 Per-Shader Internal Table (32-bit pointer) 11013 2 - 11 Application-Controlled User Data (10 32-bit values) 11014 12 Spill Table (32-bit pointer) 11015 13 - 14 Thread Group Count (64-bit pointer) 11016 15 GDS Range 11017 ============= ================================ 11018 11019Graphics User Data 11020~~~~~~~~~~~~~~~~~~ 11021 11022Graphics pipelines support a much more flexible user data mapping: 11023 11024 .. table:: PAL Graphics Shader User Data Registers 11025 :name: pal-graphics-user-data-registers 11026 11027 ============= ================================ 11028 User Register Description 11029 ============= ================================ 11030 0 Global Internal Table (32-bit pointer) 11031 + Per-Shader Internal Table (32-bit pointer) 11032 + 1-15 Application Controlled User Data 11033 (1-15 Contiguous 32-bit Values in Registers) 11034 + Spill Table (32-bit pointer) 11035 + Draw Index (First Stage Only) 11036 + Vertex Offset (First Stage Only) 11037 + Instance Offset (First Stage Only) 11038 ============= ================================ 11039 11040 The placement of the global internal table remains fixed in the first *user 11041 data SGPR register*. Otherwise all parameters are optional, and can be mapped 11042 to any desired *user data SGPR register*, with the following restrictions: 11043 11044 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 11045 active hardware stage in a graphics pipeline (i.e. where the API vertex 11046 shader runs). 11047 11048 * Application-controlled user data must be mapped into a contiguous range of 11049 user data registers. 11050 11051 * The application-controlled user data range supports compaction remapping, so 11052 only *entries* that are actually consumed by the shader must be assigned to 11053 corresponding *registers*. Note that in order to support an efficient runtime 11054 implementation, the remapping must pack *registers* in the same order as 11055 *entries*, with unused *entries* removed. 11056 11057.. _pal_global_internal_table: 11058 11059Global Internal Table 11060~~~~~~~~~~~~~~~~~~~~~ 11061 11062The global internal table is a table of *shader resource descriptors* (SRDs) 11063that define how certain engine-wide, runtime-managed resources should be 11064accessed from a shader. The majority of these resources have HW-defined formats, 11065and it is up to the compiler to write/read data as required by the target 11066hardware. 11067 11068The following table illustrates the required format: 11069 11070 .. table:: PAL Global Internal Table 11071 :name: pal-git-table 11072 11073 ============= ================================ 11074 Offset Description 11075 ============= ================================ 11076 0-3 Graphics Scratch SRD 11077 4-7 Compute Scratch SRD 11078 8-11 ES/GS Ring Output SRD 11079 12-15 ES/GS Ring Input SRD 11080 16-19 GS/VS Ring Output #0 11081 20-23 GS/VS Ring Output #1 11082 24-27 GS/VS Ring Output #2 11083 28-31 GS/VS Ring Output #3 11084 32-35 GS/VS Ring Input SRD 11085 36-39 Tessellation Factor Buffer SRD 11086 40-43 Off-Chip LDS Buffer SRD 11087 44-47 Off-Chip Param Cache Buffer SRD 11088 48-51 Sample Position Buffer SRD 11089 52 vaRange::ShadowDescriptorTable High Bits 11090 ============= ================================ 11091 11092 The pointer to the global internal table passed to the shader as user data 11093 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 11094 the top 32 bits of the pipeline, so the shader may use the program 11095 counter's top 32 bits. 11096 11097.. _pal_call-convention: 11098 11099Call Convention 11100~~~~~~~~~~~~~~~ 11101 11102For graphics use cases, the calling convention is `amdgpu_gfx`. 11103 11104.. note:: 11105 11106 `amdgpu_gfx` Function calls are currently in development and are 11107 subject to major changes. 11108 11109This calling convention shares most properties with calling non-kernel 11110functions (see 11111:ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions`). 11112Differences are: 11113 11114 - Currently there are none, differences will be listed here 11115 11116Unspecified OS 11117-------------- 11118 11119This section provides code conventions used when the target triple OS is 11120empty (see :ref:`amdgpu-target-triples`). 11121 11122Trap Handler ABI 11123~~~~~~~~~~~~~~~~ 11124 11125For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11126not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11127instructions are handled as follows: 11128 11129 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11130 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11131 11132 =============== =============== =========================================== 11133 Usage Code Sequence Description 11134 =============== =============== =========================================== 11135 llvm.trap s_endpgm Causes wavefront to be terminated. 11136 llvm.debugtrap *none* Compiler warning given that there is no 11137 trap handler installed. 11138 =============== =============== =========================================== 11139 11140Source Languages 11141================ 11142 11143.. _amdgpu-opencl: 11144 11145OpenCL 11146------ 11147 11148When the language is OpenCL the following differences occur: 11149 111501. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 111512. The AMDGPU backend appends additional arguments to the kernel's explicit 11152 arguments for the AMDHSA OS (see 11153 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 111543. Additional metadata is generated 11155 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11156 11157 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11158 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11159 11160 ======== ==== ========= =========================================== 11161 Position Byte Byte Description 11162 Size Alignment 11163 ======== ==== ========= =========================================== 11164 1 8 8 OpenCL Global Offset X 11165 2 8 8 OpenCL Global Offset Y 11166 3 8 8 OpenCL Global Offset Z 11167 4 8 8 OpenCL address of printf buffer 11168 5 8 8 OpenCL address of virtual queue used by 11169 enqueue_kernel. 11170 6 8 8 OpenCL address of AqlWrap struct used by 11171 enqueue_kernel. 11172 7 8 8 Pointer argument used for Multi-gird 11173 synchronization. 11174 ======== ==== ========= =========================================== 11175 11176.. _amdgpu-hcc: 11177 11178HCC 11179--- 11180 11181When the language is HCC the following differences occur: 11182 111831. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11184 11185.. _amdgpu-assembler: 11186 11187Assembler 11188--------- 11189 11190AMDGPU backend has LLVM-MC based assembler which is currently in development. 11191It supports AMDGCN GFX6-GFX10. 11192 11193This section describes general syntax for instructions and operands. 11194 11195Instructions 11196~~~~~~~~~~~~ 11197 11198An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11199 11200 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11201 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11202 11203:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11204:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11205 11206The order of operands and modifiers is fixed. 11207Most modifiers are optional and may be omitted. 11208 11209Links to detailed instruction syntax description may be found in the following 11210table. Note that features under development are not included 11211in this description. 11212 11213 =================================== ======================================= 11214 Core ISA ISA Extensions 11215 =================================== ======================================= 11216 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11217 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11218 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11219 11220 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11221 11222 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11223 11224 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11225 11226 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11227 11228 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11229 11230 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11231 11232 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11233 =================================== ======================================= 11234 11235For more information about instructions, their semantics and supported 11236combinations of operands, refer to one of instruction set architecture manuals 11237[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_, 11238[AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11239 11240Operands 11241~~~~~~~~ 11242 11243Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11244 11245Modifiers 11246~~~~~~~~~ 11247 11248Detailed description of modifiers may be found 11249:doc:`here<AMDGPUModifierSyntax>`. 11250 11251Instruction Examples 11252~~~~~~~~~~~~~~~~~~~~ 11253 11254DS 11255++ 11256 11257.. code-block:: nasm 11258 11259 ds_add_u32 v2, v4 offset:16 11260 ds_write_src2_b64 v2 offset0:4 offset1:8 11261 ds_cmpst_f32 v2, v4, v6 11262 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11263 11264For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11265Manual. 11266 11267FLAT 11268++++ 11269 11270.. code-block:: nasm 11271 11272 flat_load_dword v1, v[3:4] 11273 flat_store_dwordx3 v[3:4], v[5:7] 11274 flat_atomic_swap v1, v[3:4], v5 glc 11275 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11276 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11277 11278For full list of supported instructions, refer to "FLAT instructions" in ISA 11279Manual. 11280 11281MUBUF 11282+++++ 11283 11284.. code-block:: nasm 11285 11286 buffer_load_dword v1, off, s[4:7], s1 11287 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11288 buffer_store_format_xy v[1:2], off, s[4:7], s1 11289 buffer_wbinvl1 11290 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11291 11292For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11293Manual. 11294 11295SMRD/SMEM 11296+++++++++ 11297 11298.. code-block:: nasm 11299 11300 s_load_dword s1, s[2:3], 0xfc 11301 s_load_dwordx8 s[8:15], s[2:3], s4 11302 s_load_dwordx16 s[88:103], s[2:3], s4 11303 s_dcache_inv_vol 11304 s_memtime s[4:5] 11305 11306For full list of supported instructions, refer to "Scalar Memory Operations" in 11307ISA Manual. 11308 11309SOP1 11310++++ 11311 11312.. code-block:: nasm 11313 11314 s_mov_b32 s1, s2 11315 s_mov_b64 s[0:1], 0x80000000 11316 s_cmov_b32 s1, 200 11317 s_wqm_b64 s[2:3], s[4:5] 11318 s_bcnt0_i32_b64 s1, s[2:3] 11319 s_swappc_b64 s[2:3], s[4:5] 11320 s_cbranch_join s[4:5] 11321 11322For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11323Manual. 11324 11325SOP2 11326++++ 11327 11328.. code-block:: nasm 11329 11330 s_add_u32 s1, s2, s3 11331 s_and_b64 s[2:3], s[4:5], s[6:7] 11332 s_cselect_b32 s1, s2, s3 11333 s_andn2_b32 s2, s4, s6 11334 s_lshr_b64 s[2:3], s[4:5], s6 11335 s_ashr_i32 s2, s4, s6 11336 s_bfm_b64 s[2:3], s4, s6 11337 s_bfe_i64 s[2:3], s[4:5], s6 11338 s_cbranch_g_fork s[4:5], s[6:7] 11339 11340For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11341Manual. 11342 11343SOPC 11344++++ 11345 11346.. code-block:: nasm 11347 11348 s_cmp_eq_i32 s1, s2 11349 s_bitcmp1_b32 s1, s2 11350 s_bitcmp0_b64 s[2:3], s4 11351 s_setvskip s3, s5 11352 11353For full list of supported instructions, refer to "SOPC Instructions" in ISA 11354Manual. 11355 11356SOPP 11357++++ 11358 11359.. code-block:: nasm 11360 11361 s_barrier 11362 s_nop 2 11363 s_endpgm 11364 s_waitcnt 0 ; Wait for all counters to be 0 11365 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11366 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11367 s_sethalt 9 11368 s_sleep 10 11369 s_sendmsg 0x1 11370 s_sendmsg sendmsg(MSG_INTERRUPT) 11371 s_trap 1 11372 11373For full list of supported instructions, refer to "SOPP Instructions" in ISA 11374Manual. 11375 11376Unless otherwise mentioned, little verification is performed on the operands 11377of SOPP Instructions, so it is up to the programmer to be familiar with the 11378range or acceptable values. 11379 11380VALU 11381++++ 11382 11383For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11384the assembler will automatically use optimal encoding based on its operands. To 11385force specific encoding, one can add a suffix to the opcode of the instruction: 11386 11387* _e32 for 32-bit VOP1/VOP2/VOPC 11388* _e64 for 64-bit VOP3 11389* _dpp for VOP_DPP 11390* _sdwa for VOP_SDWA 11391 11392VOP1/VOP2/VOP3/VOPC examples: 11393 11394.. code-block:: nasm 11395 11396 v_mov_b32 v1, v2 11397 v_mov_b32_e32 v1, v2 11398 v_nop 11399 v_cvt_f64_i32_e32 v[1:2], v2 11400 v_floor_f32_e32 v1, v2 11401 v_bfrev_b32_e32 v1, v2 11402 v_add_f32_e32 v1, v2, v3 11403 v_mul_i32_i24_e64 v1, v2, 3 11404 v_mul_i32_i24_e32 v1, -3, v3 11405 v_mul_i32_i24_e32 v1, -100, v3 11406 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11407 v_max_f16_e32 v1, v2, v3 11408 11409VOP_DPP examples: 11410 11411.. code-block:: nasm 11412 11413 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11414 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11415 v_mov_b32 v0, v0 wave_shl:1 11416 v_mov_b32 v0, v0 row_mirror 11417 v_mov_b32 v0, v0 row_bcast:31 11418 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11419 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11420 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11421 11422VOP_SDWA examples: 11423 11424.. code-block:: nasm 11425 11426 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11427 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11428 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11429 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11430 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11431 11432For full list of supported instructions, refer to "Vector ALU instructions". 11433 11434.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11435 11436Code Object V2 Predefined Symbols 11437~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11438 11439.. warning:: 11440 Code object V2 is not the default code object version emitted by 11441 this version of LLVM. 11442 11443The AMDGPU assembler defines and updates some symbols automatically. These 11444symbols do not affect code generation. 11445 11446.option.machine_version_major 11447+++++++++++++++++++++++++++++ 11448 11449Set to the GFX major generation number of the target being assembled for. For 11450example, when assembling for a "GFX9" target this will be set to the integer 11451value "9". The possible GFX major generation numbers are presented in 11452:ref:`amdgpu-processors`. 11453 11454.option.machine_version_minor 11455+++++++++++++++++++++++++++++ 11456 11457Set to the GFX minor generation number of the target being assembled for. For 11458example, when assembling for a "GFX810" target this will be set to the integer 11459value "1". The possible GFX minor generation numbers are presented in 11460:ref:`amdgpu-processors`. 11461 11462.option.machine_version_stepping 11463++++++++++++++++++++++++++++++++ 11464 11465Set to the GFX stepping generation number of the target being assembled for. 11466For example, when assembling for a "GFX704" target this will be set to the 11467integer value "4". The possible GFX stepping generation numbers are presented 11468in :ref:`amdgpu-processors`. 11469 11470.kernel.vgpr_count 11471++++++++++++++++++ 11472 11473Set to zero each time a 11474:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11475encountered. At each instruction, if the current value of this symbol is less 11476than or equal to the maximum VGPR number explicitly referenced within that 11477instruction then the symbol value is updated to equal that VGPR number plus 11478one. 11479 11480.kernel.sgpr_count 11481++++++++++++++++++ 11482 11483Set to zero each time a 11484:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11485encountered. At each instruction, if the current value of this symbol is less 11486than or equal to the maximum VGPR number explicitly referenced within that 11487instruction then the symbol value is updated to equal that SGPR number plus 11488one. 11489 11490.. _amdgpu-amdhsa-assembler-directives-v2: 11491 11492Code Object V2 Directives 11493~~~~~~~~~~~~~~~~~~~~~~~~~ 11494 11495.. warning:: 11496 Code object V2 is not the default code object version emitted by 11497 this version of LLVM. 11498 11499AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11500one can specify them with assembler directives. 11501 11502.hsa_code_object_version major, minor 11503+++++++++++++++++++++++++++++++++++++ 11504 11505*major* and *minor* are integers that specify the version of the HSA code 11506object that will be generated by the assembler. 11507 11508.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11509+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11510 11511 11512*major*, *minor*, and *stepping* are all integers that describe the instruction 11513set architecture (ISA) version of the assembly program. 11514 11515*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11516"AMD" and *arch* should always be equal to "AMDGPU". 11517 11518By default, the assembler will derive the ISA version, *vendor*, and *arch* 11519from the value of the -mcpu option that is passed to the assembler. 11520 11521.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11522 11523.amdgpu_hsa_kernel (name) 11524+++++++++++++++++++++++++ 11525 11526This directives specifies that the symbol with given name is a kernel entry 11527point (label) and the object should contain corresponding symbol of type 11528STT_AMDGPU_HSA_KERNEL. 11529 11530.amd_kernel_code_t 11531++++++++++++++++++ 11532 11533This directive marks the beginning of a list of key / value pairs that are used 11534to specify the amd_kernel_code_t object that will be emitted by the assembler. 11535The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11536amd_kernel_code_t values that are unspecified a default value will be used. The 11537default value for all keys is 0, with the following exceptions: 11538 11539- *amd_code_version_major* defaults to 1. 11540- *amd_kernel_code_version_minor* defaults to 2. 11541- *amd_machine_kind* defaults to 1. 11542- *amd_machine_version_major*, *machine_version_minor*, and 11543 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11544 that is passed to the assembler. 11545- *kernel_code_entry_byte_offset* defaults to 256. 11546- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11547 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11548 Note that wavefront size is specified as a power of two, so a value of **n** 11549 means a size of 2^ **n**. 11550- *call_convention* defaults to -1. 11551- *kernarg_segment_alignment*, *group_segment_alignment*, and 11552 *private_segment_alignment* default to 4. Note that alignments are specified 11553 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11554- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11555 GFX90A onwards. 11556- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11557 GFX10 onwards. 11558- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11559 11560The *.amd_kernel_code_t* directive must be placed immediately after the 11561function label and before any instructions. 11562 11563For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11564comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11565 11566.. _amdgpu-amdhsa-assembler-example-v2: 11567 11568Code Object V2 Example Source Code 11569~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11570 11571.. warning:: 11572 Code Object V2 is not the default code object version emitted by 11573 this version of LLVM. 11574 11575Here is an example of a minimal assembly source file, defining one HSA kernel: 11576 11577.. code:: 11578 :number-lines: 11579 11580 .hsa_code_object_version 1,0 11581 .hsa_code_object_isa 11582 11583 .hsatext 11584 .globl hello_world 11585 .p2align 8 11586 .amdgpu_hsa_kernel hello_world 11587 11588 hello_world: 11589 11590 .amd_kernel_code_t 11591 enable_sgpr_kernarg_segment_ptr = 1 11592 is_ptr64 = 1 11593 compute_pgm_rsrc1_vgprs = 0 11594 compute_pgm_rsrc1_sgprs = 0 11595 compute_pgm_rsrc2_user_sgpr = 2 11596 compute_pgm_rsrc1_wgp_mode = 0 11597 compute_pgm_rsrc1_mem_ordered = 0 11598 compute_pgm_rsrc1_fwd_progress = 1 11599 .end_amd_kernel_code_t 11600 11601 s_load_dwordx2 s[0:1], s[0:1] 0x0 11602 v_mov_b32 v0, 3.14159 11603 s_waitcnt lgkmcnt(0) 11604 v_mov_b32 v1, s0 11605 v_mov_b32 v2, s1 11606 flat_store_dword v[1:2], v0 11607 s_endpgm 11608 .Lfunc_end0: 11609 .size hello_world, .Lfunc_end0-hello_world 11610 11611.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11612 11613Code Object V3 to V4 Predefined Symbols 11614~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11615 11616The AMDGPU assembler defines and updates some symbols automatically. These 11617symbols do not affect code generation. 11618 11619.amdgcn.gfx_generation_number 11620+++++++++++++++++++++++++++++ 11621 11622Set to the GFX major generation number of the target being assembled for. For 11623example, when assembling for a "GFX9" target this will be set to the integer 11624value "9". The possible GFX major generation numbers are presented in 11625:ref:`amdgpu-processors`. 11626 11627.amdgcn.gfx_generation_minor 11628++++++++++++++++++++++++++++ 11629 11630Set to the GFX minor generation number of the target being assembled for. For 11631example, when assembling for a "GFX810" target this will be set to the integer 11632value "1". The possible GFX minor generation numbers are presented in 11633:ref:`amdgpu-processors`. 11634 11635.amdgcn.gfx_generation_stepping 11636+++++++++++++++++++++++++++++++ 11637 11638Set to the GFX stepping generation number of the target being assembled for. 11639For example, when assembling for a "GFX704" target this will be set to the 11640integer value "4". The possible GFX stepping generation numbers are presented 11641in :ref:`amdgpu-processors`. 11642 11643.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 11644 11645.amdgcn.next_free_vgpr 11646++++++++++++++++++++++ 11647 11648Set to zero before assembly begins. At each instruction, if the current value 11649of this symbol is less than or equal to the maximum VGPR number explicitly 11650referenced within that instruction then the symbol value is updated to equal 11651that VGPR number plus one. 11652 11653May be used to set the `.amdhsa_next_free_vgpr` directive in 11654:ref:`amdhsa-kernel-directives-table`. 11655 11656May be set at any time, e.g. manually set to zero at the start of each kernel. 11657 11658.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 11659 11660.amdgcn.next_free_sgpr 11661++++++++++++++++++++++ 11662 11663Set to zero before assembly begins. At each instruction, if the current value 11664of this symbol is less than or equal the maximum SGPR number explicitly 11665referenced within that instruction then the symbol value is updated to equal 11666that SGPR number plus one. 11667 11668May be used to set the `.amdhsa_next_free_spgr` directive in 11669:ref:`amdhsa-kernel-directives-table`. 11670 11671May be set at any time, e.g. manually set to zero at the start of each kernel. 11672 11673.. _amdgpu-amdhsa-assembler-directives-v3-v4: 11674 11675Code Object V3 to V4 Directives 11676~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11677 11678Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 11679architecture processors, and are not OS-specific. Directives which begin with 11680``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 11681``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 11682:ref:`amdgpu-processors`. 11683 11684.. _amdgpu-assembler-directive-amdgcn-target: 11685 11686.amdgcn_target <target-triple> "-" <target-id> 11687++++++++++++++++++++++++++++++++++++++++++++++ 11688 11689Optional directive which declares the ``<target-triple>-<target-id>`` supported 11690by the containing assembler source file. Used by the assembler to validate 11691command-line options such as ``-triple``, ``-mcpu``, and 11692``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 11693:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 11694 11695.. note:: 11696 11697 The target ID syntax used for code object V2 to V3 for this directive differs 11698 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 11699 11700.amdhsa_kernel <name> 11701+++++++++++++++++++++ 11702 11703Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 11704``<name>.kd``, in the current location of the current section. Only valid when 11705the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 11706instruction to execute, and does not need to be previously defined. 11707 11708Marks the beginning of a list of directives used to generate the bytes of a 11709kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 11710Directives which may appear in this list are described in 11711:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 11712be valid for the target being assembled for, and cannot be repeated. Directives 11713support the range of values specified by the field they reference in 11714:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 11715assumed to have its default value, unless it is marked as "Required", in which 11716case it is an error to omit the directive. This list of directives is 11717terminated by an ``.end_amdhsa_kernel`` directive. 11718 11719 .. table:: AMDHSA Kernel Assembler Directives 11720 :name: amdhsa-kernel-directives-table 11721 11722 ======================================================== =================== ============ =================== 11723 Directive Default Supported On Description 11724 ======================================================== =================== ============ =================== 11725 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 11726 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11727 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 11728 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11729 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 11730 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11731 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 11732 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11733 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 11734 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11735 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 11736 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11737 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 11738 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11739 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 11740 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11741 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 11742 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11743 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 11744 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11745 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 11746 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11747 Specific 11748 (wavefrontsize64) 11749 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 11750 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11751 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 11752 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11753 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 11754 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11755 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 11756 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11757 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 11758 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11759 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 11760 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11761 Possible values are defined in 11762 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 11763 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 11764 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 11765 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11766 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 11767 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11768 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11769 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 11770 Used to calculate ACCUM_OFFSET in 11771 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11772 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 11773 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11774 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11775 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 11776 scratch memory. Used to calculate 11777 GRANULATED_WAVEFRONT_SGPR_COUNT in 11778 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11779 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 11780 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11781 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11782 (xnack) 11783 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 11784 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11785 Possible values are defined in 11786 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11787 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 11788 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11789 Possible values are defined in 11790 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11791 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 11792 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11793 Possible values are defined in 11794 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11795 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 11796 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11797 Possible values are defined in 11798 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11799 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 11800 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11801 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 11802 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11803 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 11804 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11805 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 11806 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11807 Specific 11808 (tgsplit) 11809 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 11810 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11811 Specific 11812 (cumode) 11813 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 11814 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11815 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 11816 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11817 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 11818 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11819 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 11820 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11821 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 11822 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11823 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 11824 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11825 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 11826 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11827 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 11828 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11829 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 11830 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11831 ======================================================== =================== ============ =================== 11832 11833.amdgpu_metadata 11834++++++++++++++++ 11835 11836Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 11837note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 11838 11839The contents must be in the [YAML]_ markup format, with the same structure and 11840semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 11841:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 11842 11843This directive is terminated by an ``.end_amdgpu_metadata`` directive. 11844 11845.. _amdgpu-amdhsa-assembler-example-v3-v4: 11846 11847Code Object V3 to V4 Example Source Code 11848~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11849 11850Here is an example of a minimal assembly source file, defining one HSA kernel: 11851 11852.. code:: 11853 :number-lines: 11854 11855 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 11856 11857 .text 11858 .globl hello_world 11859 .p2align 8 11860 .type hello_world,@function 11861 hello_world: 11862 s_load_dwordx2 s[0:1], s[0:1] 0x0 11863 v_mov_b32 v0, 3.14159 11864 s_waitcnt lgkmcnt(0) 11865 v_mov_b32 v1, s0 11866 v_mov_b32 v2, s1 11867 flat_store_dword v[1:2], v0 11868 s_endpgm 11869 .Lfunc_end0: 11870 .size hello_world, .Lfunc_end0-hello_world 11871 11872 .rodata 11873 .p2align 6 11874 .amdhsa_kernel hello_world 11875 .amdhsa_user_sgpr_kernarg_segment_ptr 1 11876 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11877 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11878 .end_amdhsa_kernel 11879 11880 .amdgpu_metadata 11881 --- 11882 amdhsa.version: 11883 - 1 11884 - 0 11885 amdhsa.kernels: 11886 - .name: hello_world 11887 .symbol: hello_world.kd 11888 .kernarg_segment_size: 48 11889 .group_segment_fixed_size: 0 11890 .private_segment_fixed_size: 0 11891 .kernarg_segment_align: 4 11892 .wavefront_size: 64 11893 .sgpr_count: 2 11894 .vgpr_count: 3 11895 .max_flat_workgroup_size: 256 11896 ... 11897 .end_amdgpu_metadata 11898 11899If an assembly source file contains multiple kernels and/or functions, the 11900:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 11901:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 11902the ``.set <symbol>, <expression>`` directive. For example, in the case of two 11903kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 11904to group the function with the kernel that calls it and reset the symbols 11905between the two connected components: 11906 11907.. code:: 11908 :number-lines: 11909 11910 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 11911 11912 // gpr tracking symbols are implicitly set to zero 11913 11914 .text 11915 .globl kern0 11916 .p2align 8 11917 .type kern0,@function 11918 kern0: 11919 // ... 11920 s_endpgm 11921 .Lkern0_end: 11922 .size kern0, .Lkern0_end-kern0 11923 11924 .rodata 11925 .p2align 6 11926 .amdhsa_kernel kern0 11927 // ... 11928 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11929 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11930 .end_amdhsa_kernel 11931 11932 // reset symbols to begin tracking usage in func1 and kern1 11933 .set .amdgcn.next_free_vgpr, 0 11934 .set .amdgcn.next_free_sgpr, 0 11935 11936 .text 11937 .hidden func1 11938 .global func1 11939 .p2align 2 11940 .type func1,@function 11941 func1: 11942 // ... 11943 s_setpc_b64 s[30:31] 11944 .Lfunc1_end: 11945 .size func1, .Lfunc1_end-func1 11946 11947 .globl kern1 11948 .p2align 8 11949 .type kern1,@function 11950 kern1: 11951 // ... 11952 s_getpc_b64 s[4:5] 11953 s_add_u32 s4, s4, func1@rel32@lo+4 11954 s_addc_u32 s5, s5, func1@rel32@lo+4 11955 s_swappc_b64 s[30:31], s[4:5] 11956 // ... 11957 s_endpgm 11958 .Lkern1_end: 11959 .size kern1, .Lkern1_end-kern1 11960 11961 .rodata 11962 .p2align 6 11963 .amdhsa_kernel kern1 11964 // ... 11965 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11966 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11967 .end_amdhsa_kernel 11968 11969These symbols cannot identify connected components in order to automatically 11970track the usage for each kernel. However, in some cases careful organization of 11971the kernels and functions in the source file means there is minimal additional 11972effort required to accurately calculate GPR usage. 11973 11974Additional Documentation 11975======================== 11976 11977.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 11978.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 11979.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 11980.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 11981.. [AMD-GCN-GFX10-RDNA1] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 11982.. [AMD-GCN-GFX10-RDNA2] `AMD "RDNA 2" Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 11983.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 11984.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 11985.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 11986.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 11987.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 11988.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 11989.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 11990.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 11991.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 11992.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 11993.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 11994.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 11995.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 11996.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 11997.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 11998.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 11999