1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 42to specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as: 73 74 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 75 loader on Linux. See *AMD ROCm Platform Release Notes* 76 [AMD-ROCm-Release-Notes]_ for supported hardware and 77 software. 78 - AMD's PAL runtime using the *pal-amdhsa* loader on 79 Windows. 80 81 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 82 runtime using the *pal-amdpal* loader on Windows and Linux 83 Pro. 84 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 85 3D runtime using the *mesa-mesa3d* loader on Linux. 86 ============== ============================================================ 87 88 .. table:: AMDGPU Environments 89 :name: amdgpu-environment-table 90 91 ============ ============================================================== 92 Environment Description 93 ============ ============================================================== 94 *<empty>* Default. 95 ============ ============================================================== 96 97.. _amdgpu-processors: 98 99Processors 100---------- 101 102Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 103specify the AMDGPU processor together with optional target features. See 104:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 105specific information. 106 107Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 108 109* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 110 111 112 .. table:: AMDGPU Processors 113 :name: amdgpu-processor-table 114 115 =========== =============== ============ ===== ================= =============== =============== ====================== 116 Processor Alternative Target dGPU/ Target Target OS Support Example 117 Processor Triple APU Features Properties *(see* Products 118 Architecture Supported `amdgpu-os`_ 119 *and 120 corresponding 121 runtime release 122 notes for 123 current 124 information and 125 level of 126 support)* 127 =========== =============== ============ ===== ================= =============== =============== ====================== 128 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 129 ----------------------------------------------------------------------------------------------------------------------- 130 ``r600`` ``r600`` dGPU - Does not 131 support 132 generic 133 address 134 space 135 ``r630`` ``r600`` dGPU - Does not 136 support 137 generic 138 address 139 space 140 ``rs880`` ``r600`` dGPU - Does not 141 support 142 generic 143 address 144 space 145 ``rv670`` ``r600`` dGPU - Does not 146 support 147 generic 148 address 149 space 150 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 151 ----------------------------------------------------------------------------------------------------------------------- 152 ``rv710`` ``r600`` dGPU - Does not 153 support 154 generic 155 address 156 space 157 ``rv730`` ``r600`` dGPU - Does not 158 support 159 generic 160 address 161 space 162 ``rv770`` ``r600`` dGPU - Does not 163 support 164 generic 165 address 166 space 167 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 168 ----------------------------------------------------------------------------------------------------------------------- 169 ``cedar`` ``r600`` dGPU - Does not 170 support 171 generic 172 address 173 space 174 ``cypress`` ``r600`` dGPU - Does not 175 support 176 generic 177 address 178 space 179 ``juniper`` ``r600`` dGPU - Does not 180 support 181 generic 182 address 183 space 184 ``redwood`` ``r600`` dGPU - Does not 185 support 186 generic 187 address 188 space 189 ``sumo`` ``r600`` dGPU - Does not 190 support 191 generic 192 address 193 space 194 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 195 ----------------------------------------------------------------------------------------------------------------------- 196 ``barts`` ``r600`` dGPU - Does not 197 support 198 generic 199 address 200 space 201 ``caicos`` ``r600`` dGPU - Does not 202 support 203 generic 204 address 205 space 206 ``cayman`` ``r600`` dGPU - Does not 207 support 208 generic 209 address 210 space 211 ``turks`` ``r600`` dGPU - Does not 212 support 213 generic 214 address 215 space 216 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 217 ----------------------------------------------------------------------------------------------------------------------- 218 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 219 support 220 generic 221 address 222 space 223 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 224 - ``verde`` support 225 generic 226 address 227 space 228 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 229 - ``oland`` support 230 generic 231 address 232 space 233 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 234 ----------------------------------------------------------------------------------------------------------------------- 235 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 236 flat - *pal-amdhsa* - A6 Pro-7050B 237 scratch - *pal-amdpal* - A8-7100 238 - A8 Pro-7150B 239 - A10-7300 240 - A10 Pro-7350B 241 - FX-7500 242 - A8-7200P 243 - A10-7400P 244 - FX-7600P 245 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 246 flat - *pal-amdhsa* - FirePro W9100 247 scratch - *pal-amdpal* - FirePro S9150 248 - FirePro S9170 249 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 250 flat - *pal-amdhsa* - Radeon R9 290x 251 scratch - *pal-amdpal* - Radeon R390 252 - Radeon R390x 253 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 254 - ``mullins`` flat - *pal-amdpal* - E1-2200 255 scratch - E1-2500 256 - E2-3000 257 - E2-3800 258 - A4-5000 259 - A4-5100 260 - A6-5200 261 - A4 Pro-3340B 262 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 263 flat - *pal-amdpal* - Radeon HD 8770 264 scratch - R7 260 265 - R7 260X 266 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 267 flat - *pal-amdpal* 268 scratch .. TODO:: 269 270 Add product 271 names. 272 273 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 274 ----------------------------------------------------------------------------------------------------------------------- 275 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 276 flat - *pal-amdhsa* - Pro A6-8500B 277 scratch - *pal-amdpal* - A8-8600P 278 - Pro A8-8600B 279 - FX-8800P 280 - Pro A12-8800B 281 - A10-8700P 282 - Pro A10-8700B 283 - A10-8780P 284 - A10-9600P 285 - A10-9630P 286 - A12-9700P 287 - A12-9730P 288 - FX-9800P 289 - FX-9830P 290 - E2-9010 291 - A6-9210 292 - A9-9410 293 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 294 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 295 scratch - *pal-amdpal* - Radeon R9 385 296 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 297 - *pal-amdhsa* - Radeon R9 Fury 298 - *pal-amdpal* - Radeon R9 FuryX 299 - Radeon Pro Duo 300 - FirePro S9300x2 301 - Radeon Instinct MI8 302 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 303 flat - *pal-amdhsa* - Radeon RX 480 304 scratch - *pal-amdpal* - Radeon Instinct MI6 305 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 306 flat - *pal-amdhsa* 307 scratch - *pal-amdpal* 308 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 309 flat - *pal-amdhsa* - FirePro S7100 310 scratch - *pal-amdpal* - FirePro W7100 311 - Mobile FirePro 312 M7170 313 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 314 flat - *pal-amdhsa* 315 scratch - *pal-amdpal* .. TODO:: 316 317 Add product 318 names. 319 320 **GCN GFX9 (Vega)** [AMD-GCN-GFX9]_ 321 ----------------------------------------------------------------------------------------------------------------------- 322 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 323 flat - *pal-amdhsa* Frontier Edition 324 scratch - *pal-amdpal* - Radeon RX Vega 56 325 - Radeon RX Vega 64 326 - Radeon RX Vega 64 327 Liquid 328 - Radeon Instinct MI25 329 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 330 flat - *pal-amdhsa* - Ryzen 5 2400G 331 scratch - *pal-amdpal* 332 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 333 - *pal-amdhsa* 334 - *pal-amdpal* .. TODO:: 335 336 Add product 337 names. 338 339 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 340 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 341 scratch - *pal-amdpal* - Radeon VII 342 - Radeon Pro VII 343 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* *TBA* 344 - xnack - Absolute 345 flat .. TODO:: 346 scratch 347 Add product 348 names. 349 350 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 351 flat 352 scratch .. TODO:: 353 354 Add product 355 names. 356 357 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 358 - tgsplit flat 359 - xnack scratch .. TODO:: 360 - Packed 361 work-item Add product 362 IDs names. 363 364 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 365 flat - Ryzen 7 4700GE 366 scratch - Ryzen 5 4600G 367 - Ryzen 5 4600GE 368 - Ryzen 3 4300G 369 - Ryzen 3 4300GE 370 - Ryzen Pro 4000G 371 - Ryzen 7 Pro 4700G 372 - Ryzen 7 Pro 4750GE 373 - Ryzen 5 Pro 4650G 374 - Ryzen 5 Pro 4650GE 375 - Ryzen 3 Pro 4350G 376 - Ryzen 3 Pro 4350GE 377 378 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 379 ----------------------------------------------------------------------------------------------------------------------- 380 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 381 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 382 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 383 - Radeon Pro 5600M 384 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* *TBA* 385 - wavefrontsize64 - Absolute - *pal-amdhsa* 386 - xnack flat - *pal-amdpal* 387 scratch .. TODO:: 388 389 Add product 390 names. 391 392 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 393 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 394 - xnack scratch - *pal-amdpal* 395 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 396 ----------------------------------------------------------------------------------------------------------------------- 397 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 398 - wavefrontsize64 flat - *pal-amdhsa* 399 scratch - *pal-amdpal* .. TODO:: 400 401 Add product 402 names. 403 404 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 405 - wavefrontsize64 flat - *pal-amdhsa* 406 scratch - *pal-amdpal* .. TODO:: 407 408 Add product 409 names. 410 411 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 412 - wavefrontsize64 flat - *pal-amdhsa* 413 scratch - *pal-amdpal* .. TODO:: 414 415 Add product 416 names. 417 418 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 419 - wavefrontsize64 flat 420 scratch .. TODO:: 421 422 Add product 423 names. 424 425 =========== =============== ============ ===== ================= =============== =============== ====================== 426 427.. _amdgpu-target-features: 428 429Target Features 430--------------- 431 432Target features control how code is generated to support certain 433processor specific features. Not all target features are supported by 434all processors. The runtime must ensure that the features supported by 435the device used to execute the code match the features enabled when 436generating the code. A mismatch of features may result in incorrect 437execution, or a reduction in performance. 438 439The target features supported by each processor is listed in 440:ref:`amdgpu-processor-table`. 441 442Target features are controlled by exactly one of the following Clang 443options: 444 445``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 446 447 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 448 optional components of the target ID. If omitted, the target feature has the 449 ``any`` value. See :ref:`amdgpu-target-id`. 450 451``-m[no-]<target-feature>`` 452 453 Target features not specified by the target ID are specified using a 454 separate option. These target features can have an ``on`` or ``off`` 455 value. ``on`` is specified by omitting the ``no-`` prefix, and 456 ``off`` is specified by including the ``no-`` prefix. The default 457 if not specified is ``off``. 458 459For example: 460 461``-mcpu=gfx908:xnack+`` 462 Enable the ``xnack`` feature. 463``-mcpu=gfx908:xnack-`` 464 Disable the ``xnack`` feature. 465``-mcumode`` 466 Enable the ``cumode`` feature. 467``-mno-cumode`` 468 Disable the ``cumode`` feature. 469 470 .. table:: AMDGPU Target Features 471 :name: amdgpu-target-features-table 472 473 =============== ============================ ================================================== 474 Target Feature Clang Option to Control Description 475 Name 476 =============== ============================ ================================================== 477 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 478 when generating code for kernels. When disabled 479 native WGP wavefront execution mode is used, 480 when enabled CU wavefront execution mode is used 481 (see :ref:`amdgpu-amdhsa-memory-model`). 482 483 sramecc - ``-mcpu`` If specified, generate code that can only be 484 - ``--offload-arch`` loaded and executed in a process that has a 485 matching setting for SRAMECC. 486 487 If not specified, generate code that can be 488 loaded and executed in a process with either 489 setting of SRAMECC. 490 491 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 492 work-groups are launched in threadgroup split mode. 493 When enabled the waves of a work-group may be 494 launched in different CUs. 495 496 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 497 generating code for kernels. When disabled 498 native wavefront size 32 is used, when enabled 499 wavefront size 64 is used. 500 501 xnack - ``-mcpu`` If specified, generate code that can only be 502 - ``--offload-arch`` loaded and executed in a process that has a 503 matching setting for XNACK replay. 504 505 If not specified, generate code that can be 506 loaded and executed in a process with either 507 setting of XNACK replay. 508 509 This is used for demand paging and page 510 migration. If XNACK replay is enabled in 511 the device, then if a page fault occurs 512 the code may execute incorrectly if the 513 ``xnack`` feature is not enabled. Executing 514 code that has the feature enabled on a 515 device that does not have XNACK replay 516 enabled will execute correctly but may 517 be less performant than code with the 518 feature disabled. 519 =============== ============================ ================================================== 520 521.. _amdgpu-target-id: 522 523Target ID 524--------- 525 526AMDGPU supports target IDs. See `Clang Offload Bundler 527<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 528description. The AMDGPU target specific information is: 529 530**processor** 531 Is a AMDGPU processor or alternative processor name specified in 532 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 533 the primary processor and alternative processor names. The canonical form 534 target ID only allow the primary processor name. 535 536**target-feature** 537 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 538 is supported by the processor. The target features supported by each processor 539 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 540 a target ID are marked as being controlled by ``-mcpu`` and 541 ``--offload-arch``. Each target feature must appear at most once in a target 542 ID. The non-canonical form target ID allows the target features to be 543 specified in any order. The canonical form target ID requires the target 544 features to be specified in alphabetic order. 545 546.. _amdgpu-target-id-v2-v3: 547 548Code Object V2 to V3 Target ID 549~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 550 551The target ID syntax for code object V2 to V3 is the same as defined in `Clang 552Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 553when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 554directive and the bundle entry ID. In those cases it has the following BNF 555syntax: 556 557.. code:: 558 559 <target-id> ::== <processor> ( "+" <target-feature> )* 560 561Where a target feature is omitted if *Off* and present if *On* or *Any*. 562 563.. note:: 564 565 The code object V2 to V3 cannot represent *Any* and treats it the same as 566 *On*. 567 568.. _amdgpu-embedding-bundled-objects: 569 570Embedding Bundled Code Objects 571------------------------------ 572 573AMDGPU supports the HIP and OpenMP languages that perform code object embedding 574as described in `Clang Offload Bundler 575<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 576 577.. note:: 578 579 The target ID syntax used for code object V2 to V3 for a bundle entry ID 580 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 581 582.. _amdgpu-address-spaces: 583 584Address Spaces 585-------------- 586 587The AMDGPU architecture supports a number of memory address spaces. The address 588space names use the OpenCL standard names, with some additions. 589 590The AMDGPU address spaces correspond to target architecture specific LLVM 591address space numbers used in LLVM IR. 592 593The AMDGPU address spaces are described in 594:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 595supported for the ``amdgcn`` target. 596 597 .. table:: AMDGPU Address Spaces 598 :name: amdgpu-address-spaces-table 599 600 ================================= =============== =========== ================ ======= ============================ 601 .. 64-Bit Process Address Space 602 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 603 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 604 Space Number Name Name Size 605 ================================= =============== =========== ================ ======= ============================ 606 Generic 0 flat flat 64 0x0000000000000000 607 Global 1 global global 64 0x0000000000000000 608 Region 2 N/A GDS 32 *not implemented for AMDHSA* 609 Local 3 group LDS 32 0xFFFFFFFF 610 Constant 4 constant *same as global* 64 0x0000000000000000 611 Private 5 private scratch 32 0xFFFFFFFF 612 Constant 32-bit 6 *TODO* 0x00000000 613 Buffer Fat Pointer (experimental) 7 *TODO* 614 ================================= =============== =========== ================ ======= ============================ 615 616**Generic** 617 The generic address space is supported unless the *Target Properties* column 618 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 619 space*. 620 621 The generic address space uses the hardware flat address support for two fixed 622 ranges of virtual addresses (the private and local apertures), that are 623 outside the range of addressable global memory, to map from a flat address to 624 a private or local address. This uses FLAT instructions that can take a flat 625 address and access global, private (scratch), and group (LDS) memory depending 626 on if the address is within one of the aperture ranges. 627 628 Flat access to scratch requires hardware aperture setup and setup in the 629 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 630 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 631 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 632 633 To convert between a private or group address space address (termed a segment 634 address) and a flat address the base address of the corresponding aperture 635 can be used. For GFX7-GFX8 these are available in the 636 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 637 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 638 GFX9-GFX10 the aperture base addresses are directly available as inline 639 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 640 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 641 aligned to 2^32 which makes it easier to convert from flat to segment or 642 segment to flat. 643 644 A global address space address has the same value when used as a flat address 645 so no conversion is needed. 646 647**Global and Constant** 648 The global and constant address spaces both use global virtual addresses, 649 which are the same virtual address space used by the CPU. However, some 650 virtual addresses may only be accessible to the CPU, some only accessible 651 by the GPU, and some by both. 652 653 Using the constant address space indicates that the data will not change 654 during the execution of the kernel. This allows scalar read instructions to 655 be used. As the constant address space could only be modified on the host 656 side, a generic pointer loaded from the constant address space is safe to be 657 assumed as a global pointer since only the device global memory is visible 658 and managed on the host side. The vector and scalar L1 caches are invalidated 659 of volatile data before each kernel dispatch execution to allow constant 660 memory to change values between kernel dispatches. 661 662**Region** 663 The region address space uses the hardware Global Data Store (GDS). All 664 wavefronts executing on the same device will access the same memory for any 665 given region address. However, the same region address accessed by wavefronts 666 executing on different devices will access different memory. It is higher 667 performance than global memory. It is allocated by the runtime. The data 668 store (DS) instructions can be used to access it. 669 670**Local** 671 The local address space uses the hardware Local Data Store (LDS) which is 672 automatically allocated when the hardware creates the wavefronts of a 673 work-group, and freed when all the wavefronts of a work-group have 674 terminated. All wavefronts belonging to the same work-group will access the 675 same memory for any given local address. However, the same local address 676 accessed by wavefronts belonging to different work-groups will access 677 different memory. It is higher performance than global memory. The data store 678 (DS) instructions can be used to access it. 679 680**Private** 681 The private address space uses the hardware scratch memory support which 682 automatically allocates memory when it creates a wavefront and frees it when 683 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 684 given private address will be different to the memory accessed by another lane 685 of the same or different wavefront for the same private address. 686 687 If a kernel dispatch uses scratch, then the hardware allocates memory from a 688 pool of backing memory allocated by the runtime for each wavefront. The lanes 689 of the wavefront access this using dword (4 byte) interleaving. The mapping 690 used from private address to backing memory address is: 691 692 ``wavefront-scratch-base + 693 ((private-address / 4) * wavefront-size * 4) + 694 (wavefront-lane-id * 4) + (private-address % 4)`` 695 696 If each lane of a wavefront accesses the same private address, the 697 interleaving results in adjacent dwords being accessed and hence requires 698 fewer cache lines to be fetched. 699 700 There are different ways that the wavefront scratch base address is 701 determined by a wavefront (see 702 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 703 704 Scratch memory can be accessed in an interleaved manner using buffer 705 instructions with the scratch buffer descriptor and per wavefront scratch 706 offset, by the scratch instructions, or by flat instructions. Multi-dword 707 access is not supported except by flat and scratch instructions in 708 GFX9-GFX10. 709 710**Constant 32-bit** 711 *TODO* 712 713**Buffer Fat Pointer** 714 The buffer fat pointer is an experimental address space that is currently 715 unsupported in the backend. It exposes a non-integral pointer that is in 716 the future intended to support the modelling of 128-bit buffer descriptors 717 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 718 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 719 model the buffer descriptors used heavily in graphics workloads targeting 720 the backend. 721 722.. _amdgpu-memory-scopes: 723 724Memory Scopes 725------------- 726 727This section provides LLVM memory synchronization scopes supported by the AMDGPU 728backend memory model when the target triple OS is ``amdhsa`` (see 729:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 730 731The memory model supported is based on the HSA memory model [HSA]_ which is 732based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 733relation is transitive over the synchronizes-with relation independent of scope 734and synchronizes-with allows the memory scope instances to be inclusive (see 735table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 736 737This is different to the OpenCL [OpenCL]_ memory model which does not have scope 738inclusion and requires the memory scopes to exactly match. However, this 739is conservatively correct for OpenCL. 740 741 .. table:: AMDHSA LLVM Sync Scopes 742 :name: amdgpu-amdhsa-llvm-sync-scopes-table 743 744 ======================= =================================================== 745 LLVM Sync Scope Description 746 ======================= =================================================== 747 *none* The default: ``system``. 748 749 Synchronizes with, and participates in modification 750 and seq_cst total orderings with, other operations 751 (except image operations) for all address spaces 752 (except private, or generic that accesses private) 753 provided the other operation's sync scope is: 754 755 - ``system``. 756 - ``agent`` and executed by a thread on the same 757 agent. 758 - ``workgroup`` and executed by a thread in the 759 same work-group. 760 - ``wavefront`` and executed by a thread in the 761 same wavefront. 762 763 ``agent`` Synchronizes with, and participates in modification 764 and seq_cst total orderings with, other operations 765 (except image operations) for all address spaces 766 (except private, or generic that accesses private) 767 provided the other operation's sync scope is: 768 769 - ``system`` or ``agent`` and executed by a thread 770 on the same agent. 771 - ``workgroup`` and executed by a thread in the 772 same work-group. 773 - ``wavefront`` and executed by a thread in the 774 same wavefront. 775 776 ``workgroup`` Synchronizes with, and participates in modification 777 and seq_cst total orderings with, other operations 778 (except image operations) for all address spaces 779 (except private, or generic that accesses private) 780 provided the other operation's sync scope is: 781 782 - ``system``, ``agent`` or ``workgroup`` and 783 executed by a thread in the same work-group. 784 - ``wavefront`` and executed by a thread in the 785 same wavefront. 786 787 ``wavefront`` Synchronizes with, and participates in modification 788 and seq_cst total orderings with, other operations 789 (except image operations) for all address spaces 790 (except private, or generic that accesses private) 791 provided the other operation's sync scope is: 792 793 - ``system``, ``agent``, ``workgroup`` or 794 ``wavefront`` and executed by a thread in the 795 same wavefront. 796 797 ``singlethread`` Only synchronizes with and participates in 798 modification and seq_cst total orderings with, 799 other operations (except image operations) running 800 in the same thread for all address spaces (for 801 example, in signal handlers). 802 803 ``one-as`` Same as ``system`` but only synchronizes with other 804 operations within the same address space. 805 806 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 807 operations within the same address space. 808 809 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 810 other operations within the same address space. 811 812 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 813 other operations within the same address space. 814 815 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 816 other operations within the same address space. 817 ======================= =================================================== 818 819LLVM IR Intrinsics 820------------------ 821 822The AMDGPU backend implements the following LLVM IR intrinsics. 823 824*This section is WIP.* 825 826.. TODO:: 827 828 List AMDGPU intrinsics. 829 830LLVM IR Attributes 831------------------ 832 833The AMDGPU backend supports the following LLVM IR attributes. 834 835 .. table:: AMDGPU LLVM IR Attributes 836 :name: amdgpu-llvm-ir-attributes-table 837 838 ======================================= ========================================================== 839 LLVM Attribute Description 840 ======================================= ========================================================== 841 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 842 will be specified when the kernel is dispatched. Generated 843 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 844 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 845 argument block size for the implicit arguments. This 846 varies by OS and language (for OpenCL see 847 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 848 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 849 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 850 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 851 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 852 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 853 execution unit. Generated by the ``amdgpu_waves_per_eu`` 854 CLANG attribute [CLANG-ATTR]_. 855 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 856 mode register to be set on entry. Overrides the default for 857 the calling convention. 858 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 859 the mode register to be set on entry. Overrides the default 860 for the calling convention. 861 ======================================= ========================================================== 862 863.. _amdgpu-elf-code-object: 864 865ELF Code Object 866=============== 867 868The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 869can be linked by ``lld`` to produce a standard ELF shared code object which can 870be loaded and executed on an AMDGPU target. 871 872.. _amdgpu-elf-header: 873 874Header 875------ 876 877The AMDGPU backend uses the following ELF header: 878 879 .. table:: AMDGPU ELF Header 880 :name: amdgpu-elf-header-table 881 882 ========================== =============================== 883 Field Value 884 ========================== =============================== 885 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 886 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 887 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 888 - ``ELFOSABI_AMDGPU_HSA`` 889 - ``ELFOSABI_AMDGPU_PAL`` 890 - ``ELFOSABI_AMDGPU_MESA3D`` 891 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 892 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 893 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 894 - ``ELFABIVERSION_AMDGPU_PAL`` 895 - ``ELFABIVERSION_AMDGPU_MESA3D`` 896 ``e_type`` - ``ET_REL`` 897 - ``ET_DYN`` 898 ``e_machine`` ``EM_AMDGPU`` 899 ``e_entry`` 0 900 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 901 :ref:`amdgpu-elf-header-e_flags-table-v3`, 902 and :ref:`amdgpu-elf-header-e_flags-table-v4` 903 ========================== =============================== 904 905.. 906 907 .. table:: AMDGPU ELF Header Enumeration Values 908 :name: amdgpu-elf-header-enumeration-values-table 909 910 =============================== ===== 911 Name Value 912 =============================== ===== 913 ``EM_AMDGPU`` 224 914 ``ELFOSABI_NONE`` 0 915 ``ELFOSABI_AMDGPU_HSA`` 64 916 ``ELFOSABI_AMDGPU_PAL`` 65 917 ``ELFOSABI_AMDGPU_MESA3D`` 66 918 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 919 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 920 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 921 ``ELFABIVERSION_AMDGPU_PAL`` 0 922 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 923 =============================== ===== 924 925``e_ident[EI_CLASS]`` 926 The ELF class is: 927 928 * ``ELFCLASS32`` for ``r600`` architecture. 929 930 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 931 process address space applications. 932 933``e_ident[EI_DATA]`` 934 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 935 936``e_ident[EI_OSABI]`` 937 One of the following AMDGPU target architecture specific OS ABIs 938 (see :ref:`amdgpu-os`): 939 940 * ``ELFOSABI_NONE`` for *unknown* OS. 941 942 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 943 944 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 945 946 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 947 948``e_ident[EI_ABIVERSION]`` 949 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 950 object conforms: 951 952 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 953 runtime ABI for code object V2. Specify using the Clang option 954 ``-mcode-object-version=2``. 955 956 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 957 runtime ABI for code object V3. Specify using the Clang option 958 ``-mcode-object-version=3``. This is the default code object 959 version if not specified. 960 961 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 962 runtime ABI for code object V4. Specify using the Clang option 963 ``-mcode-object-version=4``. 964 965 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 966 runtime ABI. 967 968 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 969 3D runtime ABI. 970 971``e_type`` 972 Can be one of the following values: 973 974 975 ``ET_REL`` 976 The type produced by the AMDGPU backend compiler as it is relocatable code 977 object. 978 979 ``ET_DYN`` 980 The type produced by the linker as it is a shared code object. 981 982 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 983 984``e_machine`` 985 The value ``EM_AMDGPU`` is used for the machine for all processors supported 986 by the ``r600`` and ``amdgcn`` architectures (see 987 :ref:`amdgpu-processor-table`). The specific processor is specified in the 988 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 989 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 990 ``e_flags`` for code object V3 to V4 (see 991 :ref:`amdgpu-elf-header-e_flags-table-v3` and 992 :ref:`amdgpu-elf-header-e_flags-table-v4`). 993 994``e_entry`` 995 The entry point is 0 as the entry points for individual kernels must be 996 selected in order to invoke them through AQL packets. 997 998``e_flags`` 999 The AMDGPU backend uses the following ELF header flags: 1000 1001 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1002 :name: amdgpu-elf-header-e_flags-v2-table 1003 1004 ===================================== ===== ============================= 1005 Name Value Description 1006 ===================================== ===== ============================= 1007 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1008 target feature is 1009 enabled for all code 1010 contained in the code object. 1011 If the processor 1012 does not support the 1013 ``xnack`` target 1014 feature then must 1015 be 0. 1016 See 1017 :ref:`amdgpu-target-features`. 1018 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1019 handler is enabled for all 1020 code contained in the code 1021 object. If the processor 1022 does not support a trap 1023 handler then must be 0. 1024 See 1025 :ref:`amdgpu-target-features`. 1026 ===================================== ===== ============================= 1027 1028 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1029 :name: amdgpu-elf-header-e_flags-table-v3 1030 1031 ================================= ===== ============================= 1032 Name Value Description 1033 ================================= ===== ============================= 1034 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1035 mask for 1036 ``EF_AMDGPU_MACH_xxx`` values 1037 defined in 1038 :ref:`amdgpu-ef-amdgpu-mach-table`. 1039 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1040 target feature is 1041 enabled for all code 1042 contained in the code object. 1043 If the processor 1044 does not support the 1045 ``xnack`` target 1046 feature then must 1047 be 0. 1048 See 1049 :ref:`amdgpu-target-features`. 1050 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1051 target feature is 1052 enabled for all code 1053 contained in the code object. 1054 If the processor 1055 does not support the 1056 ``sramecc`` target 1057 feature then must 1058 be 0. 1059 See 1060 :ref:`amdgpu-target-features`. 1061 ================================= ===== ============================= 1062 1063 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1064 :name: amdgpu-elf-header-e_flags-table-v4 1065 1066 ============================================ ===== =================================== 1067 Name Value Description 1068 ============================================ ===== =================================== 1069 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1070 mask for 1071 ``EF_AMDGPU_MACH_xxx`` values 1072 defined in 1073 :ref:`amdgpu-ef-amdgpu-mach-table`. 1074 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1075 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1076 values. 1077 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1078 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1079 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1080 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1081 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1082 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1083 values. 1084 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1085 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1086 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1087 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1088 ============================================ ===== =================================== 1089 1090 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1091 :name: amdgpu-ef-amdgpu-mach-table 1092 1093 ==================================== ========== ============================= 1094 Name Value Description (see 1095 :ref:`amdgpu-processor-table`) 1096 ==================================== ========== ============================= 1097 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1098 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1099 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1100 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1101 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1102 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1103 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1104 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1105 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1106 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1107 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1108 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1109 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1110 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1111 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1112 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1113 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1114 *reserved* 0x011 - Reserved for ``r600`` 1115 0x01f architecture processors. 1116 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1117 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1118 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1119 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1120 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1121 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1122 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1123 *reserved* 0x027 Reserved. 1124 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1125 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1126 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1127 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1128 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1129 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1130 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1131 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1132 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1133 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1134 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1135 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1136 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1137 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1138 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1139 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1140 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1141 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1142 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1143 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1144 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1145 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1146 ==================================== ========== ============================= 1147 1148Sections 1149-------- 1150 1151An AMDGPU target ELF code object has the standard ELF sections which include: 1152 1153 .. table:: AMDGPU ELF Sections 1154 :name: amdgpu-elf-sections-table 1155 1156 ================== ================ ================================= 1157 Name Type Attributes 1158 ================== ================ ================================= 1159 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1160 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1161 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1162 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1163 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1164 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1165 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1166 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1167 ``.note`` ``SHT_NOTE`` *none* 1168 ``.rela``\ *name* ``SHT_RELA`` *none* 1169 ``.rela.dyn`` ``SHT_RELA`` *none* 1170 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1171 ``.shstrtab`` ``SHT_STRTAB`` *none* 1172 ``.strtab`` ``SHT_STRTAB`` *none* 1173 ``.symtab`` ``SHT_SYMTAB`` *none* 1174 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1175 ================== ================ ================================= 1176 1177These sections have their standard meanings (see [ELF]_) and are only generated 1178if needed. 1179 1180``.debug``\ *\** 1181 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1182 information on the DWARF produced by the AMDGPU backend. 1183 1184``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1185 The standard sections used by a dynamic loader. 1186 1187``.note`` 1188 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1189 backend. 1190 1191``.rela``\ *name*, ``.rela.dyn`` 1192 For relocatable code objects, *name* is the name of the section that the 1193 relocation records apply. For example, ``.rela.text`` is the section name for 1194 relocation records associated with the ``.text`` section. 1195 1196 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1197 records from each of the relocatable code object's ``.rela``\ *name* sections. 1198 1199 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1200 the AMDGPU backend. 1201 1202``.text`` 1203 The executable machine code for the kernels and functions they call. Generated 1204 as position independent code. See :ref:`amdgpu-code-conventions` for 1205 information on conventions used in the isa generation. 1206 1207.. _amdgpu-note-records: 1208 1209Note Records 1210------------ 1211 1212The AMDGPU backend code object contains ELF note records in the ``.note`` 1213section. The set of generated notes and their semantics depend on the code 1214object version; see :ref:`amdgpu-note-records-v2` and 1215:ref:`amdgpu-note-records-v3-v4`. 1216 1217As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1218must be generated after the ``name`` field to ensure the ``desc`` field is 4 1219byte aligned. In addition, minimal zero-byte padding must be generated to 1220ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1221field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1222alignment. 1223 1224.. _amdgpu-note-records-v2: 1225 1226Code Object V2 Note Records 1227~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1228 1229.. warning:: 1230 Code object V2 is not the default code object version emitted by 1231 this version of LLVM. 1232 1233The AMDGPU backend code object uses the following ELF note record in the 1234``.note`` section when compiling for code object V2. 1235 1236The note record vendor field is "AMD". 1237 1238Additional note records may be present, but any which are not documented here 1239are deprecated and should not be used. 1240 1241 .. table:: AMDGPU Code Object V2 ELF Note Records 1242 :name: amdgpu-elf-note-records-v2-table 1243 1244 ===== ===================================== ====================================== 1245 Name Type Description 1246 ===== ===================================== ====================================== 1247 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1248 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1249 Finalizer and not the LLVM compiler. 1250 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1251 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1252 YAML [YAML]_ textual format. 1253 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1254 ===== ===================================== ====================================== 1255 1256.. 1257 1258 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1259 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1260 1261 ===================================== ===== 1262 Name Value 1263 ===================================== ===== 1264 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1265 ``NT_AMD_HSA_HSAIL`` 2 1266 ``NT_AMD_HSA_ISA_VERSION`` 3 1267 *reserved* 4-9 1268 ``NT_AMD_HSA_METADATA`` 10 1269 ``NT_AMD_HSA_ISA_NAME`` 11 1270 ===================================== ===== 1271 1272``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1273 Specifies the code object version number. The description field has the 1274 following layout: 1275 1276 .. code:: 1277 1278 struct amdgpu_hsa_note_code_object_version_s { 1279 uint32_t major_version; 1280 uint32_t minor_version; 1281 }; 1282 1283 The ``major_version`` has a value less than or equal to 2. 1284 1285``NT_AMD_HSA_HSAIL`` 1286 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1287 field has the following layout: 1288 1289 .. code:: 1290 1291 struct amdgpu_hsa_note_hsail_s { 1292 uint32_t hsail_major_version; 1293 uint32_t hsail_minor_version; 1294 uint8_t profile; 1295 uint8_t machine_model; 1296 uint8_t default_float_round; 1297 }; 1298 1299``NT_AMD_HSA_ISA_VERSION`` 1300 Specifies the target ISA version. The description field has the following layout: 1301 1302 .. code:: 1303 1304 struct amdgpu_hsa_note_isa_s { 1305 uint16_t vendor_name_size; 1306 uint16_t architecture_name_size; 1307 uint32_t major; 1308 uint32_t minor; 1309 uint32_t stepping; 1310 char vendor_and_architecture_name[1]; 1311 }; 1312 1313 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1314 vendor and architecture names respectively, including the NUL character. 1315 1316 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1317 vendor, immediately followed by the NUL terminated string for the 1318 architecture. 1319 1320 This note record is used by the HSA runtime loader. 1321 1322 Code object V2 only supports a limited number of processors and has fixed 1323 settings for target features. See 1324 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1325 processors and the corresponding target ID. In the table the note record ISA 1326 name is a concatenation of the vendor name, architecture name, major, minor, 1327 and stepping separated by a ":". 1328 1329 The target ID column shows the processor name and fixed target features used 1330 by the LLVM compiler. The LLVM compiler does not generate a 1331 ``NT_AMD_HSA_HSAIL`` note record. 1332 1333 A code object generated by the Finalizer also uses code object V2 and always 1334 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1335 ``sramecc`` target feature is as shown in 1336 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1337 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1338 bit. 1339 1340``NT_AMD_HSA_ISA_NAME`` 1341 Specifies the target ISA name as a non-NUL terminated string. 1342 1343 This note record is not used by the HSA runtime loader. 1344 1345 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1346 V2's limited support of processors and fixed settings for target features. 1347 1348 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1349 from the string to the corresponding target ID. If the ``xnack`` target 1350 feature is supported and enabled, the string produced by the LLVM compiler 1351 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1352 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1353 1354``NT_AMD_HSA_METADATA`` 1355 Specifies extensible metadata associated with the code objects executed on HSA 1356 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1357 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1358 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1359 metadata string. 1360 1361 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1362 :name: amdgpu-elf-note-record-supported_processors-v2-table 1363 1364 ==================== ========================== 1365 Note Record ISA Name Target ID 1366 ==================== ========================== 1367 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1368 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1369 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1370 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1371 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1372 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1373 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1374 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1375 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1376 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1377 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1378 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1379 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1380 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1381 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1382 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1383 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1384 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1385 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1386 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1387 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1388 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1389 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1390 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1391 ==================== ========================== 1392 1393.. _amdgpu-note-records-v3-v4: 1394 1395Code Object V3 to V4 Note Records 1396~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1397 1398The AMDGPU backend code object uses the following ELF note record in the 1399``.note`` section when compiling for code object V3 to V4. 1400 1401The note record vendor field is "AMDGPU". 1402 1403Additional note records may be present, but any which are not documented here 1404are deprecated and should not be used. 1405 1406 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1407 :name: amdgpu-elf-note-records-table-v3-v4 1408 1409 ======== ============================== ====================================== 1410 Name Type Description 1411 ======== ============================== ====================================== 1412 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1413 binary format. 1414 ======== ============================== ====================================== 1415 1416.. 1417 1418 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1419 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1420 1421 ============================== ===== 1422 Name Value 1423 ============================== ===== 1424 *reserved* 0-31 1425 ``NT_AMDGPU_METADATA`` 32 1426 ============================== ===== 1427 1428``NT_AMDGPU_METADATA`` 1429 Specifies extensible metadata associated with an AMDGPU code object. It is 1430 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1431 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1432 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1433 ``amdhsa`` OS. 1434 1435.. _amdgpu-symbols: 1436 1437Symbols 1438------- 1439 1440Symbols include the following: 1441 1442 .. table:: AMDGPU ELF Symbols 1443 :name: amdgpu-elf-symbols-table 1444 1445 ===================== ================== ================ ================== 1446 Name Type Section Description 1447 ===================== ================== ================ ================== 1448 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1449 - ``.rodata`` 1450 - ``.bss`` 1451 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1452 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1453 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1454 ===================== ================== ================ ================== 1455 1456Global variable 1457 Global variables both used and defined by the compilation unit. 1458 1459 If the symbol is defined in the compilation unit then it is allocated in the 1460 appropriate section according to if it has initialized data or is readonly. 1461 1462 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1463 will resolve relocations using the definition provided by another code object 1464 or explicitly defined by the runtime. 1465 1466 If the symbol resides in local/group memory (LDS) then its section is the 1467 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1468 ``st_value`` field describes alignment requirements as it does for common 1469 symbols. 1470 1471 .. TODO:: 1472 1473 Add description of linked shared object symbols. Seems undefined symbols 1474 are marked as STT_NOTYPE. 1475 1476Kernel descriptor 1477 Every HSA kernel has an associated kernel descriptor. It is the address of the 1478 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1479 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1480 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1481 1482Kernel entry point 1483 Every HSA kernel also has a symbol for its machine code entry point. 1484 1485.. _amdgpu-relocation-records: 1486 1487Relocation Records 1488------------------ 1489 1490AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1491relocatable fields are: 1492 1493``word32`` 1494 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1495 alignment. These values use the same byte order as other word values in the 1496 AMDGPU architecture. 1497 1498``word64`` 1499 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1500 alignment. These values use the same byte order as other word values in the 1501 AMDGPU architecture. 1502 1503Following notations are used for specifying relocation calculations: 1504 1505**A** 1506 Represents the addend used to compute the value of the relocatable field. 1507 1508**G** 1509 Represents the offset into the global offset table at which the relocation 1510 entry's symbol will reside during execution. 1511 1512**GOT** 1513 Represents the address of the global offset table. 1514 1515**P** 1516 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1517 of the storage unit being relocated (computed using ``r_offset``). 1518 1519**S** 1520 Represents the value of the symbol whose index resides in the relocation 1521 entry. Relocations not using this must specify a symbol index of 1522 ``STN_UNDEF``. 1523 1524**B** 1525 Represents the base address of a loaded executable or shared object which is 1526 the difference between the ELF address and the actual load address. 1527 Relocations using this are only valid in executable or shared objects. 1528 1529The following relocation types are supported: 1530 1531 .. table:: AMDGPU ELF Relocation Records 1532 :name: amdgpu-elf-relocation-records-table 1533 1534 ========================== ======= ===== ========== ============================== 1535 Relocation Type Kind Value Field Calculation 1536 ========================== ======= ===== ========== ============================== 1537 ``R_AMDGPU_NONE`` 0 *none* *none* 1538 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1539 Dynamic 1540 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1541 Dynamic 1542 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1543 Dynamic 1544 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1545 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1546 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1547 Dynamic 1548 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1549 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1550 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1551 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1552 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1553 *reserved* 12 1554 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1555 ========================== ======= ===== ========== ============================== 1556 1557``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1558the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1559 1560There is no current OS loader support for 32-bit programs and so 1561``R_AMDGPU_ABS32`` is not used. 1562 1563.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1564 1565Loaded Code Object Path Uniform Resource Identifier (URI) 1566--------------------------------------------------------- 1567 1568The AMD GPU code object loader represents the path of the ELF shared object from 1569which the code object was loaded as a textual Unifom Resource Identifier (URI). 1570Note that the code object is the in memory loaded relocated form of the ELF 1571shared object. Multiple code objects may be loaded at different memory 1572addresses in the same process from the same ELF shared object. 1573 1574The loaded code object path URI syntax is defined by the following BNF syntax: 1575 1576.. code:: 1577 1578 code_object_uri ::== file_uri | memory_uri 1579 file_uri ::== "file://" file_path [ range_specifier ] 1580 memory_uri ::== "memory://" process_id range_specifier 1581 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1582 file_path ::== URI_ENCODED_OS_FILE_PATH 1583 process_id ::== DECIMAL_NUMBER 1584 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1585 1586**number** 1587 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1588 and octal values by "0". 1589 1590**file_path** 1591 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1592 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1593 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1594 the path are separated by "/". 1595 1596**offset** 1597 Is a 0-based byte offset to the start of the code object. For a file URI, it 1598 is from the start of the file specified by the ``file_path``, and if omitted 1599 defaults to 0. For a memory URI, it is the memory address and is required. 1600 1601**size** 1602 Is the number of bytes in the code object. For a file URI, if omitted it 1603 defaults to the size of the file. It is required for a memory URI. 1604 1605**process_id** 1606 Is the identity of the process owning the memory. For Linux it is the C 1607 unsigned integral decimal literal for the process ID (PID). 1608 1609For example: 1610 1611.. code:: 1612 1613 file:///dir1/dir2/file1 1614 file:///dir3/dir4/file2#offset=0x2000&size=3000 1615 memory://1234#offset=0x20000&size=3000 1616 1617.. _amdgpu-dwarf-debug-information: 1618 1619DWARF Debug Information 1620======================= 1621 1622.. warning:: 1623 1624 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1625 is not currently fully implemented and is subject to change. 1626 1627AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1628:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1629object executable code and data to the source language constructs. It can be 1630used by tools such as debuggers and profilers. It uses features defined in 1631:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1632DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1633 1634This section defines the AMDGPU target architecture specific DWARF mappings. 1635 1636.. _amdgpu-dwarf-register-identifier: 1637 1638Register Identifier 1639------------------- 1640 1641This section defines the AMDGPU target architecture register numbers used in 1642DWARF operation expressions (see DWARF Version 5 section 2.5 and 1643:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1644instructions (see DWARF Version 5 section 6.4 and 1645:ref:`amdgpu-dwarf-call-frame-information`). 1646 1647A single code object can contain code for kernels that have different wavefront 1648sizes. The vector registers and some scalar registers are based on the wavefront 1649size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1650simplifies the consumer of the DWARF so that each register has a fixed size, 1651rather than being dynamic according to the wavefront size mode. Similarly, 1652distinct DWARF registers are defined for those registers that vary in size 1653according to the process address size. This allows a consumer to treat a 1654specific AMDGPU processor as a single architecture regardless of how it is 1655configured at run time. The compiler explicitly specifies the DWARF registers 1656that match the mode in which the code it is generating will be executed. 1657 1658DWARF registers are encoded as numbers, which are mapped to architecture 1659registers. The mapping for AMDGPU is defined in 1660:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1661mapping. 1662 1663.. table:: AMDGPU DWARF Register Mapping 1664 :name: amdgpu-dwarf-register-mapping-table 1665 1666 ============== ================= ======== ================================== 1667 DWARF Register AMDGPU Register Bit Size Description 1668 ============== ================= ======== ================================== 1669 0 PC_32 32 Program Counter (PC) when 1670 executing in a 32-bit process 1671 address space. Used in the CFI to 1672 describe the PC of the calling 1673 frame. 1674 1 EXEC_MASK_32 32 Execution Mask Register when 1675 executing in wavefront 32 mode. 1676 2-15 *Reserved* *Reserved for highly accessed 1677 registers using DWARF shortcut.* 1678 16 PC_64 64 Program Counter (PC) when 1679 executing in a 64-bit process 1680 address space. Used in the CFI to 1681 describe the PC of the calling 1682 frame. 1683 17 EXEC_MASK_64 64 Execution Mask Register when 1684 executing in wavefront 64 mode. 1685 18-31 *Reserved* *Reserved for highly accessed 1686 registers using DWARF shortcut.* 1687 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1688 Registers. 1689 96-127 *Reserved* *Reserved for frequently accessed 1690 registers using DWARF 1-byte ULEB.* 1691 128 STATUS 32 Status Register. 1692 129-511 *Reserved* *Reserved for future Scalar 1693 Architectural Registers.* 1694 512 VCC_32 32 Vector Condition Code Register 1695 when executing in wavefront 32 1696 mode. 1697 513-1023 *Reserved* *Reserved for future Vector 1698 Architectural Registers when 1699 executing in wavefront 32 mode.* 1700 768 VCC_64 64 Vector Condition Code Register 1701 when executing in wavefront 64 1702 mode. 1703 769-1023 *Reserved* *Reserved for future Vector 1704 Architectural Registers when 1705 executing in wavefront 64 mode.* 1706 1024-1087 *Reserved* *Reserved for padding.* 1707 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1708 1130-1535 *Reserved* *Reserved for future Scalar 1709 General Purpose Registers.* 1710 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1711 when executing in wavefront 32 1712 mode. 1713 1792-2047 *Reserved* *Reserved for future Vector 1714 General Purpose Registers when 1715 executing in wavefront 32 mode.* 1716 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1717 when executing in wavefront 32 1718 mode. 1719 2304-2559 *Reserved* *Reserved for future Vector 1720 Accumulation Registers when 1721 executing in wavefront 32 mode.* 1722 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1723 when executing in wavefront 64 1724 mode. 1725 2816-3071 *Reserved* *Reserved for future Vector 1726 General Purpose Registers when 1727 executing in wavefront 64 mode.* 1728 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1729 when executing in wavefront 64 1730 mode. 1731 3328-3583 *Reserved* *Reserved for future Vector 1732 Accumulation Registers when 1733 executing in wavefront 64 mode.* 1734 ============== ================= ======== ================================== 1735 1736The vector registers are represented as the full size for the wavefront. They 1737are organized as consecutive dwords (32-bits), one per lane, with the dword at 1738the least significant bit position corresponding to lane 0 and so forth. DWARF 1739location expressions involving the ``DW_OP_LLVM_offset`` and 1740``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1741register corresponding to the lane that is executing the current thread of 1742execution in languages that are implemented using a SIMD or SIMT execution 1743model. 1744 1745If the wavefront size is 32 lanes then the wavefront 32 mode register 1746definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1747mode register definitions are used. Some AMDGPU targets support executing in 1748both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1749to the wavefront mode of the generated code will be used. 1750 1751If code is generated to execute in a 32-bit process address space, then the 175232-bit process address space register definitions are used. If code is generated 1753to execute in a 64-bit process address space, then the 64-bit process address 1754space register definitions are used. The ``amdgcn`` target only supports the 175564-bit process address space. 1756 1757.. _amdgpu-dwarf-address-class-identifier: 1758 1759Address Class Identifier 1760------------------------ 1761 1762The DWARF address class represents the source language memory space. See DWARF 1763Version 5 section 2.12 which is updated by the *DWARF Extensions For 1764Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1765 1766The DWARF address class mapping used for AMDGPU is defined in 1767:ref:`amdgpu-dwarf-address-class-mapping-table`. 1768 1769.. table:: AMDGPU DWARF Address Class Mapping 1770 :name: amdgpu-dwarf-address-class-mapping-table 1771 1772 ========================= ====== ================= 1773 DWARF AMDGPU 1774 -------------------------------- ----------------- 1775 Address Class Name Value Address Space 1776 ========================= ====== ================= 1777 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1778 ``DW_ADDR_LLVM_global`` 0x0001 Global 1779 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1780 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1781 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1782 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1783 ========================= ====== ================= 1784 1785The DWARF address class values defined in the *DWARF Extensions For 1786Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1787 1788In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1789available for use for the AMD extension for access to the hardware GDS memory 1790which is scratchpad memory allocated per device. 1791 1792For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1793address class of ``DW_ADDR_none`` is used. 1794 1795See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1796mapping of DWARF address classes to DWARF address spaces, including address size 1797and NULL value. 1798 1799.. _amdgpu-dwarf-address-space-identifier: 1800 1801Address Space Identifier 1802------------------------ 1803 1804DWARF address spaces correspond to target architecture specific linear 1805addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1806For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1807 1808The DWARF address space mapping used for AMDGPU is defined in 1809:ref:`amdgpu-dwarf-address-space-mapping-table`. 1810 1811.. table:: AMDGPU DWARF Address Space Mapping 1812 :name: amdgpu-dwarf-address-space-mapping-table 1813 1814 ======================================= ===== ======= ======== ================= ======================= 1815 DWARF AMDGPU Notes 1816 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1817 Address Space Name Value Address Bit Size Address Space 1818 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1819 .. 64-bit 32-bit 1820 process process 1821 address address 1822 space space 1823 ======================================= ===== ======= ======== ================= ======================= 1824 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1825 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1826 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1827 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1828 *Reserved* 0x04 1829 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1830 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1831 ======================================= ===== ======= ======== ================= ======================= 1832 1833See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1834including address size and NULL value. 1835 1836The ``DW_ASPACE_none`` address space is the default target architecture address 1837space used in DWARF operations that do not specify an address space. It 1838therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1839related operations can refer to addresses in the program code. 1840 1841The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1842specify the flat address space. If the address corresponds to an address in the 1843local address space, then it corresponds to the wavefront that is executing the 1844focused thread of execution. If the address corresponds to an address in the 1845private address space, then it corresponds to the lane that is executing the 1846focused thread of execution for languages that are implemented using a SIMD or 1847SIMT execution model. 1848 1849.. note:: 1850 1851 CUDA-like languages such as HIP that do not have address spaces in the 1852 language type system, but do allow variables to be allocated in different 1853 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1854 address space in the DWARF expression operations as the default address space 1855 is the global address space. 1856 1857The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1858specify the local address space corresponding to the wavefront that is executing 1859the focused thread of execution. 1860 1861The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1862to specify the private address space corresponding to the lane that is executing 1863the focused thread of execution for languages that are implemented using a SIMD 1864or SIMT execution model. 1865 1866The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1867to specify the unswizzled private address space corresponding to the wavefront 1868that is executing the focused thread of execution. The wavefront view of private 1869memory is the per wavefront unswizzled backing memory layout defined in 1870:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1871location for the backing memory of the wavefront (namely the address is not 1872offset by ``wavefront-scratch-base``). The following formula can be used to 1873convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1874``DW_ASPACE_AMDGPU_private_wave`` address: 1875 1876:: 1877 1878 private-address-wavefront = 1879 ((private-address-lane / 4) * wavefront-size * 4) + 1880 (wavefront-lane-id * 4) + (private-address-lane % 4) 1881 1882If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1883of the dwords for each lane starting with lane 0 is required, then this 1884simplifies to: 1885 1886:: 1887 1888 private-address-wavefront = 1889 private-address-lane * wavefront-size 1890 1891A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1892complete spilled vector register back into a complete vector register in the 1893CFI. The frame pointer can be a private lane address which is dword aligned, 1894which can be shifted to multiply by the wavefront size, and then used to form a 1895private wavefront address that gives a location for a contiguous set of dwords, 1896one per lane, where the vector register dwords are spilled. The compiler knows 1897the wavefront size since it generates the code. Note that the type of the 1898address may have to be converted as the size of a 1899``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1900``DW_ASPACE_AMDGPU_private_wave`` address. 1901 1902.. _amdgpu-dwarf-lane-identifier: 1903 1904Lane identifier 1905--------------- 1906 1907DWARF lane identifies specify a target architecture lane position for hardware 1908that executes in a SIMD or SIMT manner, and on which a source language maps its 1909threads of execution onto those lanes. The DWARF lane identifier is pushed by 1910the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1911section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1912section :ref:`amdgpu-dwarf-operation-expressions`. 1913 1914For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1915wavefront. It is numbered from 0 to the wavefront size minus 1. 1916 1917Operation Expressions 1918--------------------- 1919 1920DWARF expressions are used to compute program values and the locations of 1921program objects. See DWARF Version 5 section 2.5 and 1922:ref:`amdgpu-dwarf-operation-expressions`. 1923 1924DWARF location descriptions describe how to access storage which includes memory 1925and registers. When accessing storage on AMDGPU, bytes are ordered with least 1926significant bytes first, and bits are ordered within bytes with least 1927significant bits first. 1928 1929For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1930unwinding vector registers that are spilled under the execution mask to memory: 1931the zero-single location description is the vector register, and the one-single 1932location description is the spilled memory location description. The 1933``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1934memory location description. 1935 1936In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1937``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1938controlled by the execution mask. An undefined location description together 1939with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1940to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1941 1942Debugger Information Entry Attributes 1943------------------------------------- 1944 1945This section describes how certain debugger information entry attributes are 1946used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1947by *DWARF Extensions For Heterogeneous Debugging* section 1948:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1949 1950.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1951 1952``DW_AT_LLVM_lane_pc`` 1953~~~~~~~~~~~~~~~~~~~~~~ 1954 1955For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1956location of the separate lanes of a SIMT thread. 1957 1958If the lane is an active lane then this will be the same as the current program 1959location. 1960 1961If the lane is inactive, but was active on entry to the subprogram, then this is 1962the program location in the subprogram at which execution of the lane is 1963conceptual positioned. 1964 1965If the lane was not active on entry to the subprogram, then this will be the 1966undefined location. A client debugger can check if the lane is part of a valid 1967work-group by checking that the lane is in the range of the associated 1968work-group within the grid, accounting for partial work-groups. If it is not, 1969then the debugger can omit any information for the lane. Otherwise, the debugger 1970may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1971calling subprogram until it finds a non-undefined location. Conceptually the 1972lane only has the call frames that it has a non-undefined 1973``DW_AT_LLVM_lane_pc``. 1974 1975The following example illustrates how the AMDGPU backend can generate a DWARF 1976location list expression for the nested ``IF/THEN/ELSE`` structures of the 1977following subprogram pseudo code for a target with 64 lanes per wavefront. 1978 1979.. code:: 1980 :number-lines: 1981 1982 SUBPROGRAM X 1983 BEGIN 1984 a; 1985 IF (c1) THEN 1986 b; 1987 IF (c2) THEN 1988 c; 1989 ELSE 1990 d; 1991 ENDIF 1992 e; 1993 ELSE 1994 f; 1995 ENDIF 1996 g; 1997 END 1998 1999The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2000execution mask (``EXEC``) to linearize the control flow. The condition is 2001evaluated to make a mask of the lanes for which the condition evaluates to true. 2002First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2003logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2004``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2005the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2006region the ``EXEC`` mask is restored to the value it had at the beginning of the 2007region. This is shown below. Other approaches are possible, but the basic 2008concept is the same. 2009 2010.. code:: 2011 :number-lines: 2012 2013 $lex_start: 2014 a; 2015 %1 = EXEC 2016 %2 = c1 2017 $lex_1_start: 2018 EXEC = %1 & %2 2019 $if_1_then: 2020 b; 2021 %3 = EXEC 2022 %4 = c2 2023 $lex_1_1_start: 2024 EXEC = %3 & %4 2025 $lex_1_1_then: 2026 c; 2027 EXEC = ~EXEC & %3 2028 $lex_1_1_else: 2029 d; 2030 EXEC = %3 2031 $lex_1_1_end: 2032 e; 2033 EXEC = ~EXEC & %1 2034 $lex_1_else: 2035 f; 2036 EXEC = %1 2037 $lex_1_end: 2038 g; 2039 $lex_end: 2040 2041To create the DWARF location list expression that defines the location 2042description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2043pseudo instruction can be used to annotate the linearized control flow. This can 2044be done by defining an artificial variable for the lane PC. The DWARF location 2045list expression created for it is used as the value of the 2046``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2047 2048A DWARF procedure is defined for each well nested structured control flow region 2049which provides the conceptual lane program location for a lane if it is not 2050active (namely it is divergent). The DWARF operation expression for each region 2051conceptually inherits the value of the immediately enclosing region and modifies 2052it according to the semantics of the region. 2053 2054For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2055the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2056region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2057region since the ``THEN`` region has completed. 2058 2059The lane PC artificial variable is assigned at each region transition. It uses 2060the immediately enclosing region's DWARF procedure to compute the program 2061location for each lane assuming they are divergent, and then modifies the result 2062by inserting the current program location for each lane that the ``EXEC`` mask 2063indicates is active. 2064 2065By having separate DWARF procedures for each region, they can be reused to 2066define the value for any nested region. This reduces the total size of the DWARF 2067operation expressions. 2068 2069The following provides an example using pseudo LLVM MIR. 2070 2071.. code:: 2072 :number-lines: 2073 2074 $lex_start: 2075 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2076 DW_AT_name = "__uint64"; 2077 DW_AT_byte_size = 8; 2078 DW_AT_encoding = DW_ATE_unsigned; 2079 ]; 2080 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2081 DW_AT_name = "__active_lane_pc"; 2082 DW_AT_location = [ 2083 DW_OP_regx PC; 2084 DW_OP_LLVM_extend 64, 64; 2085 DW_OP_regval_type EXEC, %uint_64; 2086 DW_OP_LLVM_select_bit_piece 64, 64; 2087 ]; 2088 ]; 2089 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2090 DW_AT_name = "__divergent_lane_pc"; 2091 DW_AT_location = [ 2092 DW_OP_LLVM_undefined; 2093 DW_OP_LLVM_extend 64, 64; 2094 ]; 2095 ]; 2096 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2097 DW_OP_call_ref %__divergent_lane_pc; 2098 DW_OP_call_ref %__active_lane_pc; 2099 ]; 2100 a; 2101 %1 = EXEC; 2102 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2103 %2 = c1; 2104 $lex_1_start: 2105 EXEC = %1 & %2; 2106 $lex_1_then: 2107 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2108 DW_AT_name = "__divergent_lane_pc_1_then"; 2109 DW_AT_location = DIExpression[ 2110 DW_OP_call_ref %__divergent_lane_pc; 2111 DW_OP_addrx &lex_1_start; 2112 DW_OP_stack_value; 2113 DW_OP_LLVM_extend 64, 64; 2114 DW_OP_call_ref %__lex_1_save_exec; 2115 DW_OP_deref_type 64, %__uint_64; 2116 DW_OP_LLVM_select_bit_piece 64, 64; 2117 ]; 2118 ]; 2119 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2120 DW_OP_call_ref %__divergent_lane_pc_1_then; 2121 DW_OP_call_ref %__active_lane_pc; 2122 ]; 2123 b; 2124 %3 = EXEC; 2125 DBG_VALUE %3, %__lex_1_1_save_exec; 2126 %4 = c2; 2127 $lex_1_1_start: 2128 EXEC = %3 & %4; 2129 $lex_1_1_then: 2130 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2131 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2132 DW_AT_location = DIExpression[ 2133 DW_OP_call_ref %__divergent_lane_pc_1_then; 2134 DW_OP_addrx &lex_1_1_start; 2135 DW_OP_stack_value; 2136 DW_OP_LLVM_extend 64, 64; 2137 DW_OP_call_ref %__lex_1_1_save_exec; 2138 DW_OP_deref_type 64, %__uint_64; 2139 DW_OP_LLVM_select_bit_piece 64, 64; 2140 ]; 2141 ]; 2142 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2143 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2144 DW_OP_call_ref %__active_lane_pc; 2145 ]; 2146 c; 2147 EXEC = ~EXEC & %3; 2148 $lex_1_1_else: 2149 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2150 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2151 DW_AT_location = DIExpression[ 2152 DW_OP_call_ref %__divergent_lane_pc_1_then; 2153 DW_OP_addrx &lex_1_1_end; 2154 DW_OP_stack_value; 2155 DW_OP_LLVM_extend 64, 64; 2156 DW_OP_call_ref %__lex_1_1_save_exec; 2157 DW_OP_deref_type 64, %__uint_64; 2158 DW_OP_LLVM_select_bit_piece 64, 64; 2159 ]; 2160 ]; 2161 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2162 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2163 DW_OP_call_ref %__active_lane_pc; 2164 ]; 2165 d; 2166 EXEC = %3; 2167 $lex_1_1_end: 2168 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2169 DW_OP_call_ref %__divergent_lane_pc; 2170 DW_OP_call_ref %__active_lane_pc; 2171 ]; 2172 e; 2173 EXEC = ~EXEC & %1; 2174 $lex_1_else: 2175 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2176 DW_AT_name = "__divergent_lane_pc_1_else"; 2177 DW_AT_location = DIExpression[ 2178 DW_OP_call_ref %__divergent_lane_pc; 2179 DW_OP_addrx &lex_1_end; 2180 DW_OP_stack_value; 2181 DW_OP_LLVM_extend 64, 64; 2182 DW_OP_call_ref %__lex_1_save_exec; 2183 DW_OP_deref_type 64, %__uint_64; 2184 DW_OP_LLVM_select_bit_piece 64, 64; 2185 ]; 2186 ]; 2187 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2188 DW_OP_call_ref %__divergent_lane_pc_1_else; 2189 DW_OP_call_ref %__active_lane_pc; 2190 ]; 2191 f; 2192 EXEC = %1; 2193 $lex_1_end: 2194 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2195 DW_OP_call_ref %__divergent_lane_pc; 2196 DW_OP_call_ref %__active_lane_pc; 2197 ]; 2198 g; 2199 $lex_end: 2200 2201The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2202that are active, with the current program location. 2203 2204Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2205the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2206instruction, location list entries will be created that describe where the 2207artificial variables are allocated at any given program location. The compiler 2208may allocate them to registers or spill them to memory. 2209 2210The DWARF procedures for each region use the values of the saved execution mask 2211artificial variables to only update the lanes that are active on entry to the 2212region. All other lanes retain the value of the enclosing region where they were 2213last active. If they were not active on entry to the subprogram, then will have 2214the undefined location description. 2215 2216Other structured control flow regions can be handled similarly. For example, 2217loops would set the divergent program location for the region at the end of the 2218loop. Any lanes active will be in the loop, and any lanes not active must have 2219exited the loop. 2220 2221An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2222``IF/THEN/ELSE`` regions. 2223 2224The DWARF procedures can use the active lane artificial variable described in 2225:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2226``EXEC`` mask in order to support whole or quad wavefront mode. 2227 2228.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2229 2230``DW_AT_LLVM_active_lane`` 2231~~~~~~~~~~~~~~~~~~~~~~~~~~ 2232 2233The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2234entry is used to specify the lanes that are conceptually active for a SIMT 2235thread. 2236 2237The execution mask may be modified to implement whole or quad wavefront mode 2238operations. For example, all lanes may need to temporarily be made active to 2239execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2240update it to enable the necessary lanes, perform the operations, and then 2241restore the ``EXEC`` mask from the saved value. While executing the whole 2242wavefront region, the conceptual execution mask is the saved value, not the 2243``EXEC`` value. 2244 2245This is handled by defining an artificial variable for the active lane mask. The 2246active lane mask artificial variable would be the actual ``EXEC`` mask for 2247normal regions, and the saved execution mask for regions where the mask is 2248temporarily updated. The location list expression created for this artificial 2249variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2250attribute. 2251 2252``DW_AT_LLVM_augmentation`` 2253~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2254 2255For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2256debugger information entry has the following value for the augmentation string: 2257 2258:: 2259 2260 [amdgpu:v0.0] 2261 2262The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2263extensions used in the DWARF of the compilation unit. The version number 2264conforms to [SEMVER]_. 2265 2266Call Frame Information 2267---------------------- 2268 2269DWARF Call Frame Information (CFI) describes how a consumer can virtually 2270*unwind* call frames in a running process or core dump. See DWARF Version 5 2271section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2272 2273For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2274 22751. ``augmentation`` string contains the following null-terminated UTF-8 string: 2276 2277 :: 2278 2279 [amd:v0.0] 2280 2281 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2282 extensions used in this CIE or to the FDEs that use it. The version number 2283 conforms to [SEMVER]_. 2284 22852. ``address_size`` for the ``Global`` address space is defined in 2286 :ref:`amdgpu-dwarf-address-space-identifier`. 2287 22883. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2289 22904. ``code_alignment_factor`` is 4 bytes. 2291 2292 .. TODO:: 2293 2294 Add to :ref:`amdgpu-processor-table` table. 2295 22965. ``data_alignment_factor`` is 4 bytes. 2297 2298 .. TODO:: 2299 2300 Add to :ref:`amdgpu-processor-table` table. 2301 23026. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2303 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2304 23057. ``initial_instructions`` Since a subprogram X with fewer registers can be 2306 called from subprogram Y that has more allocated, X will not change any of 2307 the extra registers as it cannot access them. Therefore, the default rule 2308 for all columns is ``same value``. 2309 2310For AMDGPU the register number follows the numbering defined in 2311:ref:`amdgpu-dwarf-register-identifier`. 2312 2313For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2314the return address to get the address of a byte within the call site 2315instructions. See DWARF Version 5 section 6.4.4. 2316 2317Accelerated Access 2318------------------ 2319 2320See DWARF Version 5 section 6.1. 2321 2322Lookup By Name Section Header 2323~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2324 2325See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2326 2327For AMDGPU the lookup by name section header table: 2328 2329``augmentation_string_size`` (uword) 2330 2331 Set to the length of the ``augmentation_string`` value which is always a 2332 multiple of 4. 2333 2334``augmentation_string`` (sequence of UTF-8 characters) 2335 2336 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2337 2338 :: 2339 2340 [amdgpu:v0.0] 2341 2342 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2343 extensions used in the DWARF of this index. The version number conforms to 2344 [SEMVER]_. 2345 2346 .. note:: 2347 2348 This is different to the DWARF Version 5 definition that requires the first 2349 4 characters to be the vendor ID. But this is consistent with the other 2350 augmentation strings and does allow multiple vendor contributions. However, 2351 backwards compatibility may be more desirable. 2352 2353Lookup By Address Section Header 2354~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2355 2356See DWARF Version 5 section 6.1.2. 2357 2358For AMDGPU the lookup by address section header table: 2359 2360``address_size`` (ubyte) 2361 2362 Match the address size for the ``Global`` address space defined in 2363 :ref:`amdgpu-dwarf-address-space-identifier`. 2364 2365``segment_selector_size`` (ubyte) 2366 2367 AMDGPU does not use a segment selector so this is 0. The entries in the 2368 ``.debug_aranges`` do not have a segment selector. 2369 2370Line Number Information 2371----------------------- 2372 2373See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2374 2375AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2376The instruction set must be obtained from the ELF file header ``e_flags`` field 2377in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2378<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2379 2380.. TODO:: 2381 2382 Should the ``isa`` state machine register be used to indicate if the code is 2383 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2384 2385For AMDGPU the line number program header fields have the following values (see 2386DWARF Version 5 section 6.2.4): 2387 2388``address_size`` (ubyte) 2389 Matches the address size for the ``Global`` address space defined in 2390 :ref:`amdgpu-dwarf-address-space-identifier`. 2391 2392``segment_selector_size`` (ubyte) 2393 AMDGPU does not use a segment selector so this is 0. 2394 2395``minimum_instruction_length`` (ubyte) 2396 For GFX9-GFX10 this is 4. 2397 2398``maximum_operations_per_instruction`` (ubyte) 2399 For GFX9-GFX10 this is 1. 2400 2401Source text for online-compiled programs (for example, those compiled by the 2402OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2403See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2404Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2405<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2406 2407The Clang option used to control source embedding in AMDGPU is defined in 2408:ref:`amdgpu-clang-debug-options-table`. 2409 2410 .. table:: AMDGPU Clang Debug Options 2411 :name: amdgpu-clang-debug-options-table 2412 2413 ==================== ================================================== 2414 Debug Flag Description 2415 ==================== ================================================== 2416 -g[no-]embed-source Enable/disable embedding source text in DWARF 2417 debug sections. Useful for environments where 2418 source cannot be written to disk, such as 2419 when performing online compilation. 2420 ==================== ================================================== 2421 2422For example: 2423 2424``-gembed-source`` 2425 Enable the embedded source. 2426 2427``-gno-embed-source`` 2428 Disable the embedded source. 2429 243032-Bit and 64-Bit DWARF Formats 2431------------------------------- 2432 2433See DWARF Version 5 section 7.4 and 2434:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2435 2436For AMDGPU: 2437 2438* For the ``amdgcn`` target architecture only the 64-bit process address space 2439 is supported. 2440 2441* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2442 the 32-bit DWARF format. 2443 2444Unit Headers 2445------------ 2446 2447For AMDGPU the following values apply for each of the unit headers described in 2448DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2449 2450``address_size`` (ubyte) 2451 Matches the address size for the ``Global`` address space defined in 2452 :ref:`amdgpu-dwarf-address-space-identifier`. 2453 2454.. _amdgpu-code-conventions: 2455 2456Code Conventions 2457================ 2458 2459This section provides code conventions used for each supported target triple OS 2460(see :ref:`amdgpu-target-triples`). 2461 2462AMDHSA 2463------ 2464 2465This section provides code conventions used when the target triple OS is 2466``amdhsa`` (see :ref:`amdgpu-target-triples`). 2467 2468.. _amdgpu-amdhsa-code-object-metadata: 2469 2470Code Object Metadata 2471~~~~~~~~~~~~~~~~~~~~ 2472 2473The code object metadata specifies extensible metadata associated with the code 2474objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2475encoding and semantics of this metadata depends on the code object version; see 2476:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2477:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2478:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2479 2480Code object metadata is specified in a note record (see 2481:ref:`amdgpu-note-records`) and is required when the target triple OS is 2482``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2483information necessary to support the HSA compatible runtime kernel queries. For 2484example, the segment sizes needed in a dispatch packet. In addition, a 2485high-level language runtime may require other information to be included. For 2486example, the AMD OpenCL runtime records kernel argument information. 2487 2488.. _amdgpu-amdhsa-code-object-metadata-v2: 2489 2490Code Object V2 Metadata 2491+++++++++++++++++++++++ 2492 2493.. warning:: 2494 Code object V2 is not the default code object version emitted by this version 2495 of LLVM. 2496 2497Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2498(see :ref:`amdgpu-note-records-v2`). 2499 2500The metadata is specified as a YAML formatted string (see [YAML]_ and 2501:doc:`YamlIO`). 2502 2503.. TODO:: 2504 2505 Is the string null terminated? It probably should not if YAML allows it to 2506 contain null characters, otherwise it should be. 2507 2508The metadata is represented as a single YAML document comprised of the mapping 2509defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2510referenced tables. 2511 2512For boolean values, the string values of ``false`` and ``true`` are used for 2513false and true respectively. 2514 2515Additional information can be added to the mappings. To avoid conflicts, any 2516non-AMD key names should be prefixed by "*vendor-name*.". 2517 2518 .. table:: AMDHSA Code Object V2 Metadata Map 2519 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2520 2521 ========== ============== ========= ======================================= 2522 String Key Value Type Required? Description 2523 ========== ============== ========= ======================================= 2524 "Version" sequence of Required - The first integer is the major 2525 2 integers version. Currently 1. 2526 - The second integer is the minor 2527 version. Currently 0. 2528 "Printf" sequence of Each string is encoded information 2529 strings about a printf function call. The 2530 encoded information is organized as 2531 fields separated by colon (':'): 2532 2533 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2534 2535 where: 2536 2537 ``ID`` 2538 A 32-bit integer as a unique id for 2539 each printf function call 2540 2541 ``N`` 2542 A 32-bit integer equal to the number 2543 of arguments of printf function call 2544 minus 1 2545 2546 ``S[i]`` (where i = 0, 1, ... , N-1) 2547 32-bit integers for the size in bytes 2548 of the i-th FormatString argument of 2549 the printf function call 2550 2551 FormatString 2552 The format string passed to the 2553 printf function call. 2554 "Kernels" sequence of Required Sequence of the mappings for each 2555 mapping kernel in the code object. See 2556 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2557 for the definition of the mapping. 2558 ========== ============== ========= ======================================= 2559 2560.. 2561 2562 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2563 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2564 2565 ================= ============== ========= ================================ 2566 String Key Value Type Required? Description 2567 ================= ============== ========= ================================ 2568 "Name" string Required Source name of the kernel. 2569 "SymbolName" string Required Name of the kernel 2570 descriptor ELF symbol. 2571 "Language" string Source language of the kernel. 2572 Values include: 2573 2574 - "OpenCL C" 2575 - "OpenCL C++" 2576 - "HCC" 2577 - "OpenMP" 2578 2579 "LanguageVersion" sequence of - The first integer is the major 2580 2 integers version. 2581 - The second integer is the 2582 minor version. 2583 "Attrs" mapping Mapping of kernel attributes. 2584 See 2585 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2586 for the mapping definition. 2587 "Args" sequence of Sequence of mappings of the 2588 mapping kernel arguments. See 2589 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2590 for the definition of the mapping. 2591 "CodeProps" mapping Mapping of properties related to 2592 the kernel code. See 2593 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2594 for the mapping definition. 2595 ================= ============== ========= ================================ 2596 2597.. 2598 2599 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2600 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2601 2602 =================== ============== ========= ============================== 2603 String Key Value Type Required? Description 2604 =================== ============== ========= ============================== 2605 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2606 3 integers must be >=1 and the dispatch 2607 work-group size X, Y, Z must 2608 correspond to the specified 2609 values. Defaults to 0, 0, 0. 2610 2611 Corresponds to the OpenCL 2612 ``reqd_work_group_size`` 2613 attribute. 2614 "WorkGroupSizeHint" sequence of The dispatch work-group size 2615 3 integers X, Y, Z is likely to be the 2616 specified values. 2617 2618 Corresponds to the OpenCL 2619 ``work_group_size_hint`` 2620 attribute. 2621 "VecTypeHint" string The name of a scalar or vector 2622 type. 2623 2624 Corresponds to the OpenCL 2625 ``vec_type_hint`` attribute. 2626 2627 "RuntimeHandle" string The external symbol name 2628 associated with a kernel. 2629 OpenCL runtime allocates a 2630 global buffer for the symbol 2631 and saves the kernel's address 2632 to it, which is used for 2633 device side enqueueing. Only 2634 available for device side 2635 enqueued kernels. 2636 =================== ============== ========= ============================== 2637 2638.. 2639 2640 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2641 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2642 2643 ================= ============== ========= ================================ 2644 String Key Value Type Required? Description 2645 ================= ============== ========= ================================ 2646 "Name" string Kernel argument name. 2647 "TypeName" string Kernel argument type name. 2648 "Size" integer Required Kernel argument size in bytes. 2649 "Align" integer Required Kernel argument alignment in 2650 bytes. Must be a power of two. 2651 "ValueKind" string Required Kernel argument kind that 2652 specifies how to set up the 2653 corresponding argument. 2654 Values include: 2655 2656 "ByValue" 2657 The argument is copied 2658 directly into the kernarg. 2659 2660 "GlobalBuffer" 2661 A global address space pointer 2662 to the buffer data is passed 2663 in the kernarg. 2664 2665 "DynamicSharedPointer" 2666 A group address space pointer 2667 to dynamically allocated LDS 2668 is passed in the kernarg. 2669 2670 "Sampler" 2671 A global address space 2672 pointer to a S# is passed in 2673 the kernarg. 2674 2675 "Image" 2676 A global address space 2677 pointer to a T# is passed in 2678 the kernarg. 2679 2680 "Pipe" 2681 A global address space pointer 2682 to an OpenCL pipe is passed in 2683 the kernarg. 2684 2685 "Queue" 2686 A global address space pointer 2687 to an OpenCL device enqueue 2688 queue is passed in the 2689 kernarg. 2690 2691 "HiddenGlobalOffsetX" 2692 The OpenCL grid dispatch 2693 global offset for the X 2694 dimension is passed in the 2695 kernarg. 2696 2697 "HiddenGlobalOffsetY" 2698 The OpenCL grid dispatch 2699 global offset for the Y 2700 dimension is passed in the 2701 kernarg. 2702 2703 "HiddenGlobalOffsetZ" 2704 The OpenCL grid dispatch 2705 global offset for the Z 2706 dimension is passed in the 2707 kernarg. 2708 2709 "HiddenNone" 2710 An argument that is not used 2711 by the kernel. Space needs to 2712 be left for it, but it does 2713 not need to be set up. 2714 2715 "HiddenPrintfBuffer" 2716 A global address space pointer 2717 to the runtime printf buffer 2718 is passed in kernarg. 2719 2720 "HiddenHostcallBuffer" 2721 A global address space pointer 2722 to the runtime hostcall buffer 2723 is passed in kernarg. 2724 2725 "HiddenDefaultQueue" 2726 A global address space pointer 2727 to the OpenCL device enqueue 2728 queue that should be used by 2729 the kernel by default is 2730 passed in the kernarg. 2731 2732 "HiddenCompletionAction" 2733 A global address space pointer 2734 to help link enqueued kernels into 2735 the ancestor tree for determining 2736 when the parent kernel has finished. 2737 2738 "HiddenMultiGridSyncArg" 2739 A global address space pointer for 2740 multi-grid synchronization is 2741 passed in the kernarg. 2742 2743 "ValueType" string Unused and deprecated. This should no longer 2744 be emitted, but is accepted for compatibility. 2745 2746 2747 "PointeeAlign" integer Alignment in bytes of pointee 2748 type for pointer type kernel 2749 argument. Must be a power 2750 of 2. Only present if 2751 "ValueKind" is 2752 "DynamicSharedPointer". 2753 "AddrSpaceQual" string Kernel argument address space 2754 qualifier. Only present if 2755 "ValueKind" is "GlobalBuffer" or 2756 "DynamicSharedPointer". Values 2757 are: 2758 2759 - "Private" 2760 - "Global" 2761 - "Constant" 2762 - "Local" 2763 - "Generic" 2764 - "Region" 2765 2766 .. TODO:: 2767 2768 Is GlobalBuffer only Global 2769 or Constant? Is 2770 DynamicSharedPointer always 2771 Local? Can HCC allow Generic? 2772 How can Private or Region 2773 ever happen? 2774 2775 "AccQual" string Kernel argument access 2776 qualifier. Only present if 2777 "ValueKind" is "Image" or 2778 "Pipe". Values 2779 are: 2780 2781 - "ReadOnly" 2782 - "WriteOnly" 2783 - "ReadWrite" 2784 2785 .. TODO:: 2786 2787 Does this apply to 2788 GlobalBuffer? 2789 2790 "ActualAccQual" string The actual memory accesses 2791 performed by the kernel on the 2792 kernel argument. Only present if 2793 "ValueKind" is "GlobalBuffer", 2794 "Image", or "Pipe". This may be 2795 more restrictive than indicated 2796 by "AccQual" to reflect what the 2797 kernel actual does. If not 2798 present then the runtime must 2799 assume what is implied by 2800 "AccQual" and "IsConst". Values 2801 are: 2802 2803 - "ReadOnly" 2804 - "WriteOnly" 2805 - "ReadWrite" 2806 2807 "IsConst" boolean Indicates if the kernel argument 2808 is const qualified. Only present 2809 if "ValueKind" is 2810 "GlobalBuffer". 2811 2812 "IsRestrict" boolean Indicates if the kernel argument 2813 is restrict qualified. Only 2814 present if "ValueKind" is 2815 "GlobalBuffer". 2816 2817 "IsVolatile" boolean Indicates if the kernel argument 2818 is volatile qualified. Only 2819 present if "ValueKind" is 2820 "GlobalBuffer". 2821 2822 "IsPipe" boolean Indicates if the kernel argument 2823 is pipe qualified. Only present 2824 if "ValueKind" is "Pipe". 2825 2826 .. TODO:: 2827 2828 Can GlobalBuffer be pipe 2829 qualified? 2830 2831 ================= ============== ========= ================================ 2832 2833.. 2834 2835 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2836 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2837 2838 ============================ ============== ========= ===================== 2839 String Key Value Type Required? Description 2840 ============================ ============== ========= ===================== 2841 "KernargSegmentSize" integer Required The size in bytes of 2842 the kernarg segment 2843 that holds the values 2844 of the arguments to 2845 the kernel. 2846 "GroupSegmentFixedSize" integer Required The amount of group 2847 segment memory 2848 required by a 2849 work-group in 2850 bytes. This does not 2851 include any 2852 dynamically allocated 2853 group segment memory 2854 that may be added 2855 when the kernel is 2856 dispatched. 2857 "PrivateSegmentFixedSize" integer Required The amount of fixed 2858 private address space 2859 memory required for a 2860 work-item in 2861 bytes. If the kernel 2862 uses a dynamic call 2863 stack then additional 2864 space must be added 2865 to this value for the 2866 call stack. 2867 "KernargSegmentAlign" integer Required The maximum byte 2868 alignment of 2869 arguments in the 2870 kernarg segment. Must 2871 be a power of 2. 2872 "WavefrontSize" integer Required Wavefront size. Must 2873 be a power of 2. 2874 "NumSGPRs" integer Required Number of scalar 2875 registers used by a 2876 wavefront for 2877 GFX6-GFX10. This 2878 includes the special 2879 SGPRs for VCC, Flat 2880 Scratch (GFX7-GFX10) 2881 and XNACK (for 2882 GFX8-GFX10). It does 2883 not include the 16 2884 SGPR added if a trap 2885 handler is 2886 enabled. It is not 2887 rounded up to the 2888 allocation 2889 granularity. 2890 "NumVGPRs" integer Required Number of vector 2891 registers used by 2892 each work-item for 2893 GFX6-GFX10 2894 "MaxFlatWorkGroupSize" integer Required Maximum flat 2895 work-group size 2896 supported by the 2897 kernel in work-items. 2898 Must be >=1 and 2899 consistent with 2900 ReqdWorkGroupSize if 2901 not 0, 0, 0. 2902 "NumSpilledSGPRs" integer Number of stores from 2903 a scalar register to 2904 a register allocator 2905 created spill 2906 location. 2907 "NumSpilledVGPRs" integer Number of stores from 2908 a vector register to 2909 a register allocator 2910 created spill 2911 location. 2912 ============================ ============== ========= ===================== 2913 2914.. _amdgpu-amdhsa-code-object-metadata-v3: 2915 2916Code Object V3 Metadata 2917+++++++++++++++++++++++ 2918 2919Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2920record (see :ref:`amdgpu-note-records-v3-v4`). 2921 2922The metadata is represented as Message Pack formatted binary data (see 2923[MsgPack]_). The top level is a Message Pack map that includes the 2924keys defined in table 2925:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2926tables. 2927 2928Additional information can be added to the maps. To avoid conflicts, 2929any key names should be prefixed by "*vendor-name*." where 2930``vendor-name`` can be the name of the vendor and specific vendor 2931tool that generates the information. The prefix is abbreviated to 2932simply "." when it appears within a map that has been added by the 2933same *vendor-name*. 2934 2935 .. table:: AMDHSA Code Object V3 Metadata Map 2936 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2937 2938 ================= ============== ========= ======================================= 2939 String Key Value Type Required? Description 2940 ================= ============== ========= ======================================= 2941 "amdhsa.version" sequence of Required - The first integer is the major 2942 2 integers version. Currently 1. 2943 - The second integer is the minor 2944 version. Currently 0. 2945 "amdhsa.printf" sequence of Each string is encoded information 2946 strings about a printf function call. The 2947 encoded information is organized as 2948 fields separated by colon (':'): 2949 2950 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2951 2952 where: 2953 2954 ``ID`` 2955 A 32-bit integer as a unique id for 2956 each printf function call 2957 2958 ``N`` 2959 A 32-bit integer equal to the number 2960 of arguments of printf function call 2961 minus 1 2962 2963 ``S[i]`` (where i = 0, 1, ... , N-1) 2964 32-bit integers for the size in bytes 2965 of the i-th FormatString argument of 2966 the printf function call 2967 2968 FormatString 2969 The format string passed to the 2970 printf function call. 2971 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2972 map kernel in the code object. See 2973 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2974 for the definition of the keys included 2975 in that map. 2976 ================= ============== ========= ======================================= 2977 2978.. 2979 2980 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2981 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2982 2983 =================================== ============== ========= ================================ 2984 String Key Value Type Required? Description 2985 =================================== ============== ========= ================================ 2986 ".name" string Required Source name of the kernel. 2987 ".symbol" string Required Name of the kernel 2988 descriptor ELF symbol. 2989 ".language" string Source language of the kernel. 2990 Values include: 2991 2992 - "OpenCL C" 2993 - "OpenCL C++" 2994 - "HCC" 2995 - "HIP" 2996 - "OpenMP" 2997 - "Assembler" 2998 2999 ".language_version" sequence of - The first integer is the major 3000 2 integers version. 3001 - The second integer is the 3002 minor version. 3003 ".args" sequence of Sequence of maps of the 3004 map kernel arguments. See 3005 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3006 for the definition of the keys 3007 included in that map. 3008 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3009 3 integers must be >=1 and the dispatch 3010 work-group size X, Y, Z must 3011 correspond to the specified 3012 values. Defaults to 0, 0, 0. 3013 3014 Corresponds to the OpenCL 3015 ``reqd_work_group_size`` 3016 attribute. 3017 ".workgroup_size_hint" sequence of The dispatch work-group size 3018 3 integers X, Y, Z is likely to be the 3019 specified values. 3020 3021 Corresponds to the OpenCL 3022 ``work_group_size_hint`` 3023 attribute. 3024 ".vec_type_hint" string The name of a scalar or vector 3025 type. 3026 3027 Corresponds to the OpenCL 3028 ``vec_type_hint`` attribute. 3029 3030 ".device_enqueue_symbol" string The external symbol name 3031 associated with a kernel. 3032 OpenCL runtime allocates a 3033 global buffer for the symbol 3034 and saves the kernel's address 3035 to it, which is used for 3036 device side enqueueing. Only 3037 available for device side 3038 enqueued kernels. 3039 ".kernarg_segment_size" integer Required The size in bytes of 3040 the kernarg segment 3041 that holds the values 3042 of the arguments to 3043 the kernel. 3044 ".group_segment_fixed_size" integer Required The amount of group 3045 segment memory 3046 required by a 3047 work-group in 3048 bytes. This does not 3049 include any 3050 dynamically allocated 3051 group segment memory 3052 that may be added 3053 when the kernel is 3054 dispatched. 3055 ".private_segment_fixed_size" integer Required The amount of fixed 3056 private address space 3057 memory required for a 3058 work-item in 3059 bytes. If the kernel 3060 uses a dynamic call 3061 stack then additional 3062 space must be added 3063 to this value for the 3064 call stack. 3065 ".kernarg_segment_align" integer Required The maximum byte 3066 alignment of 3067 arguments in the 3068 kernarg segment. Must 3069 be a power of 2. 3070 ".wavefront_size" integer Required Wavefront size. Must 3071 be a power of 2. 3072 ".sgpr_count" integer Required Number of scalar 3073 registers required by a 3074 wavefront for 3075 GFX6-GFX9. A register 3076 is required if it is 3077 used explicitly, or 3078 if a higher numbered 3079 register is used 3080 explicitly. This 3081 includes the special 3082 SGPRs for VCC, Flat 3083 Scratch (GFX7-GFX9) 3084 and XNACK (for 3085 GFX8-GFX9). It does 3086 not include the 16 3087 SGPR added if a trap 3088 handler is 3089 enabled. It is not 3090 rounded up to the 3091 allocation 3092 granularity. 3093 ".vgpr_count" integer Required Number of vector 3094 registers required by 3095 each work-item for 3096 GFX6-GFX9. A register 3097 is required if it is 3098 used explicitly, or 3099 if a higher numbered 3100 register is used 3101 explicitly. 3102 ".max_flat_workgroup_size" integer Required Maximum flat 3103 work-group size 3104 supported by the 3105 kernel in work-items. 3106 Must be >=1 and 3107 consistent with 3108 ReqdWorkGroupSize if 3109 not 0, 0, 0. 3110 ".sgpr_spill_count" integer Number of stores from 3111 a scalar register to 3112 a register allocator 3113 created spill 3114 location. 3115 ".vgpr_spill_count" integer Number of stores from 3116 a vector register to 3117 a register allocator 3118 created spill 3119 location. 3120 =================================== ============== ========= ================================ 3121 3122.. 3123 3124 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3125 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3126 3127 ====================== ============== ========= ================================ 3128 String Key Value Type Required? Description 3129 ====================== ============== ========= ================================ 3130 ".name" string Kernel argument name. 3131 ".type_name" string Kernel argument type name. 3132 ".size" integer Required Kernel argument size in bytes. 3133 ".offset" integer Required Kernel argument offset in 3134 bytes. The offset must be a 3135 multiple of the alignment 3136 required by the argument. 3137 ".value_kind" string Required Kernel argument kind that 3138 specifies how to set up the 3139 corresponding argument. 3140 Values include: 3141 3142 "by_value" 3143 The argument is copied 3144 directly into the kernarg. 3145 3146 "global_buffer" 3147 A global address space pointer 3148 to the buffer data is passed 3149 in the kernarg. 3150 3151 "dynamic_shared_pointer" 3152 A group address space pointer 3153 to dynamically allocated LDS 3154 is passed in the kernarg. 3155 3156 "sampler" 3157 A global address space 3158 pointer to a S# is passed in 3159 the kernarg. 3160 3161 "image" 3162 A global address space 3163 pointer to a T# is passed in 3164 the kernarg. 3165 3166 "pipe" 3167 A global address space pointer 3168 to an OpenCL pipe is passed in 3169 the kernarg. 3170 3171 "queue" 3172 A global address space pointer 3173 to an OpenCL device enqueue 3174 queue is passed in the 3175 kernarg. 3176 3177 "hidden_global_offset_x" 3178 The OpenCL grid dispatch 3179 global offset for the X 3180 dimension is passed in the 3181 kernarg. 3182 3183 "hidden_global_offset_y" 3184 The OpenCL grid dispatch 3185 global offset for the Y 3186 dimension is passed in the 3187 kernarg. 3188 3189 "hidden_global_offset_z" 3190 The OpenCL grid dispatch 3191 global offset for the Z 3192 dimension is passed in the 3193 kernarg. 3194 3195 "hidden_none" 3196 An argument that is not used 3197 by the kernel. Space needs to 3198 be left for it, but it does 3199 not need to be set up. 3200 3201 "hidden_printf_buffer" 3202 A global address space pointer 3203 to the runtime printf buffer 3204 is passed in kernarg. 3205 3206 "hidden_hostcall_buffer" 3207 A global address space pointer 3208 to the runtime hostcall buffer 3209 is passed in kernarg. 3210 3211 "hidden_default_queue" 3212 A global address space pointer 3213 to the OpenCL device enqueue 3214 queue that should be used by 3215 the kernel by default is 3216 passed in the kernarg. 3217 3218 "hidden_completion_action" 3219 A global address space pointer 3220 to help link enqueued kernels into 3221 the ancestor tree for determining 3222 when the parent kernel has finished. 3223 3224 "hidden_multigrid_sync_arg" 3225 A global address space pointer for 3226 multi-grid synchronization is 3227 passed in the kernarg. 3228 3229 ".value_type" string Unused and deprecated. This should no longer 3230 be emitted, but is accepted for compatibility. 3231 3232 ".pointee_align" integer Alignment in bytes of pointee 3233 type for pointer type kernel 3234 argument. Must be a power 3235 of 2. Only present if 3236 ".value_kind" is 3237 "dynamic_shared_pointer". 3238 ".address_space" string Kernel argument address space 3239 qualifier. Only present if 3240 ".value_kind" is "global_buffer" or 3241 "dynamic_shared_pointer". Values 3242 are: 3243 3244 - "private" 3245 - "global" 3246 - "constant" 3247 - "local" 3248 - "generic" 3249 - "region" 3250 3251 .. TODO:: 3252 3253 Is "global_buffer" only "global" 3254 or "constant"? Is 3255 "dynamic_shared_pointer" always 3256 "local"? Can HCC allow "generic"? 3257 How can "private" or "region" 3258 ever happen? 3259 3260 ".access" string Kernel argument access 3261 qualifier. Only present if 3262 ".value_kind" is "image" or 3263 "pipe". Values 3264 are: 3265 3266 - "read_only" 3267 - "write_only" 3268 - "read_write" 3269 3270 .. TODO:: 3271 3272 Does this apply to 3273 "global_buffer"? 3274 3275 ".actual_access" string The actual memory accesses 3276 performed by the kernel on the 3277 kernel argument. Only present if 3278 ".value_kind" is "global_buffer", 3279 "image", or "pipe". This may be 3280 more restrictive than indicated 3281 by ".access" to reflect what the 3282 kernel actual does. If not 3283 present then the runtime must 3284 assume what is implied by 3285 ".access" and ".is_const" . Values 3286 are: 3287 3288 - "read_only" 3289 - "write_only" 3290 - "read_write" 3291 3292 ".is_const" boolean Indicates if the kernel argument 3293 is const qualified. Only present 3294 if ".value_kind" is 3295 "global_buffer". 3296 3297 ".is_restrict" boolean Indicates if the kernel argument 3298 is restrict qualified. Only 3299 present if ".value_kind" is 3300 "global_buffer". 3301 3302 ".is_volatile" boolean Indicates if the kernel argument 3303 is volatile qualified. Only 3304 present if ".value_kind" is 3305 "global_buffer". 3306 3307 ".is_pipe" boolean Indicates if the kernel argument 3308 is pipe qualified. Only present 3309 if ".value_kind" is "pipe". 3310 3311 .. TODO:: 3312 3313 Can "global_buffer" be pipe 3314 qualified? 3315 3316 ====================== ============== ========= ================================ 3317 3318.. _amdgpu-amdhsa-code-object-metadata-v4: 3319 3320Code Object V4 Metadata 3321+++++++++++++++++++++++ 3322 3323.. warning:: 3324 Code object V4 is not the default code object version emitted by this version 3325 of LLVM. 3326 3327Code object V4 metadata is the same as 3328:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3329defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3330 3331 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3332 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3333 3334 ================= ============== ========= ======================================= 3335 String Key Value Type Required? Description 3336 ================= ============== ========= ======================================= 3337 "amdhsa.version" sequence of Required - The first integer is the major 3338 2 integers version. Currently 1. 3339 - The second integer is the minor 3340 version. Currently 1. 3341 "amdhsa.target" string Required The target name of the code using the syntax: 3342 3343 .. code:: 3344 3345 <target-triple> [ "-" <target-id> ] 3346 3347 A canonical target ID must be 3348 used. See :ref:`amdgpu-target-triples` 3349 and :ref:`amdgpu-target-id`. 3350 ================= ============== ========= ======================================= 3351 3352.. 3353 3354Kernel Dispatch 3355~~~~~~~~~~~~~~~ 3356 3357The HSA architected queuing language (AQL) defines a user space memory interface 3358that can be used to control the dispatch of kernels, in an agent independent 3359way. An agent can have zero or more AQL queues created for it using an HSA 3360compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3361are 64 bytes) can be placed. See the *HSA Platform System Architecture 3362Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3363 3364The packet processor of a kernel agent is responsible for detecting and 3365dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3366packet processor is implemented by the hardware command processor (CP), 3367asynchronous dispatch controller (ADC) and shader processor input controller 3368(SPI). 3369 3370An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3371the kernel mode driver to initialize and register the AQL queue with CP. 3372 3373To dispatch a kernel the following actions are performed. This can occur in the 3374CPU host program, or from an HSA kernel executing on a GPU. 3375 33761. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3377 executed is obtained. 33782. A pointer to the kernel descriptor (see 3379 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3380 It must be for a kernel that is contained in a code object that that was 3381 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3382 queue is associated. 33833. Space is allocated for the kernel arguments using the HSA compatible runtime 3384 allocator for a memory region with the kernarg property for the kernel agent 3385 that will execute the kernel. It must be at least 16-byte aligned. 33864. Kernel argument values are assigned to the kernel argument memory 3387 allocation. The layout is defined in the *HSA Programmer's Language 3388 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3389 kernel argument memory in the same way constant memory is accessed. (Note 3390 that the HSA specification allows an implementation to copy the kernel 3391 argument contents to another location that is accessed by the kernel.) 33925. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3393 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3394 for the packet. The packet must be set up, and the final write must use an 3395 atomic store release to set the packet kind to ensure the packet contents are 3396 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3397 notify the kernel agent that the AQL queue has been updated. These rules, and 3398 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3399 System Architecture Specification* [HSA]_. 34006. A kernel dispatch packet includes information about the actual dispatch, 3401 such as grid and work-group size, together with information from the code 3402 object about the kernel, such as segment sizes. The HSA compatible runtime 3403 queries on the kernel symbol can be used to obtain the code object values 3404 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 34057. CP executes micro-code and is responsible for detecting and setting up the 3406 GPU to execute the wavefronts of a kernel dispatch. 34078. CP ensures that when the a wavefront starts executing the kernel machine 3408 code, the scalar general purpose registers (SGPR) and vector general purpose 3409 registers (VGPR) are set up as required by the machine code. The required 3410 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3411 register state is defined in 3412 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 34139. The prolog of the kernel machine code (see 3414 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3415 before continuing executing the machine code that corresponds to the kernel. 341610. When the kernel dispatch has completed execution, CP signals the completion 3417 signal specified in the kernel dispatch packet if not 0. 3418 3419.. _amdgpu-amdhsa-memory-spaces: 3420 3421Memory Spaces 3422~~~~~~~~~~~~~ 3423 3424The memory space properties are: 3425 3426 .. table:: AMDHSA Memory Spaces 3427 :name: amdgpu-amdhsa-memory-spaces-table 3428 3429 ================= =========== ======== ======= ================== 3430 Memory Space Name HSA Segment Hardware Address NULL Value 3431 Name Name Size 3432 ================= =========== ======== ======= ================== 3433 Private private scratch 32 0x00000000 3434 Local group LDS 32 0xFFFFFFFF 3435 Global global global 64 0x0000000000000000 3436 Constant constant *same as 64 0x0000000000000000 3437 global* 3438 Generic flat flat 64 0x0000000000000000 3439 Region N/A GDS 32 *not implemented 3440 for AMDHSA* 3441 ================= =========== ======== ======= ================== 3442 3443The global and constant memory spaces both use global virtual addresses, which 3444are the same virtual address space used by the CPU. However, some virtual 3445addresses may only be accessible to the CPU, some only accessible by the GPU, 3446and some by both. 3447 3448Using the constant memory space indicates that the data will not change during 3449the execution of the kernel. This allows scalar read instructions to be 3450used. The vector and scalar L1 caches are invalidated of volatile data before 3451each kernel dispatch execution to allow constant memory to change values between 3452kernel dispatches. 3453 3454The local memory space uses the hardware Local Data Store (LDS) which is 3455automatically allocated when the hardware creates work-groups of wavefronts, and 3456freed when all the wavefronts of a work-group have terminated. The data store 3457(DS) instructions can be used to access it. 3458 3459The private memory space uses the hardware scratch memory support. If the kernel 3460uses scratch, then the hardware allocates memory that is accessed using 3461wavefront lane dword (4 byte) interleaving. The mapping used from private 3462address to physical address is: 3463 3464 ``wavefront-scratch-base + 3465 (private-address * wavefront-size * 4) + 3466 (wavefront-lane-id * 4)`` 3467 3468There are different ways that the wavefront scratch base address is determined 3469by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3470memory can be accessed in an interleaved manner using buffer instruction with 3471the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3472instructions, or by flat instructions. If each lane of a wavefront accesses the 3473same private address, the interleaving results in adjacent dwords being accessed 3474and hence requires fewer cache lines to be fetched. Multi-dword access is not 3475supported except by flat and scratch instructions in GFX9-GFX10. 3476 3477The generic address space uses the hardware flat address support available in 3478GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3479local apertures), that are outside the range of addressible global memory, to 3480map from a flat address to a private or local address. 3481 3482FLAT instructions can take a flat address and access global, private (scratch) 3483and group (LDS) memory depending in if the address is within one of the 3484aperture ranges. Flat access to scratch requires hardware aperture setup and 3485setup in the kernel prologue (see 3486:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3487hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3488:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3489 3490To convert between a segment address and a flat address the base address of the 3491apertures address can be used. For GFX7-GFX8 these are available in the 3492:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3493Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3494GFX9-GFX10 the aperture base addresses are directly available as inline constant 3495registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3496address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3497which makes it easier to convert from flat to segment or segment to flat. 3498 3499Image and Samplers 3500~~~~~~~~~~~~~~~~~~ 3501 3502Image and sample handles created by an HSA compatible runtime (see 3503:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3504object respectively. In order to support the HSA ``query_sampler`` operations 3505two extra dwords are used to store the HSA BRIG enumeration values for the 3506queries that are not trivially deducible from the S# representation. 3507 3508HSA Signals 3509~~~~~~~~~~~ 3510 3511HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3512are 64-bit addresses of a structure allocated in memory accessible from both the 3513CPU and GPU. The structure is defined by the runtime and subject to change 3514between releases. For example, see [AMD-ROCm-github]_. 3515 3516.. _amdgpu-amdhsa-hsa-aql-queue: 3517 3518HSA AQL Queue 3519~~~~~~~~~~~~~ 3520 3521The HSA AQL queue structure is defined by an HSA compatible runtime (see 3522:ref:`amdgpu-os`) and subject to change between releases. For example, see 3523[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3524certain language features such as the flat address aperture bases. It also 3525contains fields used by CP such as managing the allocation of scratch memory. 3526 3527.. _amdgpu-amdhsa-kernel-descriptor: 3528 3529Kernel Descriptor 3530~~~~~~~~~~~~~~~~~ 3531 3532A kernel descriptor consists of the information needed by CP to initiate the 3533execution of a kernel, including the entry point address of the machine code 3534that implements the kernel. 3535 3536Code Object V3 Kernel Descriptor 3537++++++++++++++++++++++++++++++++ 3538 3539CP microcode requires the Kernel descriptor to be allocated on 64-byte 3540alignment. 3541 3542The fields used by CP for code objects before V3 also match those specified in 3543:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3544 3545 .. table:: Code Object V3 Kernel Descriptor 3546 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3547 3548 ======= ======= =============================== ============================ 3549 Bits Size Field Name Description 3550 ======= ======= =============================== ============================ 3551 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3552 address space memory 3553 required for a work-group 3554 in bytes. This does not 3555 include any dynamically 3556 allocated local address 3557 space memory that may be 3558 added when the kernel is 3559 dispatched. 3560 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3561 private address space 3562 memory required for a 3563 work-item in bytes. 3564 Additional space may need to 3565 be added to this value if 3566 the call stack has 3567 non-inlined function calls. 3568 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3569 memory pointed to by the 3570 AQL dispatch packet. The 3571 kernarg memory is used to 3572 pass arguments to the 3573 kernel. 3574 3575 * If the kernarg pointer in 3576 the dispatch packet is NULL 3577 then there are no kernel 3578 arguments. 3579 * If the kernarg pointer in 3580 the dispatch packet is 3581 not NULL and this value 3582 is 0 then the kernarg 3583 memory size is 3584 unspecified. 3585 * If the kernarg pointer in 3586 the dispatch packet is 3587 not NULL and this value 3588 is not 0 then the value 3589 specifies the kernarg 3590 memory size in bytes. It 3591 is recommended to provide 3592 a value as it may be used 3593 by CP to optimize making 3594 the kernarg memory 3595 visible to the kernel 3596 code. 3597 3598 127:96 4 bytes Reserved, must be 0. 3599 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3600 negative) from base 3601 address of kernel 3602 descriptor to kernel's 3603 entry point instruction 3604 which must be 256 byte 3605 aligned. 3606 351:272 20 Reserved, must be 0. 3607 bytes 3608 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3609 Reserved, must be 0. 3610 GFX90A 3611 Compute Shader (CS) 3612 program settings used by 3613 CP to set up 3614 ``COMPUTE_PGM_RSRC3`` 3615 configuration 3616 register. See 3617 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3618 GFX10 3619 Compute Shader (CS) 3620 program settings used by 3621 CP to set up 3622 ``COMPUTE_PGM_RSRC3`` 3623 configuration 3624 register. See 3625 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3626 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3627 program settings used by 3628 CP to set up 3629 ``COMPUTE_PGM_RSRC1`` 3630 configuration 3631 register. See 3632 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3633 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3634 program settings used by 3635 CP to set up 3636 ``COMPUTE_PGM_RSRC2`` 3637 configuration 3638 register. See 3639 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3640 458:448 7 bits *See separate bits below.* Enable the setup of the 3641 SGPR user data registers 3642 (see 3643 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3644 3645 The total number of SGPR 3646 user data registers 3647 requested must not exceed 3648 16 and match value in 3649 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3650 Any requests beyond 16 3651 will be ignored. 3652 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3653 _BUFFER 3654 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3655 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3656 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3657 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3658 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT 3659 3660 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3661 _SIZE 3662 457:455 3 bits Reserved, must be 0. 3663 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3664 Reserved, must be 0. 3665 GFX10 3666 - If 0 execute in 3667 wavefront size 64 mode. 3668 - If 1 execute in 3669 native wavefront size 3670 32 mode. 3671 463:459 1 bit Reserved, must be 0. 3672 464 1 bit RESERVED_464 Deprecated, must be 0. 3673 467:465 3 bits Reserved, must be 0. 3674 468 1 bit RESERVED_468 Deprecated, must be 0. 3675 469:471 3 bits Reserved, must be 0. 3676 511:472 5 bytes Reserved, must be 0. 3677 512 **Total size 64 bytes.** 3678 ======= ==================================================================== 3679 3680.. 3681 3682 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3683 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3684 3685 ======= ======= =============================== =========================================================================== 3686 Bits Size Field Name Description 3687 ======= ======= =============================== =========================================================================== 3688 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3689 blocks used by each work-item; 3690 granularity is device 3691 specific: 3692 3693 GFX6-GFX9 3694 - vgprs_used 0..256 3695 - max(0, ceil(vgprs_used / 4) - 1) 3696 GFX90A 3697 - vgprs_used 0..512 3698 - vgprs_used = align(arch_vgprs, 4) 3699 + acc_vgprs 3700 - max(0, ceil(vgprs_used / 8) - 1) 3701 GFX10 (wavefront size 64) 3702 - max_vgpr 1..256 3703 - max(0, ceil(vgprs_used / 4) - 1) 3704 GFX10 (wavefront size 32) 3705 - max_vgpr 1..256 3706 - max(0, ceil(vgprs_used / 8) - 1) 3707 3708 Where vgprs_used is defined 3709 as the highest VGPR number 3710 explicitly referenced plus 3711 one. 3712 3713 Used by CP to set up 3714 ``COMPUTE_PGM_RSRC1.VGPRS``. 3715 3716 The 3717 :ref:`amdgpu-assembler` 3718 calculates this 3719 automatically for the 3720 selected processor from 3721 values provided to the 3722 `.amdhsa_kernel` directive 3723 by the 3724 `.amdhsa_next_free_vgpr` 3725 nested directive (see 3726 :ref:`amdhsa-kernel-directives-table`). 3727 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3728 blocks used by a wavefront; 3729 granularity is device 3730 specific: 3731 3732 GFX6-GFX8 3733 - sgprs_used 0..112 3734 - max(0, ceil(sgprs_used / 8) - 1) 3735 GFX9 3736 - sgprs_used 0..112 3737 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3738 GFX10 3739 Reserved, must be 0. 3740 (128 SGPRs always 3741 allocated.) 3742 3743 Where sgprs_used is 3744 defined as the highest 3745 SGPR number explicitly 3746 referenced plus one, plus 3747 a target specific number 3748 of additional special 3749 SGPRs for VCC, 3750 FLAT_SCRATCH (GFX7+) and 3751 XNACK_MASK (GFX8+), and 3752 any additional 3753 target specific 3754 limitations. It does not 3755 include the 16 SGPRs added 3756 if a trap handler is 3757 enabled. 3758 3759 The target specific 3760 limitations and special 3761 SGPR layout are defined in 3762 the hardware 3763 documentation, which can 3764 be found in the 3765 :ref:`amdgpu-processors` 3766 table. 3767 3768 Used by CP to set up 3769 ``COMPUTE_PGM_RSRC1.SGPRS``. 3770 3771 The 3772 :ref:`amdgpu-assembler` 3773 calculates this 3774 automatically for the 3775 selected processor from 3776 values provided to the 3777 `.amdhsa_kernel` directive 3778 by the 3779 `.amdhsa_next_free_sgpr` 3780 and `.amdhsa_reserve_*` 3781 nested directives (see 3782 :ref:`amdhsa-kernel-directives-table`). 3783 11:10 2 bits PRIORITY Must be 0. 3784 3785 Start executing wavefront 3786 at the specified priority. 3787 3788 CP is responsible for 3789 filling in 3790 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3791 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3792 with specified rounding 3793 mode for single (32 3794 bit) floating point 3795 precision floating point 3796 operations. 3797 3798 Floating point rounding 3799 mode values are defined in 3800 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3801 3802 Used by CP to set up 3803 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3804 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3805 with specified rounding 3806 denorm mode for half/double (16 3807 and 64-bit) floating point 3808 precision floating point 3809 operations. 3810 3811 Floating point rounding 3812 mode values are defined in 3813 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3814 3815 Used by CP to set up 3816 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3817 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3818 with specified denorm mode 3819 for single (32 3820 bit) floating point 3821 precision floating point 3822 operations. 3823 3824 Floating point denorm mode 3825 values are defined in 3826 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3827 3828 Used by CP to set up 3829 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3830 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3831 with specified denorm mode 3832 for half/double (16 3833 and 64-bit) floating point 3834 precision floating point 3835 operations. 3836 3837 Floating point denorm mode 3838 values are defined in 3839 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3840 3841 Used by CP to set up 3842 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3843 20 1 bit PRIV Must be 0. 3844 3845 Start executing wavefront 3846 in privilege trap handler 3847 mode. 3848 3849 CP is responsible for 3850 filling in 3851 ``COMPUTE_PGM_RSRC1.PRIV``. 3852 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3853 with DX10 clamp mode 3854 enabled. Used by the vector 3855 ALU to force DX10 style 3856 treatment of NaN's (when 3857 set, clamp NaN to zero, 3858 otherwise pass NaN 3859 through). 3860 3861 Used by CP to set up 3862 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3863 22 1 bit DEBUG_MODE Must be 0. 3864 3865 Start executing wavefront 3866 in single step mode. 3867 3868 CP is responsible for 3869 filling in 3870 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3871 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3872 with IEEE mode 3873 enabled. Floating point 3874 opcodes that support 3875 exception flag gathering 3876 will quiet and propagate 3877 signaling-NaN inputs per 3878 IEEE 754-2008. Min_dx10 and 3879 max_dx10 become IEEE 3880 754-2008 compliant due to 3881 signaling-NaN propagation 3882 and quieting. 3883 3884 Used by CP to set up 3885 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3886 24 1 bit BULKY Must be 0. 3887 3888 Only one work-group allowed 3889 to execute on a compute 3890 unit. 3891 3892 CP is responsible for 3893 filling in 3894 ``COMPUTE_PGM_RSRC1.BULKY``. 3895 25 1 bit CDBG_USER Must be 0. 3896 3897 Flag that can be used to 3898 control debugging code. 3899 3900 CP is responsible for 3901 filling in 3902 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3903 26 1 bit FP16_OVFL GFX6-GFX8 3904 Reserved, must be 0. 3905 GFX9-GFX10 3906 Wavefront starts execution 3907 with specified fp16 overflow 3908 mode. 3909 3910 - If 0, fp16 overflow generates 3911 +/-INF values. 3912 - If 1, fp16 overflow that is the 3913 result of an +/-INF input value 3914 or divide by 0 produces a +/-INF, 3915 otherwise clamps computed 3916 overflow to +/-MAX_FP16 as 3917 appropriate. 3918 3919 Used by CP to set up 3920 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3921 28:27 2 bits Reserved, must be 0. 3922 29 1 bit WGP_MODE GFX6-GFX9 3923 Reserved, must be 0. 3924 GFX10 3925 - If 0 execute work-groups in 3926 CU wavefront execution mode. 3927 - If 1 execute work-groups on 3928 in WGP wavefront execution mode. 3929 3930 See :ref:`amdgpu-amdhsa-memory-model`. 3931 3932 Used by CP to set up 3933 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3934 30 1 bit MEM_ORDERED GFX6-GFX9 3935 Reserved, must be 0. 3936 GFX10 3937 Controls the behavior of the 3938 s_waitcnt's vmcnt and vscnt 3939 counters. 3940 3941 - If 0 vmcnt reports completion 3942 of load and atomic with return 3943 out of order with sample 3944 instructions, and the vscnt 3945 reports the completion of 3946 store and atomic without 3947 return in order. 3948 - If 1 vmcnt reports completion 3949 of load, atomic with return 3950 and sample instructions in 3951 order, and the vscnt reports 3952 the completion of store and 3953 atomic without return in order. 3954 3955 Used by CP to set up 3956 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3957 31 1 bit FWD_PROGRESS GFX6-GFX9 3958 Reserved, must be 0. 3959 GFX10 3960 - If 0 execute SIMD wavefronts 3961 using oldest first policy. 3962 - If 1 execute SIMD wavefronts to 3963 ensure wavefronts will make some 3964 forward progress. 3965 3966 Used by CP to set up 3967 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3968 32 **Total size 4 bytes** 3969 ======= =================================================================================================================== 3970 3971.. 3972 3973 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3974 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3975 3976 ======= ======= =============================== =========================================================================== 3977 Bits Size Field Name Description 3978 ======= ======= =============================== =========================================================================== 3979 0 1 bit ENABLE_PRIVATE_SEGMENT Enable the setup of the 3980 private segment. 3981 3982 In addition, enable the 3983 setup of the SGPR 3984 wavefront scratch offset 3985 system register (see 3986 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3987 3988 Used by CP to set up 3989 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3990 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3991 user data registers 3992 requested. This number must 3993 match the number of user 3994 data registers enabled. 3995 3996 Used by CP to set up 3997 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 3998 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 3999 4000 This bit represents 4001 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4002 which is set by the CP if 4003 the runtime has installed a 4004 trap handler. 4005 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4006 system SGPR register for 4007 the work-group id in the X 4008 dimension (see 4009 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4010 4011 Used by CP to set up 4012 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4013 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4014 system SGPR register for 4015 the work-group id in the Y 4016 dimension (see 4017 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4018 4019 Used by CP to set up 4020 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4021 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4022 system SGPR register for 4023 the work-group id in the Z 4024 dimension (see 4025 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4026 4027 Used by CP to set up 4028 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4029 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4030 system SGPR register for 4031 work-group information (see 4032 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4033 4034 Used by CP to set up 4035 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4036 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4037 VGPR system registers used 4038 for the work-item ID. 4039 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4040 defines the values. 4041 4042 Used by CP to set up 4043 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4044 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4045 4046 Wavefront starts execution 4047 with address watch 4048 exceptions enabled which 4049 are generated when L1 has 4050 witnessed a thread access 4051 an *address of 4052 interest*. 4053 4054 CP is responsible for 4055 filling in the address 4056 watch bit in 4057 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4058 according to what the 4059 runtime requests. 4060 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4061 4062 Wavefront starts execution 4063 with memory violation 4064 exceptions exceptions 4065 enabled which are generated 4066 when a memory violation has 4067 occurred for this wavefront from 4068 L1 or LDS 4069 (write-to-read-only-memory, 4070 mis-aligned atomic, LDS 4071 address out of range, 4072 illegal address, etc.). 4073 4074 CP sets the memory 4075 violation bit in 4076 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4077 according to what the 4078 runtime requests. 4079 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4080 4081 CP uses the rounded value 4082 from the dispatch packet, 4083 not this value, as the 4084 dispatch may contain 4085 dynamically allocated group 4086 segment memory. CP writes 4087 directly to 4088 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4089 4090 Amount of group segment 4091 (LDS) to allocate for each 4092 work-group. Granularity is 4093 device specific: 4094 4095 GFX6: 4096 roundup(lds-size / (64 * 4)) 4097 GFX7-GFX10: 4098 roundup(lds-size / (128 * 4)) 4099 4100 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4101 _INVALID_OPERATION with specified exceptions 4102 enabled. 4103 4104 Used by CP to set up 4105 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4106 (set from bits 0..6). 4107 4108 IEEE 754 FP Invalid 4109 Operation 4110 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4111 _SOURCE input operands is a 4112 denormal number 4113 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4114 _DIVISION_BY_ZERO Zero 4115 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4116 _OVERFLOW 4117 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4118 _UNDERFLOW 4119 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4120 _INEXACT 4121 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4122 _ZERO (rcp_iflag_f32 instruction 4123 only) 4124 31 1 bit Reserved, must be 0. 4125 32 **Total size 4 bytes.** 4126 ======= =================================================================================================================== 4127 4128.. 4129 4130 .. table:: compute_pgm_rsrc3 for GFX90A 4131 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4132 4133 ======= ======= =============================== =========================================================================== 4134 Bits Size Field Name Description 4135 ======= ======= =============================== =========================================================================== 4136 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4137 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4138 63 - accum-offset = 256. 4139 6:15 10 Reserved, must be 0. 4140 bits 4141 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4142 launched in the same CU. 4143 - If 1 the waves of a work-group can be 4144 launched in different CUs. The waves 4145 cannot use S_BARRIER or LDS. 4146 17:31 15 Reserved, must be 0. 4147 bits 4148 32 **Total size 4 bytes.** 4149 ======= =================================================================================================================== 4150 4151.. 4152 4153 .. table:: compute_pgm_rsrc3 for GFX10 4154 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4155 4156 ======= ======= =============================== =========================================================================== 4157 Bits Size Field Name Description 4158 ======= ======= =============================== =========================================================================== 4159 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4160 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4161 31:4 28 Reserved, must be 0. 4162 bits 4163 32 **Total size 4 bytes.** 4164 ======= =================================================================================================================== 4165 4166.. 4167 4168 .. table:: Floating Point Rounding Mode Enumeration Values 4169 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4170 4171 ====================================== ===== ============================== 4172 Enumeration Name Value Description 4173 ====================================== ===== ============================== 4174 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4175 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4176 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4177 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4178 ====================================== ===== ============================== 4179 4180.. 4181 4182 .. table:: Floating Point Denorm Mode Enumeration Values 4183 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4184 4185 ====================================== ===== ============================== 4186 Enumeration Name Value Description 4187 ====================================== ===== ============================== 4188 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4189 Denorms 4190 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4191 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4192 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4193 ====================================== ===== ============================== 4194 4195.. 4196 4197 .. table:: System VGPR Work-Item ID Enumeration Values 4198 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4199 4200 ======================================== ===== ============================ 4201 Enumeration Name Value Description 4202 ======================================== ===== ============================ 4203 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4204 ID. 4205 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4206 dimensions ID. 4207 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4208 dimensions ID. 4209 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4210 ======================================== ===== ============================ 4211 4212.. _amdgpu-amdhsa-initial-kernel-execution-state: 4213 4214Initial Kernel Execution State 4215~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4216 4217This section defines the register state that will be set up by the packet 4218processor prior to the start of execution of every wavefront. This is limited by 4219the constraints of the hardware controllers of CP/ADC/SPI. 4220 4221The order of the SGPR registers is defined, but the compiler can specify which 4222ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4223fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4224for enabled registers are dense starting at SGPR0: the first enabled register is 4225SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4226an SGPR number. 4227 4228The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4229all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4230using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4231actually initialized. These are then immediately followed by the System SGPRs 4232that are set up by ADC/SPI and can have different values for each wavefront of 4233the grid dispatch. 4234 4235SGPR register initial state is defined in 4236:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4237 4238 .. table:: SGPR Register Set Up Order 4239 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4240 4241 ========== ========================== ====== ============================== 4242 SGPR Order Name Number Description 4243 (kernel descriptor enable of 4244 field) SGPRs 4245 ========== ========================== ====== ============================== 4246 First Private Segment Buffer 4 See 4247 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4248 _segment_buffer) 4249 then Dispatch Ptr 2 64-bit address of AQL dispatch 4250 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4251 actually executing. 4252 then Queue Ptr 2 64-bit address of amd_queue_t 4253 (enable_sgpr_queue_ptr) object for AQL queue on which 4254 the dispatch packet was 4255 queued. 4256 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4257 (enable_sgpr_kernarg segment. This is directly 4258 _segment_ptr) copied from the 4259 kernarg_address in the kernel 4260 dispatch packet. 4261 4262 Having CP load it once avoids 4263 loading it at the beginning of 4264 every wavefront. 4265 then Dispatch Id 2 64-bit Dispatch ID of the 4266 (enable_sgpr_dispatch_id) dispatch packet being 4267 executed. 4268 then Flat Scratch Init 2 See 4269 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4270 then Private Segment Size 1 The 32-bit byte size of a 4271 (enable_sgpr_private single 4272 work-item's 4273 scratch_segment_size) memory 4274 allocation. This is the 4275 value from the kernel 4276 dispatch packet Private 4277 Segment Byte Size rounded up 4278 by CP to a multiple of 4279 DWORD. 4280 4281 Having CP load it once avoids 4282 loading it at the beginning of 4283 every wavefront. 4284 4285 This is not used for 4286 GFX7-GFX8 since it is the same 4287 value as the second SGPR of 4288 Flat Scratch Init. However, it 4289 may be needed for GFX9-GFX10 which 4290 changes the meaning of the 4291 Flat Scratch Init value. 4292 then Grid Work-Group Count X 1 32-bit count of the number of 4293 (enable_sgpr_grid work-groups in the X dimension 4294 _workgroup_count_X) for the grid being 4295 executed. Computed from the 4296 fields in the kernel dispatch 4297 packet as ((grid_size.x + 4298 workgroup_size.x - 1) / 4299 workgroup_size.x). 4300 then Grid Work-Group Count Y 1 32-bit count of the number of 4301 (enable_sgpr_grid work-groups in the Y dimension 4302 _workgroup_count_Y && for the grid being 4303 less than 16 previous executed. Computed from the 4304 SGPRs) fields in the kernel dispatch 4305 packet as ((grid_size.y + 4306 workgroup_size.y - 1) / 4307 workgroupSize.y). 4308 4309 Only initialized if <16 4310 previous SGPRs initialized. 4311 then Grid Work-Group Count Z 1 32-bit count of the number of 4312 (enable_sgpr_grid work-groups in the Z dimension 4313 _workgroup_count_Z && for the grid being 4314 less than 16 previous executed. Computed from the 4315 SGPRs) fields in the kernel dispatch 4316 packet as ((grid_size.z + 4317 workgroup_size.z - 1) / 4318 workgroupSize.z). 4319 4320 Only initialized if <16 4321 previous SGPRs initialized. 4322 then Work-Group Id X 1 32-bit work-group id in X 4323 (enable_sgpr_workgroup_id dimension of grid for 4324 _X) wavefront. 4325 then Work-Group Id Y 1 32-bit work-group id in Y 4326 (enable_sgpr_workgroup_id dimension of grid for 4327 _Y) wavefront. 4328 then Work-Group Id Z 1 32-bit work-group id in Z 4329 (enable_sgpr_workgroup_id dimension of grid for 4330 _Z) wavefront. 4331 then Work-Group Info 1 {first_wavefront, 14'b0000, 4332 (enable_sgpr_workgroup ordered_append_term[10:0], 4333 _info) threadgroup_size_in_wavefronts[5:0]} 4334 then Scratch Wavefront Offset 1 See 4335 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4336 _segment_wavefront_offset) and 4337 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4338 ========== ========================== ====== ============================== 4339 4340The order of the VGPR registers is defined, but the compiler can specify which 4341ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4342fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4343for enabled registers are dense starting at VGPR0: the first enabled register is 4344VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4345VGPR number. 4346 4347There are different methods used for the VGPR initial state: 4348 4349* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4350 specifies otherwise, a separate VGPR register is used per work-item ID. The 4351 VGPR register initial state for this method is defined in 4352 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4353* If *Target Properties* column of :ref:`amdgpu-processor-table` 4354 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4355 for all work-item IDs. The register layout for this method is defined in 4356 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4357 4358 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4359 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4360 4361 ========== ========================== ====== ============================== 4362 VGPR Order Name Number Description 4363 (kernel descriptor enable of 4364 field) VGPRs 4365 ========== ========================== ====== ============================== 4366 First Work-Item Id X 1 32-bit work-item id in X 4367 (Always initialized) dimension of work-group for 4368 wavefront lane. 4369 then Work-Item Id Y 1 32-bit work-item id in Y 4370 (enable_vgpr_workitem_id dimension of work-group for 4371 > 0) wavefront lane. 4372 then Work-Item Id Z 1 32-bit work-item id in Z 4373 (enable_vgpr_workitem_id dimension of work-group for 4374 > 1) wavefront lane. 4375 ========== ========================== ====== ============================== 4376 4377.. 4378 4379 .. table:: Register Layout for Packed Work-Item ID Method 4380 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4381 4382 ======= ======= ================ ========================================= 4383 Bits Size Field Name Description 4384 ======= ======= ================ ========================================= 4385 0:9 10 bits Work-Item Id X Work-item id in X 4386 dimension of work-group for 4387 wavefront lane. 4388 4389 Always initialized. 4390 4391 10:19 10 bits Work-Item Id Y Work-item id in Y 4392 dimension of work-group for 4393 wavefront lane. 4394 4395 Initialized if enable_vgpr_workitem_id > 4396 0, otherwise set to 0. 4397 20:29 10 bits Work-Item Id Z Work-item id in Z 4398 dimension of work-group for 4399 wavefront lane. 4400 4401 Initialized if enable_vgpr_workitem_id > 4402 1, otherwise set to 0. 4403 30:31 2 bits Reserved, set to 0. 4404 ======= ======= ================ ========================================= 4405 4406The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4407 44081. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4409 registers. 44102. Work-group Id registers X, Y, Z are set by ADC which supports any 4411 combination including none. 44123. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4413 its value cannot be included with the flat scratch init value which is per 4414 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 44154. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4416 or (X, Y, Z). 44175. Flat Scratch register pair initialization is described in 4418 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4419 4420The global segment can be accessed either using buffer instructions (GFX6 which 4421has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4422instructions (GFX9-GFX10). 4423 4424If buffer operations are used, then the compiler can generate a V# with the 4425following properties: 4426 4427* base address of 0 4428* no swizzle 4429* ATC: 1 if IOMMU present (such as APU) 4430* ptr64: 1 4431* MTYPE set to support memory coherence that matches the runtime (such as CC for 4432 APU and NC for dGPU). 4433 4434.. _amdgpu-amdhsa-kernel-prolog: 4435 4436Kernel Prolog 4437~~~~~~~~~~~~~ 4438 4439The compiler performs initialization in the kernel prologue depending on the 4440target and information about things like stack usage in the kernel and called 4441functions. Some of this initialization requires the compiler to request certain 4442User and System SGPRs be present in the 4443:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4444:ref:`amdgpu-amdhsa-kernel-descriptor`. 4445 4446.. _amdgpu-amdhsa-kernel-prolog-cfi: 4447 4448CFI 4449+++ 4450 44511. The CFI return address is undefined. 4452 44532. The CFI CFA is defined using an expression which evaluates to a location 4454 description that comprises one memory location description for the 4455 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4456 4457.. _amdgpu-amdhsa-kernel-prolog-m0: 4458 4459M0 4460++ 4461 4462GFX6-GFX8 4463 The M0 register must be initialized with a value at least the total LDS size 4464 if the kernel may access LDS via DS or flat operations. Total LDS size is 4465 available in dispatch packet. For M0, it is also possible to use maximum 4466 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4467 GFX7-GFX8). 4468GFX9-GFX10 4469 The M0 register is not used for range checking LDS accesses and so does not 4470 need to be initialized in the prolog. 4471 4472.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4473 4474Stack Pointer 4475+++++++++++++ 4476 4477If the kernel has function calls it must set up the ABI stack pointer described 4478in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4479SGPR32 to the unswizzled scratch offset of the address past the last local 4480allocation. 4481 4482.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4483 4484Frame Pointer 4485+++++++++++++ 4486 4487If the kernel needs a frame pointer for the reasons defined in 4488``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4489kernel prolog. If a frame pointer is not required then all uses of the frame 4490pointer are replaced with immediate ``0`` offsets. 4491 4492.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4493 4494Flat Scratch 4495++++++++++++ 4496 4497There are different methods used for initializing flat scratch: 4498 4499* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4500 specifies *Does not support generic address space*: 4501 4502 Flat scratch is not supported and there is no flat scratch register pair. 4503 4504* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4505 specifies *Offset flat scratch*: 4506 4507 If the kernel or any function it calls may use flat operations to access 4508 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4509 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4510 Scratch Wavefront Offset SGPR registers (see 4511 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4512 4513 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4514 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4515 being managed by SPI for the queue executing the kernel dispatch. This is 4516 the same value used in the Scratch Segment Buffer V# base address. 4517 4518 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4519 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4520 4521 The prolog must add the value of Scratch Wavefront Offset to get the 4522 wavefront's byte scratch backing memory offset from 4523 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4524 4525 The Scratch Wavefront Offset must also be used as an offset with Private 4526 segment address when using the Scratch Segment Buffer. 4527 4528 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4529 shifted by 8 before moving into FLAT_SCRATCH_HI. 4530 4531 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4532 SGPRn is the highest numbered SGPR allocated to the wavefront). 4533 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4534 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4535 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4536 aperture. 4537 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4538 work-items scratch memory usage. 4539 4540 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4541 checks that the value in the kernel dispatch packet Private Segment Byte 4542 Size is not larger and requests the runtime to increase the queue's scratch 4543 size if necessary. 4544 4545 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4546 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4547 loading it at the beginning of every wavefront. 4548 4549 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4550 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4551 in flat memory instructions. 4552 4553* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4554 specifies *Absolute flat scratch*: 4555 4556 If the kernel or any function it calls may use flat operations to access 4557 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4558 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4559 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4560 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4561 4562 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4563 memory being managed by SPI for the queue executing the kernel dispatch. 4564 4565 CP obtains this from the runtime. 4566 4567 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4568 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4569 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4570 memory instructions. 4571 4572 The Scratch Wavefront Offset must also be used as an offset with Private 4573 segment address when using the Scratch Segment Buffer (see 4574 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4575 4576.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4577 4578Private Segment Buffer 4579++++++++++++++++++++++ 4580 4581Private Segment Buffer SGPR register is used to initilize 4 SGPRs 4582that are used as a V# to access scratch. CP uses the value provided by the 4583runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4584access the private memory space using a segment address. See 4585:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4586 4587The scratch V# is a four-aligned SGPR and always selected for the kernel as 4588follows: 4589 4590 - If it is known during instruction selection that there is stack usage, 4591 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4592 optimizations are disabled (``-O0``), if stack objects already exist (for 4593 locals, etc.), or if there are any function calls. 4594 4595 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4596 are reserved for the tentative scratch V#. These will be used if it is 4597 determined that spilling is needed. 4598 4599 - If no use is made of the tentative scratch V#, then it is unreserved, 4600 and the register count is determined ignoring it. 4601 - If use is made of the tentative scratch V#, then its register numbers 4602 are shifted to the first four-aligned SGPR index after the highest one 4603 allocated by the register allocator, and all uses are updated. The 4604 register count includes them in the shifted location. 4605 - In either case, if the processor has the SGPR allocation bug, the 4606 tentative allocation is not shifted or unreserved in order to ensure 4607 the register count is higher to workaround the bug. 4608 4609 .. note:: 4610 4611 This approach of using a tentative scratch V# and shifting the register 4612 numbers if used avoids having to perform register allocation a second 4613 time if the tentative V# is eliminated. This is more efficient and 4614 avoids the problem that the second register allocation may perform 4615 spilling which will fail as there is no longer a scratch V#. 4616 4617When the kernel prolog code is being emitted it is known whether the scratch V# 4618described above is actually used. If it is, the prolog code must set it up by 4619copying the Private Segment Buffer to the scratch V# registers and then adding 4620the Private Segment Wavefront Offset to the queue base address in the V#. The 4621result is a V# with a base address pointing to the beginning of the wavefront 4622scratch backing memory. 4623 4624The Private Segment Buffer is always requested, but the Private Segment 4625Wavefront Offset is only requested if it is used (see 4626:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4627 4628.. _amdgpu-amdhsa-memory-model: 4629 4630Memory Model 4631~~~~~~~~~~~~ 4632 4633This section describes the mapping of the LLVM memory model onto AMDGPU machine 4634code (see :ref:`memmodel`). 4635 4636The AMDGPU backend supports the memory synchronization scopes specified in 4637:ref:`amdgpu-memory-scopes`. 4638 4639The code sequences used to implement the memory model specify the order of 4640instructions that a single thread must execute. The ``s_waitcnt`` and cache 4641management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4642to other memory instructions executed by the same thread. This allows them to be 4643moved earlier or later which can allow them to be combined with other instances 4644of the same instruction, or hoisted/sunk out of loops to improve performance. 4645Only the instructions related to the memory model are given; additional 4646``s_waitcnt`` instructions are required to ensure registers are defined before 4647being used. These may be able to be combined with the memory model ``s_waitcnt`` 4648instructions as described above. 4649 4650The AMDGPU backend supports the following memory models: 4651 4652 HSA Memory Model [HSA]_ 4653 The HSA memory model uses a single happens-before relation for all address 4654 spaces (see :ref:`amdgpu-address-spaces`). 4655 OpenCL Memory Model [OpenCL]_ 4656 The OpenCL memory model which has separate happens-before relations for the 4657 global and local address spaces. Only a fence specifying both global and 4658 local address space, and seq_cst instructions join the relationships. Since 4659 the LLVM ``memfence`` instruction does not allow an address space to be 4660 specified the OpenCL fence has to conservatively assume both local and 4661 global address space was specified. However, optimizations can often be 4662 done to eliminate the additional ``s_waitcnt`` instructions when there are 4663 no intervening memory instructions which access the corresponding address 4664 space. The code sequences in the table indicate what can be omitted for the 4665 OpenCL memory. The target triple environment is used to determine if the 4666 source language is OpenCL (see :ref:`amdgpu-opencl`). 4667 4668``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4669operations. 4670 4671``buffer/global/flat_load/store/atomic`` instructions to global memory are 4672termed vector memory operations. 4673 4674Private address space uses ``buffer_load/store`` using the scratch V# 4675(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4676is accessing the memory, atomic memory orderings are not meaningful, and all 4677accesses are treated as non-atomic. 4678 4679Constant address space uses ``buffer/global_load`` instructions (or equivalent 4680scalar memory instructions). Since the constant address space contents do not 4681change during the execution of a kernel dispatch it is not legal to perform 4682stores, and atomic memory orderings are not meaningful, and all accesses are 4683treated as non-atomic. 4684 4685A memory synchronization scope wider than work-group is not meaningful for the 4686group (LDS) address space and is treated as work-group. 4687 4688The memory model does not support the region address space which is treated as 4689non-atomic. 4690 4691Acquire memory ordering is not meaningful on store atomic instructions and is 4692treated as non-atomic. 4693 4694Release memory ordering is not meaningful on load atomic instructions and is 4695treated a non-atomic. 4696 4697Acquire-release memory ordering is not meaningful on load or store atomic 4698instructions and is treated as acquire and release respectively. 4699 4700The memory order also adds the single thread optimization constraints defined in 4701table 4702:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4703 4704 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4705 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4706 4707 ============ ============================================================== 4708 LLVM Memory Optimization Constraints 4709 Ordering 4710 ============ ============================================================== 4711 unordered *none* 4712 monotonic *none* 4713 acquire - If a load atomic/atomicrmw then no following load/load 4714 atomic/store/store atomic/atomicrmw/fence instruction can be 4715 moved before the acquire. 4716 - If a fence then same as load atomic, plus no preceding 4717 associated fence-paired-atomic can be moved after the fence. 4718 release - If a store atomic/atomicrmw then no preceding load/load 4719 atomic/store/store atomic/atomicrmw/fence instruction can be 4720 moved after the release. 4721 - If a fence then same as store atomic, plus no following 4722 associated fence-paired-atomic can be moved before the 4723 fence. 4724 acq_rel Same constraints as both acquire and release. 4725 seq_cst - If a load atomic then same constraints as acquire, plus no 4726 preceding sequentially consistent load atomic/store 4727 atomic/atomicrmw/fence instruction can be moved after the 4728 seq_cst. 4729 - If a store atomic then the same constraints as release, plus 4730 no following sequentially consistent load atomic/store 4731 atomic/atomicrmw/fence instruction can be moved before the 4732 seq_cst. 4733 - If an atomicrmw/fence then same constraints as acq_rel. 4734 ============ ============================================================== 4735 4736The code sequences used to implement the memory model are defined in the 4737following sections: 4738 4739* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4740* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4741* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4742 4743.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4744 4745Memory Model GFX6-GFX9 4746++++++++++++++++++++++ 4747 4748For GFX6-GFX9: 4749 4750* Each agent has multiple shader arrays (SA). 4751* Each SA has multiple compute units (CU). 4752* Each CU has multiple SIMDs that execute wavefronts. 4753* The wavefronts for a single work-group are executed in the same CU but may be 4754 executed by different SIMDs. 4755* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4756 executing on it. 4757* All LDS operations of a CU are performed as wavefront wide operations in a 4758 global order and involve no caching. Completion is reported to a wavefront in 4759 execution order. 4760* The LDS memory has multiple request queues shared by the SIMDs of a 4761 CU. Therefore, the LDS operations performed by different wavefronts of a 4762 work-group can be reordered relative to each other, which can result in 4763 reordering the visibility of vector memory operations with respect to LDS 4764 operations of other wavefronts in the same work-group. A ``s_waitcnt 4765 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4766 vector memory operations between wavefronts of a work-group, but not between 4767 operations performed by the same wavefront. 4768* The vector memory operations are performed as wavefront wide operations and 4769 completion is reported to a wavefront in execution order. The exception is 4770 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4771 vector memory order if they access LDS memory, and out of LDS operation order 4772 if they access global memory. 4773* The vector memory operations access a single vector L1 cache shared by all 4774 SIMDs a CU. Therefore, no special action is required for coherence between the 4775 lanes of a single wavefront, or for coherence between wavefronts in the same 4776 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4777 wavefronts executing in different work-groups as they may be executing on 4778 different CUs. 4779* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4780 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4781 scalar operations are used in a restricted way so do not impact the memory 4782 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4783* The vector and scalar memory operations use an L2 cache shared by all CUs on 4784 the same agent. 4785* The L2 cache has independent channels to service disjoint ranges of virtual 4786 addresses. 4787* Each CU has a separate request queue per channel. Therefore, the vector and 4788 scalar memory operations performed by wavefronts executing in different 4789 work-groups (which may be executing on different CUs) of an agent can be 4790 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4791 ensure synchronization between vector memory operations of different CUs. It 4792 ensures a previous vector memory operation has completed before executing a 4793 subsequent vector memory or LDS operation and so can be used to meet the 4794 requirements of acquire and release. 4795* The L2 cache can be kept coherent with other agents on some targets, or ranges 4796 of virtual addresses can be set up to bypass it to ensure system coherence. 4797 4798Scalar memory operations are only used to access memory that is proven to not 4799change during the execution of the kernel dispatch. This includes constant 4800address space and global address space for program scope ``const`` variables. 4801Therefore, the kernel machine code does not have to maintain the scalar cache to 4802ensure it is coherent with the vector caches. The scalar and vector caches are 4803invalidated between kernel dispatches by CP since constant address space data 4804may change between kernel dispatch executions. See 4805:ref:`amdgpu-amdhsa-memory-spaces`. 4806 4807The one exception is if scalar writes are used to spill SGPR registers. In this 4808case the AMDGPU backend ensures the memory location used to spill is never 4809accessed by vector memory operations at the same time. If scalar writes are used 4810then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4811return since the locations may be used for vector memory instructions by a 4812future wavefront that uses the same scratch area, or a function call that 4813creates a frame at the same address, respectively. There is no need for a 4814``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4815 4816For kernarg backing memory: 4817 4818* CP invalidates the L1 cache at the start of each kernel dispatch. 4819* On dGPU the kernarg backing memory is allocated in host memory accessed as 4820 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4821 causes it to be treated as non-volatile and so is not invalidated by 4822 ``*_vol``. 4823* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4824 and so the L2 cache will be coherent with the CPU and other agents. 4825 4826Scratch backing memory (which is used for the private address space) is accessed 4827with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4828only accessed by a single thread, and is always write-before-read, there is 4829never a need to invalidate these entries from the L1 cache. Hence all cache 4830invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4831 4832The code sequences used to implement the memory model for GFX6-GFX9 are defined 4833in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4834 4835 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4836 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4837 4838 ============ ============ ============== ========== ================================ 4839 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4840 Ordering Sync Scope Address GFX6-GFX9 4841 Space 4842 ============ ============ ============== ========== ================================ 4843 **Non-Atomic** 4844 ------------------------------------------------------------------------------------ 4845 load *none* *none* - global - !volatile & !nontemporal 4846 - generic 4847 - private 1. buffer/global/flat_load 4848 - constant 4849 - !volatile & nontemporal 4850 4851 1. buffer/global/flat_load 4852 glc=1 slc=1 4853 4854 - volatile 4855 4856 1. buffer/global/flat_load 4857 glc=1 4858 2. s_waitcnt vmcnt(0) 4859 4860 - Must happen before 4861 any following volatile 4862 global/generic 4863 load/store. 4864 - Ensures that 4865 volatile 4866 operations to 4867 different 4868 addresses will not 4869 be reordered by 4870 hardware. 4871 4872 load *none* *none* - local 1. ds_load 4873 store *none* *none* - global - !volatile & !nontemporal 4874 - generic 4875 - private 1. buffer/global/flat_store 4876 - constant 4877 - !volatile & nontemporal 4878 4879 1. buffer/global/flat_store 4880 glc=1 slc=1 4881 4882 - volatile 4883 4884 1. buffer/global/flat_store 4885 2. s_waitcnt vmcnt(0) 4886 4887 - Must happen before 4888 any following volatile 4889 global/generic 4890 load/store. 4891 - Ensures that 4892 volatile 4893 operations to 4894 different 4895 addresses will not 4896 be reordered by 4897 hardware. 4898 4899 store *none* *none* - local 1. ds_store 4900 **Unordered Atomic** 4901 ------------------------------------------------------------------------------------ 4902 load atomic unordered *any* *any* *Same as non-atomic*. 4903 store atomic unordered *any* *any* *Same as non-atomic*. 4904 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4905 **Monotonic Atomic** 4906 ------------------------------------------------------------------------------------ 4907 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4908 - wavefront - local 4909 - workgroup - generic 4910 load atomic monotonic - agent - global 1. buffer/global/flat_load 4911 - system - generic glc=1 4912 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4913 - wavefront - generic 4914 - workgroup 4915 - agent 4916 - system 4917 store atomic monotonic - singlethread - local 1. ds_store 4918 - wavefront 4919 - workgroup 4920 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4921 - wavefront - generic 4922 - workgroup 4923 - agent 4924 - system 4925 atomicrmw monotonic - singlethread - local 1. ds_atomic 4926 - wavefront 4927 - workgroup 4928 **Acquire Atomic** 4929 ------------------------------------------------------------------------------------ 4930 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4931 - wavefront - local 4932 - generic 4933 load atomic acquire - workgroup - global 1. buffer/global_load 4934 load atomic acquire - workgroup - local 1. ds/flat_load 4935 - generic 2. s_waitcnt lgkmcnt(0) 4936 4937 - If OpenCL, omit. 4938 - Must happen before 4939 any following 4940 global/generic 4941 load/load 4942 atomic/store/store 4943 atomic/atomicrmw. 4944 - Ensures any 4945 following global 4946 data read is no 4947 older than a local load 4948 atomic value being 4949 acquired. 4950 4951 load atomic acquire - agent - global 1. buffer/global_load 4952 - system glc=1 4953 2. s_waitcnt vmcnt(0) 4954 4955 - Must happen before 4956 following 4957 buffer_wbinvl1_vol. 4958 - Ensures the load 4959 has completed 4960 before invalidating 4961 the cache. 4962 4963 3. buffer_wbinvl1_vol 4964 4965 - Must happen before 4966 any following 4967 global/generic 4968 load/load 4969 atomic/atomicrmw. 4970 - Ensures that 4971 following 4972 loads will not see 4973 stale global data. 4974 4975 load atomic acquire - agent - generic 1. flat_load glc=1 4976 - system 2. s_waitcnt vmcnt(0) & 4977 lgkmcnt(0) 4978 4979 - If OpenCL omit 4980 lgkmcnt(0). 4981 - Must happen before 4982 following 4983 buffer_wbinvl1_vol. 4984 - Ensures the flat_load 4985 has completed 4986 before invalidating 4987 the cache. 4988 4989 3. buffer_wbinvl1_vol 4990 4991 - Must happen before 4992 any following 4993 global/generic 4994 load/load 4995 atomic/atomicrmw. 4996 - Ensures that 4997 following loads 4998 will not see stale 4999 global data. 5000 5001 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5002 - wavefront - local 5003 - generic 5004 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5005 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5006 - generic 2. s_waitcnt lgkmcnt(0) 5007 5008 - If OpenCL, omit. 5009 - Must happen before 5010 any following 5011 global/generic 5012 load/load 5013 atomic/store/store 5014 atomic/atomicrmw. 5015 - Ensures any 5016 following global 5017 data read is no 5018 older than a local 5019 atomicrmw value 5020 being acquired. 5021 5022 atomicrmw acquire - agent - global 1. buffer/global_atomic 5023 - system 2. s_waitcnt vmcnt(0) 5024 5025 - Must happen before 5026 following 5027 buffer_wbinvl1_vol. 5028 - Ensures the 5029 atomicrmw has 5030 completed before 5031 invalidating the 5032 cache. 5033 5034 3. buffer_wbinvl1_vol 5035 5036 - Must happen before 5037 any following 5038 global/generic 5039 load/load 5040 atomic/atomicrmw. 5041 - Ensures that 5042 following loads 5043 will not see stale 5044 global data. 5045 5046 atomicrmw acquire - agent - generic 1. flat_atomic 5047 - system 2. s_waitcnt vmcnt(0) & 5048 lgkmcnt(0) 5049 5050 - If OpenCL, omit 5051 lgkmcnt(0). 5052 - Must happen before 5053 following 5054 buffer_wbinvl1_vol. 5055 - Ensures the 5056 atomicrmw has 5057 completed before 5058 invalidating the 5059 cache. 5060 5061 3. buffer_wbinvl1_vol 5062 5063 - Must happen before 5064 any following 5065 global/generic 5066 load/load 5067 atomic/atomicrmw. 5068 - Ensures that 5069 following loads 5070 will not see stale 5071 global data. 5072 5073 fence acquire - singlethread *none* *none* 5074 - wavefront 5075 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5076 5077 - If OpenCL and 5078 address space is 5079 not generic, omit. 5080 - However, since LLVM 5081 currently has no 5082 address space on 5083 the fence need to 5084 conservatively 5085 always generate. If 5086 fence had an 5087 address space then 5088 set to address 5089 space of OpenCL 5090 fence flag, or to 5091 generic if both 5092 local and global 5093 flags are 5094 specified. 5095 - Must happen after 5096 any preceding 5097 local/generic load 5098 atomic/atomicrmw 5099 with an equal or 5100 wider sync scope 5101 and memory ordering 5102 stronger than 5103 unordered (this is 5104 termed the 5105 fence-paired-atomic). 5106 - Must happen before 5107 any following 5108 global/generic 5109 load/load 5110 atomic/store/store 5111 atomic/atomicrmw. 5112 - Ensures any 5113 following global 5114 data read is no 5115 older than the 5116 value read by the 5117 fence-paired-atomic. 5118 5119 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5120 - system vmcnt(0) 5121 5122 - If OpenCL and 5123 address space is 5124 not generic, omit 5125 lgkmcnt(0). 5126 - However, since LLVM 5127 currently has no 5128 address space on 5129 the fence need to 5130 conservatively 5131 always generate 5132 (see comment for 5133 previous fence). 5134 - Could be split into 5135 separate s_waitcnt 5136 vmcnt(0) and 5137 s_waitcnt 5138 lgkmcnt(0) to allow 5139 them to be 5140 independently moved 5141 according to the 5142 following rules. 5143 - s_waitcnt vmcnt(0) 5144 must happen after 5145 any preceding 5146 global/generic load 5147 atomic/atomicrmw 5148 with an equal or 5149 wider sync scope 5150 and memory ordering 5151 stronger than 5152 unordered (this is 5153 termed the 5154 fence-paired-atomic). 5155 - s_waitcnt lgkmcnt(0) 5156 must happen after 5157 any preceding 5158 local/generic load 5159 atomic/atomicrmw 5160 with an equal or 5161 wider sync scope 5162 and memory ordering 5163 stronger than 5164 unordered (this is 5165 termed the 5166 fence-paired-atomic). 5167 - Must happen before 5168 the following 5169 buffer_wbinvl1_vol. 5170 - Ensures that the 5171 fence-paired atomic 5172 has completed 5173 before invalidating 5174 the 5175 cache. Therefore 5176 any following 5177 locations read must 5178 be no older than 5179 the value read by 5180 the 5181 fence-paired-atomic. 5182 5183 2. buffer_wbinvl1_vol 5184 5185 - Must happen before any 5186 following global/generic 5187 load/load 5188 atomic/store/store 5189 atomic/atomicrmw. 5190 - Ensures that 5191 following loads 5192 will not see stale 5193 global data. 5194 5195 **Release Atomic** 5196 ------------------------------------------------------------------------------------ 5197 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5198 - wavefront - local 5199 - generic 5200 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5201 - generic 5202 - If OpenCL, omit. 5203 - Must happen after 5204 any preceding 5205 local/generic 5206 load/store/load 5207 atomic/store 5208 atomic/atomicrmw. 5209 - Must happen before 5210 the following 5211 store. 5212 - Ensures that all 5213 memory operations 5214 to local have 5215 completed before 5216 performing the 5217 store that is being 5218 released. 5219 5220 2. buffer/global/flat_store 5221 store atomic release - workgroup - local 1. ds_store 5222 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5223 - system - generic vmcnt(0) 5224 5225 - If OpenCL and 5226 address space is 5227 not generic, omit 5228 lgkmcnt(0). 5229 - Could be split into 5230 separate s_waitcnt 5231 vmcnt(0) and 5232 s_waitcnt 5233 lgkmcnt(0) to allow 5234 them to be 5235 independently moved 5236 according to the 5237 following rules. 5238 - s_waitcnt vmcnt(0) 5239 must happen after 5240 any preceding 5241 global/generic 5242 load/store/load 5243 atomic/store 5244 atomic/atomicrmw. 5245 - s_waitcnt lgkmcnt(0) 5246 must happen after 5247 any preceding 5248 local/generic 5249 load/store/load 5250 atomic/store 5251 atomic/atomicrmw. 5252 - Must happen before 5253 the following 5254 store. 5255 - Ensures that all 5256 memory operations 5257 to memory have 5258 completed before 5259 performing the 5260 store that is being 5261 released. 5262 5263 2. buffer/global/flat_store 5264 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5265 - wavefront - local 5266 - generic 5267 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5268 - generic 5269 - If OpenCL, omit. 5270 - Must happen after 5271 any preceding 5272 local/generic 5273 load/store/load 5274 atomic/store 5275 atomic/atomicrmw. 5276 - Must happen before 5277 the following 5278 atomicrmw. 5279 - Ensures that all 5280 memory operations 5281 to local have 5282 completed before 5283 performing the 5284 atomicrmw that is 5285 being released. 5286 5287 2. buffer/global/flat_atomic 5288 atomicrmw release - workgroup - local 1. ds_atomic 5289 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5290 - system - generic vmcnt(0) 5291 5292 - If OpenCL, omit 5293 lgkmcnt(0). 5294 - Could be split into 5295 separate s_waitcnt 5296 vmcnt(0) and 5297 s_waitcnt 5298 lgkmcnt(0) to allow 5299 them to be 5300 independently moved 5301 according to the 5302 following rules. 5303 - s_waitcnt vmcnt(0) 5304 must happen after 5305 any preceding 5306 global/generic 5307 load/store/load 5308 atomic/store 5309 atomic/atomicrmw. 5310 - s_waitcnt lgkmcnt(0) 5311 must happen after 5312 any preceding 5313 local/generic 5314 load/store/load 5315 atomic/store 5316 atomic/atomicrmw. 5317 - Must happen before 5318 the following 5319 atomicrmw. 5320 - Ensures that all 5321 memory operations 5322 to global and local 5323 have completed 5324 before performing 5325 the atomicrmw that 5326 is being released. 5327 5328 2. buffer/global/flat_atomic 5329 fence release - singlethread *none* *none* 5330 - wavefront 5331 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5332 5333 - If OpenCL and 5334 address space is 5335 not generic, omit. 5336 - However, since LLVM 5337 currently has no 5338 address space on 5339 the fence need to 5340 conservatively 5341 always generate. If 5342 fence had an 5343 address space then 5344 set to address 5345 space of OpenCL 5346 fence flag, or to 5347 generic if both 5348 local and global 5349 flags are 5350 specified. 5351 - Must happen after 5352 any preceding 5353 local/generic 5354 load/load 5355 atomic/store/store 5356 atomic/atomicrmw. 5357 - Must happen before 5358 any following store 5359 atomic/atomicrmw 5360 with an equal or 5361 wider sync scope 5362 and memory ordering 5363 stronger than 5364 unordered (this is 5365 termed the 5366 fence-paired-atomic). 5367 - Ensures that all 5368 memory operations 5369 to local have 5370 completed before 5371 performing the 5372 following 5373 fence-paired-atomic. 5374 5375 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5376 - system vmcnt(0) 5377 5378 - If OpenCL and 5379 address space is 5380 not generic, omit 5381 lgkmcnt(0). 5382 - If OpenCL and 5383 address space is 5384 local, omit 5385 vmcnt(0). 5386 - However, since LLVM 5387 currently has no 5388 address space on 5389 the fence need to 5390 conservatively 5391 always generate. If 5392 fence had an 5393 address space then 5394 set to address 5395 space of OpenCL 5396 fence flag, or to 5397 generic if both 5398 local and global 5399 flags are 5400 specified. 5401 - Could be split into 5402 separate s_waitcnt 5403 vmcnt(0) and 5404 s_waitcnt 5405 lgkmcnt(0) to allow 5406 them to be 5407 independently moved 5408 according to the 5409 following rules. 5410 - s_waitcnt vmcnt(0) 5411 must happen after 5412 any preceding 5413 global/generic 5414 load/store/load 5415 atomic/store 5416 atomic/atomicrmw. 5417 - s_waitcnt lgkmcnt(0) 5418 must happen after 5419 any preceding 5420 local/generic 5421 load/store/load 5422 atomic/store 5423 atomic/atomicrmw. 5424 - Must happen before 5425 any following store 5426 atomic/atomicrmw 5427 with an equal or 5428 wider sync scope 5429 and memory ordering 5430 stronger than 5431 unordered (this is 5432 termed the 5433 fence-paired-atomic). 5434 - Ensures that all 5435 memory operations 5436 have 5437 completed before 5438 performing the 5439 following 5440 fence-paired-atomic. 5441 5442 **Acquire-Release Atomic** 5443 ------------------------------------------------------------------------------------ 5444 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5445 - wavefront - local 5446 - generic 5447 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5448 5449 - If OpenCL, omit. 5450 - Must happen after 5451 any preceding 5452 local/generic 5453 load/store/load 5454 atomic/store 5455 atomic/atomicrmw. 5456 - Must happen before 5457 the following 5458 atomicrmw. 5459 - Ensures that all 5460 memory operations 5461 to local have 5462 completed before 5463 performing the 5464 atomicrmw that is 5465 being released. 5466 5467 2. buffer/global_atomic 5468 5469 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5470 2. s_waitcnt lgkmcnt(0) 5471 5472 - If OpenCL, omit. 5473 - Must happen before 5474 any following 5475 global/generic 5476 load/load 5477 atomic/store/store 5478 atomic/atomicrmw. 5479 - Ensures any 5480 following global 5481 data read is no 5482 older than the local load 5483 atomic value being 5484 acquired. 5485 5486 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5487 5488 - If OpenCL, omit. 5489 - Must happen after 5490 any preceding 5491 local/generic 5492 load/store/load 5493 atomic/store 5494 atomic/atomicrmw. 5495 - Must happen before 5496 the following 5497 atomicrmw. 5498 - Ensures that all 5499 memory operations 5500 to local have 5501 completed before 5502 performing the 5503 atomicrmw that is 5504 being released. 5505 5506 2. flat_atomic 5507 3. s_waitcnt lgkmcnt(0) 5508 5509 - If OpenCL, omit. 5510 - Must happen before 5511 any following 5512 global/generic 5513 load/load 5514 atomic/store/store 5515 atomic/atomicrmw. 5516 - Ensures any 5517 following global 5518 data read is no 5519 older than a local load 5520 atomic value being 5521 acquired. 5522 5523 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5524 - system vmcnt(0) 5525 5526 - If OpenCL, omit 5527 lgkmcnt(0). 5528 - Could be split into 5529 separate s_waitcnt 5530 vmcnt(0) and 5531 s_waitcnt 5532 lgkmcnt(0) to allow 5533 them to be 5534 independently moved 5535 according to the 5536 following rules. 5537 - s_waitcnt vmcnt(0) 5538 must happen after 5539 any preceding 5540 global/generic 5541 load/store/load 5542 atomic/store 5543 atomic/atomicrmw. 5544 - s_waitcnt lgkmcnt(0) 5545 must happen after 5546 any preceding 5547 local/generic 5548 load/store/load 5549 atomic/store 5550 atomic/atomicrmw. 5551 - Must happen before 5552 the following 5553 atomicrmw. 5554 - Ensures that all 5555 memory operations 5556 to global have 5557 completed before 5558 performing the 5559 atomicrmw that is 5560 being released. 5561 5562 2. buffer/global_atomic 5563 3. s_waitcnt vmcnt(0) 5564 5565 - Must happen before 5566 following 5567 buffer_wbinvl1_vol. 5568 - Ensures the 5569 atomicrmw has 5570 completed before 5571 invalidating the 5572 cache. 5573 5574 4. buffer_wbinvl1_vol 5575 5576 - Must happen before 5577 any following 5578 global/generic 5579 load/load 5580 atomic/atomicrmw. 5581 - Ensures that 5582 following loads 5583 will not see stale 5584 global data. 5585 5586 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5587 - system vmcnt(0) 5588 5589 - If OpenCL, omit 5590 lgkmcnt(0). 5591 - Could be split into 5592 separate s_waitcnt 5593 vmcnt(0) and 5594 s_waitcnt 5595 lgkmcnt(0) to allow 5596 them to be 5597 independently moved 5598 according to the 5599 following rules. 5600 - s_waitcnt vmcnt(0) 5601 must happen after 5602 any preceding 5603 global/generic 5604 load/store/load 5605 atomic/store 5606 atomic/atomicrmw. 5607 - s_waitcnt lgkmcnt(0) 5608 must happen after 5609 any preceding 5610 local/generic 5611 load/store/load 5612 atomic/store 5613 atomic/atomicrmw. 5614 - Must happen before 5615 the following 5616 atomicrmw. 5617 - Ensures that all 5618 memory operations 5619 to global have 5620 completed before 5621 performing the 5622 atomicrmw that is 5623 being released. 5624 5625 2. flat_atomic 5626 3. s_waitcnt vmcnt(0) & 5627 lgkmcnt(0) 5628 5629 - If OpenCL, omit 5630 lgkmcnt(0). 5631 - Must happen before 5632 following 5633 buffer_wbinvl1_vol. 5634 - Ensures the 5635 atomicrmw has 5636 completed before 5637 invalidating the 5638 cache. 5639 5640 4. buffer_wbinvl1_vol 5641 5642 - Must happen before 5643 any following 5644 global/generic 5645 load/load 5646 atomic/atomicrmw. 5647 - Ensures that 5648 following loads 5649 will not see stale 5650 global data. 5651 5652 fence acq_rel - singlethread *none* *none* 5653 - wavefront 5654 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5655 5656 - If OpenCL and 5657 address space is 5658 not generic, omit. 5659 - However, 5660 since LLVM 5661 currently has no 5662 address space on 5663 the fence need to 5664 conservatively 5665 always generate 5666 (see comment for 5667 previous fence). 5668 - Must happen after 5669 any preceding 5670 local/generic 5671 load/load 5672 atomic/store/store 5673 atomic/atomicrmw. 5674 - Must happen before 5675 any following 5676 global/generic 5677 load/load 5678 atomic/store/store 5679 atomic/atomicrmw. 5680 - Ensures that all 5681 memory operations 5682 to local have 5683 completed before 5684 performing any 5685 following global 5686 memory operations. 5687 - Ensures that the 5688 preceding 5689 local/generic load 5690 atomic/atomicrmw 5691 with an equal or 5692 wider sync scope 5693 and memory ordering 5694 stronger than 5695 unordered (this is 5696 termed the 5697 acquire-fence-paired-atomic) 5698 has completed 5699 before following 5700 global memory 5701 operations. This 5702 satisfies the 5703 requirements of 5704 acquire. 5705 - Ensures that all 5706 previous memory 5707 operations have 5708 completed before a 5709 following 5710 local/generic store 5711 atomic/atomicrmw 5712 with an equal or 5713 wider sync scope 5714 and memory ordering 5715 stronger than 5716 unordered (this is 5717 termed the 5718 release-fence-paired-atomic). 5719 This satisfies the 5720 requirements of 5721 release. 5722 5723 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5724 - system vmcnt(0) 5725 5726 - If OpenCL and 5727 address space is 5728 not generic, omit 5729 lgkmcnt(0). 5730 - However, since LLVM 5731 currently has no 5732 address space on 5733 the fence need to 5734 conservatively 5735 always generate 5736 (see comment for 5737 previous fence). 5738 - Could be split into 5739 separate s_waitcnt 5740 vmcnt(0) and 5741 s_waitcnt 5742 lgkmcnt(0) to allow 5743 them to be 5744 independently moved 5745 according to the 5746 following rules. 5747 - s_waitcnt vmcnt(0) 5748 must happen after 5749 any preceding 5750 global/generic 5751 load/store/load 5752 atomic/store 5753 atomic/atomicrmw. 5754 - s_waitcnt lgkmcnt(0) 5755 must happen after 5756 any preceding 5757 local/generic 5758 load/store/load 5759 atomic/store 5760 atomic/atomicrmw. 5761 - Must happen before 5762 the following 5763 buffer_wbinvl1_vol. 5764 - Ensures that the 5765 preceding 5766 global/local/generic 5767 load 5768 atomic/atomicrmw 5769 with an equal or 5770 wider sync scope 5771 and memory ordering 5772 stronger than 5773 unordered (this is 5774 termed the 5775 acquire-fence-paired-atomic) 5776 has completed 5777 before invalidating 5778 the cache. This 5779 satisfies the 5780 requirements of 5781 acquire. 5782 - Ensures that all 5783 previous memory 5784 operations have 5785 completed before a 5786 following 5787 global/local/generic 5788 store 5789 atomic/atomicrmw 5790 with an equal or 5791 wider sync scope 5792 and memory ordering 5793 stronger than 5794 unordered (this is 5795 termed the 5796 release-fence-paired-atomic). 5797 This satisfies the 5798 requirements of 5799 release. 5800 5801 2. buffer_wbinvl1_vol 5802 5803 - Must happen before 5804 any following 5805 global/generic 5806 load/load 5807 atomic/store/store 5808 atomic/atomicrmw. 5809 - Ensures that 5810 following loads 5811 will not see stale 5812 global data. This 5813 satisfies the 5814 requirements of 5815 acquire. 5816 5817 **Sequential Consistent Atomic** 5818 ------------------------------------------------------------------------------------ 5819 load atomic seq_cst - singlethread - global *Same as corresponding 5820 - wavefront - local load atomic acquire, 5821 - generic except must generated 5822 all instructions even 5823 for OpenCL.* 5824 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5825 - generic 5826 5827 - Must 5828 happen after 5829 preceding 5830 local/generic load 5831 atomic/store 5832 atomic/atomicrmw 5833 with memory 5834 ordering of seq_cst 5835 and with equal or 5836 wider sync scope. 5837 (Note that seq_cst 5838 fences have their 5839 own s_waitcnt 5840 lgkmcnt(0) and so do 5841 not need to be 5842 considered.) 5843 - Ensures any 5844 preceding 5845 sequential 5846 consistent local 5847 memory instructions 5848 have completed 5849 before executing 5850 this sequentially 5851 consistent 5852 instruction. This 5853 prevents reordering 5854 a seq_cst store 5855 followed by a 5856 seq_cst load. (Note 5857 that seq_cst is 5858 stronger than 5859 acquire/release as 5860 the reordering of 5861 load acquire 5862 followed by a store 5863 release is 5864 prevented by the 5865 s_waitcnt of 5866 the release, but 5867 there is nothing 5868 preventing a store 5869 release followed by 5870 load acquire from 5871 completing out of 5872 order. The s_waitcnt 5873 could be placed after 5874 seq_store or before 5875 the seq_load. We 5876 choose the load to 5877 make the s_waitcnt be 5878 as late as possible 5879 so that the store 5880 may have already 5881 completed.) 5882 5883 2. *Following 5884 instructions same as 5885 corresponding load 5886 atomic acquire, 5887 except must generated 5888 all instructions even 5889 for OpenCL.* 5890 load atomic seq_cst - workgroup - local *Same as corresponding 5891 load atomic acquire, 5892 except must generated 5893 all instructions even 5894 for OpenCL.* 5895 5896 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5897 - system - generic vmcnt(0) 5898 5899 - Could be split into 5900 separate s_waitcnt 5901 vmcnt(0) 5902 and s_waitcnt 5903 lgkmcnt(0) to allow 5904 them to be 5905 independently moved 5906 according to the 5907 following rules. 5908 - s_waitcnt lgkmcnt(0) 5909 must happen after 5910 preceding 5911 global/generic load 5912 atomic/store 5913 atomic/atomicrmw 5914 with memory 5915 ordering of seq_cst 5916 and with equal or 5917 wider sync scope. 5918 (Note that seq_cst 5919 fences have their 5920 own s_waitcnt 5921 lgkmcnt(0) and so do 5922 not need to be 5923 considered.) 5924 - s_waitcnt vmcnt(0) 5925 must happen after 5926 preceding 5927 global/generic load 5928 atomic/store 5929 atomic/atomicrmw 5930 with memory 5931 ordering of seq_cst 5932 and with equal or 5933 wider sync scope. 5934 (Note that seq_cst 5935 fences have their 5936 own s_waitcnt 5937 vmcnt(0) and so do 5938 not need to be 5939 considered.) 5940 - Ensures any 5941 preceding 5942 sequential 5943 consistent global 5944 memory instructions 5945 have completed 5946 before executing 5947 this sequentially 5948 consistent 5949 instruction. This 5950 prevents reordering 5951 a seq_cst store 5952 followed by a 5953 seq_cst load. (Note 5954 that seq_cst is 5955 stronger than 5956 acquire/release as 5957 the reordering of 5958 load acquire 5959 followed by a store 5960 release is 5961 prevented by the 5962 s_waitcnt of 5963 the release, but 5964 there is nothing 5965 preventing a store 5966 release followed by 5967 load acquire from 5968 completing out of 5969 order. The s_waitcnt 5970 could be placed after 5971 seq_store or before 5972 the seq_load. We 5973 choose the load to 5974 make the s_waitcnt be 5975 as late as possible 5976 so that the store 5977 may have already 5978 completed.) 5979 5980 2. *Following 5981 instructions same as 5982 corresponding load 5983 atomic acquire, 5984 except must generated 5985 all instructions even 5986 for OpenCL.* 5987 store atomic seq_cst - singlethread - global *Same as corresponding 5988 - wavefront - local store atomic release, 5989 - workgroup - generic except must generated 5990 - agent all instructions even 5991 - system for OpenCL.* 5992 atomicrmw seq_cst - singlethread - global *Same as corresponding 5993 - wavefront - local atomicrmw acq_rel, 5994 - workgroup - generic except must generated 5995 - agent all instructions even 5996 - system for OpenCL.* 5997 fence seq_cst - singlethread *none* *Same as corresponding 5998 - wavefront fence acq_rel, 5999 - workgroup except must generated 6000 - agent all instructions even 6001 - system for OpenCL.* 6002 ============ ============ ============== ========== ================================ 6003 6004.. _amdgpu-amdhsa-memory-model-gfx90a: 6005 6006Memory Model GFX90A 6007+++++++++++++++++++ 6008 6009For GFX90A: 6010 6011* Each agent has multiple shader arrays (SA). 6012* Each SA has multiple compute units (CU). 6013* Each CU has multiple SIMDs that execute wavefronts. 6014* The wavefronts for a single work-group are executed in the same CU but may be 6015 executed by different SIMDs. The exception is when in tgsplit execution mode 6016 when the wavefronts may be executed by different SIMDs in different CUs. 6017* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6018 executing on it. The exception is when in tgsplit execution mode when no LDS 6019 is allocated as wavefronts of the same work-group can be in different CUs. 6020* All LDS operations of a CU are performed as wavefront wide operations in a 6021 global order and involve no caching. Completion is reported to a wavefront in 6022 execution order. 6023* The LDS memory has multiple request queues shared by the SIMDs of a 6024 CU. Therefore, the LDS operations performed by different wavefronts of a 6025 work-group can be reordered relative to each other, which can result in 6026 reordering the visibility of vector memory operations with respect to LDS 6027 operations of other wavefronts in the same work-group. A ``s_waitcnt 6028 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6029 vector memory operations between wavefronts of a work-group, but not between 6030 operations performed by the same wavefront. 6031* The vector memory operations are performed as wavefront wide operations and 6032 completion is reported to a wavefront in execution order. The exception is 6033 that ``flat_load/store/atomic`` instructions can report out of vector memory 6034 order if they access LDS memory, and out of LDS operation order if they access 6035 global memory. 6036* The vector memory operations access a single vector L1 cache shared by all 6037 SIMDs a CU. Therefore: 6038 6039 * No special action is required for coherence between the lanes of a single 6040 wavefront. 6041 6042 * No special action is required for coherence between wavefronts in the same 6043 work-group since they exeute on the same CU. The exception is when in 6044 tgsplit execution mode as wavefronts of the same work-group can be in 6045 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6046 the following item. 6047 6048 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6049 executing in different work-groups as they may be executing on different 6050 CUs. 6051 6052* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6053 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6054 scalar operations are used in a restricted way so do not impact the memory 6055 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6056* The vector and scalar memory operations use an L2 cache shared by all CUs on 6057 the same agent. 6058 6059 * The L2 cache has independent channels to service disjoint ranges of virtual 6060 addresses. 6061 * Each CU has a separate request queue per channel. Therefore, the vector and 6062 scalar memory operations performed by wavefronts executing in different 6063 work-groups (which may be executing on different CUs), or the same 6064 work-group if executing in tgsplit mode, of an agent can be reordered 6065 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6066 synchronization between vector memory operations of different CUs. It 6067 ensures a previous vector memory operation has completed before executing a 6068 subsequent vector memory or LDS operation and so can be used to meet the 6069 requirements of acquire and release. 6070 * The L2 cache of one agent can be kept coherent with other agents by using 6071 the MTYPE RW (read-write) for memory local to the L2, and MTYPE NC 6072 (non-coherent) with the PTE C-bit set for memory not local to the L2. 6073 6074 * Any local memory cache lines will be automatically invalidated by writes 6075 from CUs associated with other L2 caches, or writes from the CPU, due to 6076 the cache probe caused by the PTE C-bit. 6077 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6078 Subsequent access from the GPU will automatically invalidate or writeback 6079 the CPU cache due to the L2 probe filter. 6080 * Since all work-groups on the same agent share the same L2, no L2 6081 invalidation or writeback is required for coherence. 6082 * To ensure coherence of local memory writes of work-groups in different 6083 agents a ``buffer_wbl2`` is required. It will writeback dirty L2 cache 6084 lines. 6085 * To ensure coherence of local memory reads of work-groups in different 6086 agents a ``buffer_invl2`` is required. It will invalidate non-local L2 6087 cache lines. 6088 6089 * PCIe access from the GPU to the CPU memory can be kept coherent by using the 6090 MTYPE UC (uncached) which bypasses the L2. 6091 6092Scalar memory operations are only used to access memory that is proven to not 6093change during the execution of the kernel dispatch. This includes constant 6094address space and global address space for program scope ``const`` variables. 6095Therefore, the kernel machine code does not have to maintain the scalar cache to 6096ensure it is coherent with the vector caches. The scalar and vector caches are 6097invalidated between kernel dispatches by CP since constant address space data 6098may change between kernel dispatch executions. See 6099:ref:`amdgpu-amdhsa-memory-spaces`. 6100 6101The one exception is if scalar writes are used to spill SGPR registers. In this 6102case the AMDGPU backend ensures the memory location used to spill is never 6103accessed by vector memory operations at the same time. If scalar writes are used 6104then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6105return since the locations may be used for vector memory instructions by a 6106future wavefront that uses the same scratch area, or a function call that 6107creates a frame at the same address, respectively. There is no need for a 6108``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6109 6110For kernarg backing memory: 6111 6112* CP invalidates the L1 cache at the start of each kernel dispatch. 6113* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6114 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6115 cache. This also causes it to be treated as non-volatile and so is not 6116 invalidated by ``*_vol``. 6117* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6118 so the L2 cache will be coherent with the CPU and other agents. 6119 6120Scratch backing memory (which is used for the private address space) is accessed 6121with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6122only accessed by a single thread, and is always write-before-read, there is 6123never a need to invalidate these entries from the L1 cache. Hence all cache 6124invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6125 6126The code sequences used to implement the memory model for GFX90A are defined 6127in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6128 6129 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6130 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6131 6132 ============ ============ ============== ========== ================================ 6133 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6134 Ordering Sync Scope Address GFX90A 6135 Space 6136 ============ ============ ============== ========== ================================ 6137 **Non-Atomic** 6138 ------------------------------------------------------------------------------------ 6139 load *none* *none* - global - !volatile & !nontemporal 6140 - generic 6141 - private 1. buffer/global/flat_load 6142 - constant 6143 - !volatile & nontemporal 6144 6145 1. buffer/global/flat_load 6146 glc=1 slc=1 6147 6148 - volatile 6149 6150 1. buffer/global/flat_load 6151 glc=1 scc=1 6152 2. s_waitcnt vmcnt(0) 6153 6154 - Must happen before 6155 any following volatile 6156 global/generic 6157 load/store. 6158 - Ensures that 6159 volatile 6160 operations to 6161 different 6162 addresses will not 6163 be reordered by 6164 hardware. 6165 6166 load *none* *none* - local 1. ds_load 6167 store *none* *none* - global - !volatile & !nontemporal 6168 - generic 6169 - private 1. buffer/global/flat_store 6170 - constant 6171 - !volatile & nontemporal 6172 6173 1. buffer/global/flat_store 6174 glc=1 slc=1 6175 6176 - volatile 6177 6178 1. buffer/global/flat_store 6179 scc=1 6180 2. s_waitcnt vmcnt(0) 6181 6182 - Must happen before 6183 any following volatile 6184 global/generic 6185 load/store. 6186 - Ensures that 6187 volatile 6188 operations to 6189 different 6190 addresses will not 6191 be reordered by 6192 hardware. 6193 6194 store *none* *none* - local 1. ds_store 6195 **Unordered Atomic** 6196 ------------------------------------------------------------------------------------ 6197 load atomic unordered *any* *any* *Same as non-atomic*. 6198 store atomic unordered *any* *any* *Same as non-atomic*. 6199 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6200 **Monotonic Atomic** 6201 ------------------------------------------------------------------------------------ 6202 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6203 - wavefront - generic 6204 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6205 - generic glc=1 6206 6207 - If not TgSplit execution 6208 mode, omit glc=1. 6209 6210 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6211 - wavefront local address space cannot 6212 - workgroup be used.* 6213 6214 1. ds_load 6215 load atomic monotonic - agent - global 1. buffer/global/flat_load 6216 - generic glc=1 6217 load atomic monotonic - system - global 1. buffer/global/flat_load 6218 - generic glc=1 scc=1 6219 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6220 - wavefront - generic 6221 - workgroup 6222 - agent 6223 store atomic monotonic - system - global 1. buffer/global/flat_store 6224 - generic scc=1 6225 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6226 - wavefront local address space cannot 6227 - workgroup be used.* 6228 6229 1. ds_store 6230 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6231 - wavefront - generic 6232 - workgroup 6233 - agent 6234 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6235 - generic scc=1 6236 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6237 - wavefront local address space cannot 6238 - workgroup be used.* 6239 6240 1. ds_atomic 6241 **Acquire Atomic** 6242 ------------------------------------------------------------------------------------ 6243 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6244 - wavefront - local 6245 - generic 6246 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6247 6248 - If not TgSplit execution 6249 mode, omit glc=1. 6250 6251 2. s_waitcnt vmcnt(0) 6252 6253 - If not TgSplit execution 6254 mode, omit. 6255 - Must happen before the 6256 following buffer_wbinvl1_vol. 6257 6258 3. buffer_wbinvl1_vol 6259 6260 - If not TgSplit execution 6261 mode, omit. 6262 - Must happen before 6263 any following 6264 global/generic 6265 load/load 6266 atomic/store/store 6267 atomic/atomicrmw. 6268 - Ensures that 6269 following 6270 loads will not see 6271 stale data. 6272 6273 load atomic acquire - workgroup - local *If TgSplit execution mode, 6274 local address space cannot 6275 be used.* 6276 6277 1. ds_load 6278 2. s_waitcnt lgkmcnt(0) 6279 6280 - If OpenCL, omit. 6281 - Must happen before 6282 any following 6283 global/generic 6284 load/load 6285 atomic/store/store 6286 atomic/atomicrmw. 6287 - Ensures any 6288 following global 6289 data read is no 6290 older than the local load 6291 atomic value being 6292 acquired. 6293 6294 load atomic acquire - workgroup - generic 1. flat_load glc=1 6295 6296 - If not TgSplit execution 6297 mode, omit glc=1. 6298 6299 2. s_waitcnt lgkm/vmcnt(0) 6300 6301 - Use lgkmcnt(0) if not 6302 TgSplit execution mode 6303 and vmcnt(0) if TgSplit 6304 execution mode. 6305 - If OpenCL, omit lgkmcnt(0). 6306 - Must happen before 6307 the following 6308 buffer_wbinvl1_vol and any 6309 following global/generic 6310 load/load 6311 atomic/store/store 6312 atomic/atomicrmw. 6313 - Ensures any 6314 following global 6315 data read is no 6316 older than a local load 6317 atomic value being 6318 acquired. 6319 6320 3. buffer_wbinvl1_vol 6321 6322 - If not TgSplit execution 6323 mode, omit. 6324 - Ensures that 6325 following 6326 loads will not see 6327 stale data. 6328 6329 load atomic acquire - agent - global 1. buffer/global_load 6330 glc=1 6331 2. s_waitcnt vmcnt(0) 6332 6333 - Must happen before 6334 following 6335 buffer_wbinvl1_vol. 6336 - Ensures the load 6337 has completed 6338 before invalidating 6339 the cache. 6340 6341 3. buffer_wbinvl1_vol 6342 6343 - Must happen before 6344 any following 6345 global/generic 6346 load/load 6347 atomic/atomicrmw. 6348 - Ensures that 6349 following 6350 loads will not see 6351 stale global data. 6352 6353 load atomic acquire - system - global 1. buffer/global/flat_load 6354 glc=1 scc=1 6355 2. s_waitcnt vmcnt(0) 6356 6357 - Must happen before 6358 following buffer_invl2 and 6359 buffer_wbinvl1_vol. 6360 - Ensures the load 6361 has completed 6362 before invalidating 6363 the cache. 6364 6365 3. buffer_invl2; 6366 buffer_wbinvl1_vol 6367 6368 - Must happen before 6369 any following 6370 global/generic 6371 load/load 6372 atomic/atomicrmw. 6373 - Ensures that 6374 following 6375 loads will not see 6376 stale MTYPE NC global data. 6377 MTYPE RW and CC memory will 6378 never be stale due to the 6379 memory probes. 6380 6381 load atomic acquire - agent - generic 1. flat_load glc=1 6382 2. s_waitcnt vmcnt(0) & 6383 lgkmcnt(0) 6384 6385 - If TgSplit execution mode, 6386 omit lgkmcnt(0). 6387 - If OpenCL omit 6388 lgkmcnt(0). 6389 - Must happen before 6390 following 6391 buffer_wbinvl1_vol. 6392 - Ensures the flat_load 6393 has completed 6394 before invalidating 6395 the cache. 6396 6397 3. buffer_wbinvl1_vol 6398 6399 - Must happen before 6400 any following 6401 global/generic 6402 load/load 6403 atomic/atomicrmw. 6404 - Ensures that 6405 following loads 6406 will not see stale 6407 global data. 6408 6409 load atomic acquire - system - generic 1. flat_load glc=1 scc=1 6410 2. s_waitcnt vmcnt(0) & 6411 lgkmcnt(0) 6412 6413 - If TgSplit execution mode, 6414 omit lgkmcnt(0). 6415 - If OpenCL omit 6416 lgkmcnt(0). 6417 - Must happen before 6418 following 6419 buffer_invl2 and 6420 buffer_wbinvl1_vol. 6421 - Ensures the flat_load 6422 has completed 6423 before invalidating 6424 the caches. 6425 6426 3. buffer_invl2; 6427 buffer_wbinvl1_vol 6428 6429 - Must happen before 6430 any following 6431 global/generic 6432 load/load 6433 atomic/atomicrmw. 6434 - Ensures that 6435 following 6436 loads will not see 6437 stale MTYPE NC global data. 6438 MTYPE RW and CC memory will 6439 never be stale due to the 6440 memory probes. 6441 6442 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6443 - wavefront - generic 6444 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6445 - wavefront local address space cannot 6446 be used.* 6447 6448 1. ds_atomic 6449 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6450 2. s_waitcnt vmcnt(0) 6451 6452 - If not TgSplit execution 6453 mode, omit. 6454 - Must happen before the 6455 following buffer_wbinvl1_vol. 6456 - Ensures the atomicrmw 6457 has completed 6458 before invalidating 6459 the cache. 6460 6461 3. buffer_wbinvl1_vol 6462 6463 - If not TgSplit execution 6464 mode, omit. 6465 - Must happen before 6466 any following 6467 global/generic 6468 load/load 6469 atomic/atomicrmw. 6470 - Ensures that 6471 following loads 6472 will not see stale 6473 global data. 6474 6475 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6476 local address space cannot 6477 be used.* 6478 6479 1. ds_atomic 6480 2. s_waitcnt lgkmcnt(0) 6481 6482 - If OpenCL, omit. 6483 - Must happen before 6484 any following 6485 global/generic 6486 load/load 6487 atomic/store/store 6488 atomic/atomicrmw. 6489 - Ensures any 6490 following global 6491 data read is no 6492 older than the local 6493 atomicrmw value 6494 being acquired. 6495 6496 atomicrmw acquire - workgroup - generic 1. flat_atomic 6497 2. s_waitcnt lgkm/vmcnt(0) 6498 6499 - Use lgkmcnt(0) if not 6500 TgSplit execution mode 6501 and vmcnt(0) if TgSplit 6502 execution mode. 6503 - If OpenCL, omit lgkmcnt(0). 6504 - Must happen before 6505 the following 6506 buffer_wbinvl1_vol and 6507 any following 6508 global/generic 6509 load/load 6510 atomic/store/store 6511 atomic/atomicrmw. 6512 - Ensures any 6513 following global 6514 data read is no 6515 older than a local 6516 atomicrmw value 6517 being acquired. 6518 6519 3. buffer_wbinvl1_vol 6520 6521 - If not TgSplit execution 6522 mode, omit. 6523 - Ensures that 6524 following 6525 loads will not see 6526 stale data. 6527 6528 atomicrmw acquire - agent - global 1. buffer/global_atomic 6529 2. s_waitcnt vmcnt(0) 6530 6531 - Must happen before 6532 following 6533 buffer_wbinvl1_vol. 6534 - Ensures the 6535 atomicrmw has 6536 completed before 6537 invalidating the 6538 cache. 6539 6540 3. buffer_wbinvl1_vol 6541 6542 - Must happen before 6543 any following 6544 global/generic 6545 load/load 6546 atomic/atomicrmw. 6547 - Ensures that 6548 following loads 6549 will not see stale 6550 global data. 6551 6552 atomicrmw acquire - system - global 1. buffer/global_atomic 6553 scc=1 6554 2. s_waitcnt vmcnt(0) 6555 6556 - Must happen before 6557 following buffer_invl2 and 6558 buffer_wbinvl1_vol. 6559 - Ensures the 6560 atomicrmw has 6561 completed before 6562 invalidating the 6563 caches. 6564 6565 3. buffer_invl2; 6566 buffer_wbinvl1_vol 6567 6568 - Must happen before 6569 any following 6570 global/generic 6571 load/load 6572 atomic/atomicrmw. 6573 - Ensures that 6574 following 6575 loads will not see 6576 stale MTYPE NC global data. 6577 MTYPE RW and CC memory will 6578 never be stale due to the 6579 memory probes. 6580 6581 atomicrmw acquire - agent - generic 1. flat_atomic 6582 2. s_waitcnt vmcnt(0) & 6583 lgkmcnt(0) 6584 6585 - If TgSplit execution mode, 6586 omit lgkmcnt(0). 6587 - If OpenCL, omit 6588 lgkmcnt(0). 6589 - Must happen before 6590 following 6591 buffer_wbinvl1_vol. 6592 - Ensures the 6593 atomicrmw has 6594 completed before 6595 invalidating the 6596 cache. 6597 6598 3. buffer_wbinvl1_vol 6599 6600 - Must happen before 6601 any following 6602 global/generic 6603 load/load 6604 atomic/atomicrmw. 6605 - Ensures that 6606 following loads 6607 will not see stale 6608 global data. 6609 6610 atomicrmw acquire - system - generic 1. flat_atomic scc=1 6611 2. s_waitcnt vmcnt(0) & 6612 lgkmcnt(0) 6613 6614 - If TgSplit execution mode, 6615 omit lgkmcnt(0). 6616 - If OpenCL, omit 6617 lgkmcnt(0). 6618 - Must happen before 6619 following 6620 buffer_invl2 and 6621 buffer_wbinvl1_vol. 6622 - Ensures the 6623 atomicrmw has 6624 completed before 6625 invalidating the 6626 caches. 6627 6628 3. buffer_invl2; 6629 buffer_wbinvl1_vol 6630 6631 - Must happen before 6632 any following 6633 global/generic 6634 load/load 6635 atomic/atomicrmw. 6636 - Ensures that 6637 following 6638 loads will not see 6639 stale MTYPE NC global data. 6640 MTYPE RW and CC memory will 6641 never be stale due to the 6642 memory probes. 6643 6644 fence acquire - singlethread *none* *none* 6645 - wavefront 6646 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6647 6648 - Use lgkmcnt(0) if not 6649 TgSplit execution mode 6650 and vmcnt(0) if TgSplit 6651 execution mode. 6652 - If OpenCL and 6653 address space is 6654 not generic, omit 6655 lgkmcnt(0). 6656 - If OpenCL and 6657 address space is 6658 local, omit 6659 vmcnt(0). 6660 - However, since LLVM 6661 currently has no 6662 address space on 6663 the fence need to 6664 conservatively 6665 always generate. If 6666 fence had an 6667 address space then 6668 set to address 6669 space of OpenCL 6670 fence flag, or to 6671 generic if both 6672 local and global 6673 flags are 6674 specified. 6675 - s_waitcnt vmcnt(0) 6676 must happen after 6677 any preceding 6678 global/generic load 6679 atomic/ 6680 atomicrmw 6681 with an equal or 6682 wider sync scope 6683 and memory ordering 6684 stronger than 6685 unordered (this is 6686 termed the 6687 fence-paired-atomic). 6688 - s_waitcnt lgkmcnt(0) 6689 must happen after 6690 any preceding 6691 local/generic load 6692 atomic/atomicrmw 6693 with an equal or 6694 wider sync scope 6695 and memory ordering 6696 stronger than 6697 unordered (this is 6698 termed the 6699 fence-paired-atomic). 6700 - Must happen before 6701 the following 6702 buffer_wbinvl1_vol and 6703 any following 6704 global/generic 6705 load/load 6706 atomic/store/store 6707 atomic/atomicrmw. 6708 - Ensures any 6709 following global 6710 data read is no 6711 older than the 6712 value read by the 6713 fence-paired-atomic. 6714 6715 3. buffer_wbinvl1_vol 6716 6717 - If not TgSplit execution 6718 mode, omit. 6719 - Ensures that 6720 following 6721 loads will not see 6722 stale data. 6723 6724 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6725 vmcnt(0) 6726 6727 - If TgSplit execution mode, 6728 omit lgkmcnt(0). 6729 - If OpenCL and 6730 address space is 6731 not generic, omit 6732 lgkmcnt(0). 6733 - However, since LLVM 6734 currently has no 6735 address space on 6736 the fence need to 6737 conservatively 6738 always generate 6739 (see comment for 6740 previous fence). 6741 - Could be split into 6742 separate s_waitcnt 6743 vmcnt(0) and 6744 s_waitcnt 6745 lgkmcnt(0) to allow 6746 them to be 6747 independently moved 6748 according to the 6749 following rules. 6750 - s_waitcnt vmcnt(0) 6751 must happen after 6752 any preceding 6753 global/generic load 6754 atomic/atomicrmw 6755 with an equal or 6756 wider sync scope 6757 and memory ordering 6758 stronger than 6759 unordered (this is 6760 termed the 6761 fence-paired-atomic). 6762 - s_waitcnt lgkmcnt(0) 6763 must happen after 6764 any preceding 6765 local/generic load 6766 atomic/atomicrmw 6767 with an equal or 6768 wider sync scope 6769 and memory ordering 6770 stronger than 6771 unordered (this is 6772 termed the 6773 fence-paired-atomic). 6774 - Must happen before 6775 the following 6776 buffer_wbinvl1_vol. 6777 - Ensures that the 6778 fence-paired atomic 6779 has completed 6780 before invalidating 6781 the 6782 cache. Therefore 6783 any following 6784 locations read must 6785 be no older than 6786 the value read by 6787 the 6788 fence-paired-atomic. 6789 6790 2. buffer_wbinvl1_vol 6791 6792 - Must happen before any 6793 following global/generic 6794 load/load 6795 atomic/store/store 6796 atomic/atomicrmw. 6797 - Ensures that 6798 following loads 6799 will not see stale 6800 global data. 6801 6802 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6803 vmcnt(0) 6804 6805 - If TgSplit execution mode, 6806 omit lgkmcnt(0). 6807 - If OpenCL and 6808 address space is 6809 not generic, omit 6810 lgkmcnt(0). 6811 - However, since LLVM 6812 currently has no 6813 address space on 6814 the fence need to 6815 conservatively 6816 always generate 6817 (see comment for 6818 previous fence). 6819 - Could be split into 6820 separate s_waitcnt 6821 vmcnt(0) and 6822 s_waitcnt 6823 lgkmcnt(0) to allow 6824 them to be 6825 independently moved 6826 according to the 6827 following rules. 6828 - s_waitcnt vmcnt(0) 6829 must happen after 6830 any preceding 6831 global/generic load 6832 atomic/atomicrmw 6833 with an equal or 6834 wider sync scope 6835 and memory ordering 6836 stronger than 6837 unordered (this is 6838 termed the 6839 fence-paired-atomic). 6840 - s_waitcnt lgkmcnt(0) 6841 must happen after 6842 any preceding 6843 local/generic load 6844 atomic/atomicrmw 6845 with an equal or 6846 wider sync scope 6847 and memory ordering 6848 stronger than 6849 unordered (this is 6850 termed the 6851 fence-paired-atomic). 6852 - Must happen before 6853 the following buffer_invl2 and 6854 buffer_wbinvl1_vol. 6855 - Ensures that the 6856 fence-paired atomic 6857 has completed 6858 before invalidating 6859 the 6860 cache. Therefore 6861 any following 6862 locations read must 6863 be no older than 6864 the value read by 6865 the 6866 fence-paired-atomic. 6867 6868 2. buffer_invl2; 6869 buffer_wbinvl1_vol 6870 6871 - Must happen before any 6872 following global/generic 6873 load/load 6874 atomic/store/store 6875 atomic/atomicrmw. 6876 - Ensures that 6877 following loads 6878 will not see stale 6879 global data. 6880 6881 **Release Atomic** 6882 ------------------------------------------------------------------------------------ 6883 store atomic release - singlethread - global 1. buffer/global/flat_store 6884 - wavefront - generic 6885 store atomic release - singlethread - local *If TgSplit execution mode, 6886 - wavefront local address space cannot 6887 be used.* 6888 6889 1. ds_store 6890 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6891 - generic 6892 - Use lgkmcnt(0) if not 6893 TgSplit execution mode 6894 and vmcnt(0) if TgSplit 6895 execution mode. 6896 - If OpenCL, omit lgkmcnt(0). 6897 - s_waitcnt vmcnt(0) 6898 must happen after 6899 any preceding 6900 global/generic load/store/ 6901 load atomic/store atomic/ 6902 atomicrmw. 6903 - s_waitcnt lgkmcnt(0) 6904 must happen after 6905 any preceding 6906 local/generic 6907 load/store/load 6908 atomic/store 6909 atomic/atomicrmw. 6910 - Must happen before 6911 the following 6912 store. 6913 - Ensures that all 6914 memory operations 6915 have 6916 completed before 6917 performing the 6918 store that is being 6919 released. 6920 6921 2. buffer/global/flat_store 6922 store atomic release - workgroup - local *If TgSplit execution mode, 6923 local address space cannot 6924 be used.* 6925 6926 1. ds_store 6927 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6928 - generic vmcnt(0) 6929 6930 - If TgSplit execution mode, 6931 omit lgkmcnt(0). 6932 - If OpenCL and 6933 address space is 6934 not generic, omit 6935 lgkmcnt(0). 6936 - Could be split into 6937 separate s_waitcnt 6938 vmcnt(0) and 6939 s_waitcnt 6940 lgkmcnt(0) to allow 6941 them to be 6942 independently moved 6943 according to the 6944 following rules. 6945 - s_waitcnt vmcnt(0) 6946 must happen after 6947 any preceding 6948 global/generic 6949 load/store/load 6950 atomic/store 6951 atomic/atomicrmw. 6952 - s_waitcnt lgkmcnt(0) 6953 must happen after 6954 any preceding 6955 local/generic 6956 load/store/load 6957 atomic/store 6958 atomic/atomicrmw. 6959 - Must happen before 6960 the following 6961 store. 6962 - Ensures that all 6963 memory operations 6964 to memory have 6965 completed before 6966 performing the 6967 store that is being 6968 released. 6969 6970 2. buffer/global/flat_store 6971 store atomic release - system - global 1. buffer_wbl2 6972 - generic 6973 - Must happen before 6974 following s_waitcnt. 6975 - Performs L2 writeback to 6976 ensure previous 6977 global/generic 6978 store/atomicrmw are 6979 visible at system scope. 6980 6981 2. s_waitcnt lgkmcnt(0) & 6982 vmcnt(0) 6983 6984 - If TgSplit execution mode, 6985 omit lgkmcnt(0). 6986 - If OpenCL and 6987 address space is 6988 not generic, omit 6989 lgkmcnt(0). 6990 - Could be split into 6991 separate s_waitcnt 6992 vmcnt(0) and 6993 s_waitcnt 6994 lgkmcnt(0) to allow 6995 them to be 6996 independently moved 6997 according to the 6998 following rules. 6999 - s_waitcnt vmcnt(0) 7000 must happen after any 7001 preceding 7002 global/generic 7003 load/store/load 7004 atomic/store 7005 atomic/atomicrmw. 7006 - s_waitcnt lgkmcnt(0) 7007 must happen after any 7008 preceding 7009 local/generic 7010 load/store/load 7011 atomic/store 7012 atomic/atomicrmw. 7013 - Must happen before 7014 the following 7015 store. 7016 - Ensures that all 7017 memory operations 7018 to memory and the L2 7019 writeback have 7020 completed before 7021 performing the 7022 store that is being 7023 released. 7024 7025 2. buffer/global/flat_store 7026 scc=1 7027 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7028 - wavefront - generic 7029 atomicrmw release - singlethread - local *If TgSplit execution mode, 7030 - wavefront local address space cannot 7031 be used.* 7032 7033 1. ds_atomic 7034 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7035 - generic 7036 - Use lgkmcnt(0) if not 7037 TgSplit execution mode 7038 and vmcnt(0) if TgSplit 7039 execution mode. 7040 - If OpenCL, omit 7041 lgkmcnt(0). 7042 - s_waitcnt vmcnt(0) 7043 must happen after 7044 any preceding 7045 global/generic load/store/ 7046 load atomic/store atomic/ 7047 atomicrmw. 7048 - s_waitcnt lgkmcnt(0) 7049 must happen after 7050 any preceding 7051 local/generic 7052 load/store/load 7053 atomic/store 7054 atomic/atomicrmw. 7055 - Must happen before 7056 the following 7057 atomicrmw. 7058 - Ensures that all 7059 memory operations 7060 have 7061 completed before 7062 performing the 7063 atomicrmw that is 7064 being released. 7065 7066 2. buffer/global/flat_atomic 7067 atomicrmw release - workgroup - local *If TgSplit execution mode, 7068 local address space cannot 7069 be used.* 7070 7071 1. ds_atomic 7072 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7073 - generic vmcnt(0) 7074 7075 - If TgSplit execution mode, 7076 omit lgkmcnt(0). 7077 - If OpenCL, omit 7078 lgkmcnt(0). 7079 - Could be split into 7080 separate s_waitcnt 7081 vmcnt(0) and 7082 s_waitcnt 7083 lgkmcnt(0) to allow 7084 them to be 7085 independently moved 7086 according to the 7087 following rules. 7088 - s_waitcnt vmcnt(0) 7089 must happen after 7090 any preceding 7091 global/generic 7092 load/store/load 7093 atomic/store 7094 atomic/atomicrmw. 7095 - s_waitcnt lgkmcnt(0) 7096 must happen after 7097 any preceding 7098 local/generic 7099 load/store/load 7100 atomic/store 7101 atomic/atomicrmw. 7102 - Must happen before 7103 the following 7104 atomicrmw. 7105 - Ensures that all 7106 memory operations 7107 to global and local 7108 have completed 7109 before performing 7110 the atomicrmw that 7111 is being released. 7112 7113 2. buffer/global/flat_atomic 7114 atomicrmw release - system - global 1. buffer_wbl2 7115 - generic 7116 - Must happen before 7117 following s_waitcnt. 7118 - Performs L2 writeback to 7119 ensure previous 7120 global/generic 7121 store/atomicrmw are 7122 visible at system scope. 7123 7124 2. s_waitcnt lgkmcnt(0) & 7125 vmcnt(0) 7126 7127 - If TgSplit execution mode, 7128 omit lgkmcnt(0). 7129 - If OpenCL, omit 7130 lgkmcnt(0). 7131 - Could be split into 7132 separate s_waitcnt 7133 vmcnt(0) and 7134 s_waitcnt 7135 lgkmcnt(0) to allow 7136 them to be 7137 independently moved 7138 according to the 7139 following rules. 7140 - s_waitcnt vmcnt(0) 7141 must happen after 7142 any preceding 7143 global/generic 7144 load/store/load 7145 atomic/store 7146 atomic/atomicrmw. 7147 - s_waitcnt lgkmcnt(0) 7148 must happen after 7149 any preceding 7150 local/generic 7151 load/store/load 7152 atomic/store 7153 atomic/atomicrmw. 7154 - Must happen before 7155 the following 7156 atomicrmw. 7157 - Ensures that all 7158 memory operations 7159 to memory and the L2 7160 writeback have 7161 completed before 7162 performing the 7163 store that is being 7164 released. 7165 7166 3. buffer/global/flat_atomic 7167 scc=1 7168 fence release - singlethread *none* *none* 7169 - wavefront 7170 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7171 7172 - Use lgkmcnt(0) if not 7173 TgSplit execution mode 7174 and vmcnt(0) if TgSplit 7175 execution mode. 7176 - If OpenCL and 7177 address space is 7178 not generic, omit 7179 lgkmcnt(0). 7180 - If OpenCL and 7181 address space is 7182 local, omit 7183 vmcnt(0). 7184 - However, since LLVM 7185 currently has no 7186 address space on 7187 the fence need to 7188 conservatively 7189 always generate. If 7190 fence had an 7191 address space then 7192 set to address 7193 space of OpenCL 7194 fence flag, or to 7195 generic if both 7196 local and global 7197 flags are 7198 specified. 7199 - s_waitcnt vmcnt(0) 7200 must happen after 7201 any preceding 7202 global/generic 7203 load/store/ 7204 load atomic/store atomic/ 7205 atomicrmw. 7206 - s_waitcnt lgkmcnt(0) 7207 must happen after 7208 any preceding 7209 local/generic 7210 load/load 7211 atomic/store/store 7212 atomic/atomicrmw. 7213 - Must happen before 7214 any following store 7215 atomic/atomicrmw 7216 with an equal or 7217 wider sync scope 7218 and memory ordering 7219 stronger than 7220 unordered (this is 7221 termed the 7222 fence-paired-atomic). 7223 - Ensures that all 7224 memory operations 7225 have 7226 completed before 7227 performing the 7228 following 7229 fence-paired-atomic. 7230 7231 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7232 vmcnt(0) 7233 7234 - If TgSplit execution mode, 7235 omit lgkmcnt(0). 7236 - If OpenCL and 7237 address space is 7238 not generic, omit 7239 lgkmcnt(0). 7240 - If OpenCL and 7241 address space is 7242 local, omit 7243 vmcnt(0). 7244 - However, since LLVM 7245 currently has no 7246 address space on 7247 the fence need to 7248 conservatively 7249 always generate. If 7250 fence had an 7251 address space then 7252 set to address 7253 space of OpenCL 7254 fence flag, or to 7255 generic if both 7256 local and global 7257 flags are 7258 specified. 7259 - Could be split into 7260 separate s_waitcnt 7261 vmcnt(0) and 7262 s_waitcnt 7263 lgkmcnt(0) to allow 7264 them to be 7265 independently moved 7266 according to the 7267 following rules. 7268 - s_waitcnt vmcnt(0) 7269 must happen after 7270 any preceding 7271 global/generic 7272 load/store/load 7273 atomic/store 7274 atomic/atomicrmw. 7275 - s_waitcnt lgkmcnt(0) 7276 must happen after 7277 any preceding 7278 local/generic 7279 load/store/load 7280 atomic/store 7281 atomic/atomicrmw. 7282 - Must happen before 7283 any following store 7284 atomic/atomicrmw 7285 with an equal or 7286 wider sync scope 7287 and memory ordering 7288 stronger than 7289 unordered (this is 7290 termed the 7291 fence-paired-atomic). 7292 - Ensures that all 7293 memory operations 7294 have 7295 completed before 7296 performing the 7297 following 7298 fence-paired-atomic. 7299 7300 fence release - system *none* 1. buffer_wbl2 7301 7302 - If OpenCL and 7303 address space is 7304 local, omit. 7305 - Must happen before 7306 following s_waitcnt. 7307 - Performs L2 writeback to 7308 ensure previous 7309 global/generic 7310 store/atomicrmw are 7311 visible at system scope. 7312 7313 2. s_waitcnt lgkmcnt(0) & 7314 vmcnt(0) 7315 7316 - If TgSplit execution mode, 7317 omit lgkmcnt(0). 7318 - If OpenCL and 7319 address space is 7320 not generic, omit 7321 lgkmcnt(0). 7322 - If OpenCL and 7323 address space is 7324 local, omit 7325 vmcnt(0). 7326 - However, since LLVM 7327 currently has no 7328 address space on 7329 the fence need to 7330 conservatively 7331 always generate. If 7332 fence had an 7333 address space then 7334 set to address 7335 space of OpenCL 7336 fence flag, or to 7337 generic if both 7338 local and global 7339 flags are 7340 specified. 7341 - Could be split into 7342 separate s_waitcnt 7343 vmcnt(0) and 7344 s_waitcnt 7345 lgkmcnt(0) to allow 7346 them to be 7347 independently moved 7348 according to the 7349 following rules. 7350 - s_waitcnt vmcnt(0) 7351 must happen after 7352 any preceding 7353 global/generic 7354 load/store/load 7355 atomic/store 7356 atomic/atomicrmw. 7357 - s_waitcnt lgkmcnt(0) 7358 must happen after 7359 any preceding 7360 local/generic 7361 load/store/load 7362 atomic/store 7363 atomic/atomicrmw. 7364 - Must happen before 7365 any following store 7366 atomic/atomicrmw 7367 with an equal or 7368 wider sync scope 7369 and memory ordering 7370 stronger than 7371 unordered (this is 7372 termed the 7373 fence-paired-atomic). 7374 - Ensures that all 7375 memory operations 7376 have 7377 completed before 7378 performing the 7379 following 7380 fence-paired-atomic. 7381 7382 **Acquire-Release Atomic** 7383 ------------------------------------------------------------------------------------ 7384 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7385 - wavefront - generic 7386 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7387 - wavefront local address space cannot 7388 be used.* 7389 7390 1. ds_atomic 7391 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7392 7393 - Use lgkmcnt(0) if not 7394 TgSplit execution mode 7395 and vmcnt(0) if TgSplit 7396 execution mode. 7397 - If OpenCL, omit 7398 lgkmcnt(0). 7399 - Must happen after 7400 any preceding 7401 local/generic 7402 load/store/load 7403 atomic/store 7404 atomic/atomicrmw. 7405 - s_waitcnt vmcnt(0) 7406 must happen after 7407 any preceding 7408 global/generic load/store/ 7409 load atomic/store atomic/ 7410 atomicrmw. 7411 - s_waitcnt lgkmcnt(0) 7412 must happen after 7413 any preceding 7414 local/generic 7415 load/store/load 7416 atomic/store 7417 atomic/atomicrmw. 7418 - Must happen before 7419 the following 7420 atomicrmw. 7421 - Ensures that all 7422 memory operations 7423 have 7424 completed before 7425 performing the 7426 atomicrmw that is 7427 being released. 7428 7429 2. buffer/global_atomic 7430 3. s_waitcnt vmcnt(0) 7431 7432 - If not TgSplit execution 7433 mode, omit. 7434 - Must happen before 7435 the following 7436 buffer_wbinvl1_vol. 7437 - Ensures any 7438 following global 7439 data read is no 7440 older than the 7441 atomicrmw value 7442 being acquired. 7443 7444 4. buffer_wbinvl1_vol 7445 7446 - If not TgSplit execution 7447 mode, omit. 7448 - Ensures that 7449 following 7450 loads will not see 7451 stale data. 7452 7453 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7454 local address space cannot 7455 be used.* 7456 7457 1. ds_atomic 7458 2. s_waitcnt lgkmcnt(0) 7459 7460 - If OpenCL, omit. 7461 - Must happen before 7462 any following 7463 global/generic 7464 load/load 7465 atomic/store/store 7466 atomic/atomicrmw. 7467 - Ensures any 7468 following global 7469 data read is no 7470 older than the local load 7471 atomic value being 7472 acquired. 7473 7474 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7475 7476 - Use lgkmcnt(0) if not 7477 TgSplit execution mode 7478 and vmcnt(0) if TgSplit 7479 execution mode. 7480 - If OpenCL, omit 7481 lgkmcnt(0). 7482 - s_waitcnt vmcnt(0) 7483 must happen after 7484 any preceding 7485 global/generic load/store/ 7486 load atomic/store atomic/ 7487 atomicrmw. 7488 - s_waitcnt lgkmcnt(0) 7489 must happen after 7490 any preceding 7491 local/generic 7492 load/store/load 7493 atomic/store 7494 atomic/atomicrmw. 7495 - Must happen before 7496 the following 7497 atomicrmw. 7498 - Ensures that all 7499 memory operations 7500 have 7501 completed before 7502 performing the 7503 atomicrmw that is 7504 being released. 7505 7506 2. flat_atomic 7507 3. s_waitcnt lgkmcnt(0) & 7508 vmcnt(0) 7509 7510 - If not TgSplit execution 7511 mode, omit vmcnt(0). 7512 - If OpenCL, omit 7513 lgkmcnt(0). 7514 - Must happen before 7515 the following 7516 buffer_wbinvl1_vol and 7517 any following 7518 global/generic 7519 load/load 7520 atomic/store/store 7521 atomic/atomicrmw. 7522 - Ensures any 7523 following global 7524 data read is no 7525 older than a local load 7526 atomic value being 7527 acquired. 7528 7529 3. buffer_wbinvl1_vol 7530 7531 - If not TgSplit execution 7532 mode, omit. 7533 - Ensures that 7534 following 7535 loads will not see 7536 stale data. 7537 7538 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7539 vmcnt(0) 7540 7541 - If TgSplit execution mode, 7542 omit lgkmcnt(0). 7543 - If OpenCL, omit 7544 lgkmcnt(0). 7545 - Could be split into 7546 separate s_waitcnt 7547 vmcnt(0) and 7548 s_waitcnt 7549 lgkmcnt(0) to allow 7550 them to be 7551 independently moved 7552 according to the 7553 following rules. 7554 - s_waitcnt vmcnt(0) 7555 must happen after 7556 any preceding 7557 global/generic 7558 load/store/load 7559 atomic/store 7560 atomic/atomicrmw. 7561 - s_waitcnt lgkmcnt(0) 7562 must happen after 7563 any preceding 7564 local/generic 7565 load/store/load 7566 atomic/store 7567 atomic/atomicrmw. 7568 - Must happen before 7569 the following 7570 atomicrmw. 7571 - Ensures that all 7572 memory operations 7573 to global have 7574 completed before 7575 performing the 7576 atomicrmw that is 7577 being released. 7578 7579 2. buffer/global_atomic 7580 3. s_waitcnt vmcnt(0) 7581 7582 - Must happen before 7583 following 7584 buffer_wbinvl1_vol. 7585 - Ensures the 7586 atomicrmw has 7587 completed before 7588 invalidating the 7589 cache. 7590 7591 4. buffer_wbinvl1_vol 7592 7593 - Must happen before 7594 any following 7595 global/generic 7596 load/load 7597 atomic/atomicrmw. 7598 - Ensures that 7599 following loads 7600 will not see stale 7601 global data. 7602 7603 atomicrmw acq_rel - system - global 1. buffer_wbl2 7604 7605 - Must happen before 7606 following s_waitcnt. 7607 - Performs L2 writeback to 7608 ensure previous 7609 global/generic 7610 store/atomicrmw are 7611 visible at system scope. 7612 7613 2. s_waitcnt lgkmcnt(0) & 7614 vmcnt(0) 7615 7616 - If TgSplit execution mode, 7617 omit lgkmcnt(0). 7618 - If OpenCL, omit 7619 lgkmcnt(0). 7620 - Could be split into 7621 separate s_waitcnt 7622 vmcnt(0) and 7623 s_waitcnt 7624 lgkmcnt(0) to allow 7625 them to be 7626 independently moved 7627 according to the 7628 following rules. 7629 - s_waitcnt vmcnt(0) 7630 must happen after 7631 any preceding 7632 global/generic 7633 load/store/load 7634 atomic/store 7635 atomic/atomicrmw. 7636 - s_waitcnt lgkmcnt(0) 7637 must happen after 7638 any preceding 7639 local/generic 7640 load/store/load 7641 atomic/store 7642 atomic/atomicrmw. 7643 - Must happen before 7644 the following 7645 atomicrmw. 7646 - Ensures that all 7647 memory operations 7648 to global and L2 writeback 7649 have completed before 7650 performing the 7651 atomicrmw that is 7652 being released. 7653 7654 3. buffer/global_atomic 7655 scc=1 7656 4. s_waitcnt vmcnt(0) 7657 7658 - Must happen before 7659 following buffer_invl2 and 7660 buffer_wbinvl1_vol. 7661 - Ensures the 7662 atomicrmw has 7663 completed before 7664 invalidating the 7665 caches. 7666 7667 5. buffer_invl2; 7668 buffer_wbinvl1_vol 7669 7670 - Must happen before 7671 any following 7672 global/generic 7673 load/load 7674 atomic/atomicrmw. 7675 - Ensures that 7676 following loads 7677 will not see stale 7678 MTYPE NC global data. 7679 MTYPE RW and CC memory will 7680 never be stale due to the 7681 memory probes. 7682 7683 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7684 vmcnt(0) 7685 7686 - If TgSplit execution mode, 7687 omit lgkmcnt(0). 7688 - If OpenCL, omit 7689 lgkmcnt(0). 7690 - Could be split into 7691 separate s_waitcnt 7692 vmcnt(0) and 7693 s_waitcnt 7694 lgkmcnt(0) to allow 7695 them to be 7696 independently moved 7697 according to the 7698 following rules. 7699 - s_waitcnt vmcnt(0) 7700 must happen after 7701 any preceding 7702 global/generic 7703 load/store/load 7704 atomic/store 7705 atomic/atomicrmw. 7706 - s_waitcnt lgkmcnt(0) 7707 must happen after 7708 any preceding 7709 local/generic 7710 load/store/load 7711 atomic/store 7712 atomic/atomicrmw. 7713 - Must happen before 7714 the following 7715 atomicrmw. 7716 - Ensures that all 7717 memory operations 7718 to global have 7719 completed before 7720 performing the 7721 atomicrmw that is 7722 being released. 7723 7724 2. flat_atomic 7725 3. s_waitcnt vmcnt(0) & 7726 lgkmcnt(0) 7727 7728 - If TgSplit execution mode, 7729 omit lgkmcnt(0). 7730 - If OpenCL, omit 7731 lgkmcnt(0). 7732 - Must happen before 7733 following 7734 buffer_wbinvl1_vol. 7735 - Ensures the 7736 atomicrmw has 7737 completed before 7738 invalidating the 7739 cache. 7740 7741 4. buffer_wbinvl1_vol 7742 7743 - Must happen before 7744 any following 7745 global/generic 7746 load/load 7747 atomic/atomicrmw. 7748 - Ensures that 7749 following loads 7750 will not see stale 7751 global data. 7752 7753 atomicrmw acq_rel - system - generic 1. buffer_wbl2 7754 7755 - Must happen before 7756 following s_waitcnt. 7757 - Performs L2 writeback to 7758 ensure previous 7759 global/generic 7760 store/atomicrmw are 7761 visible at system scope. 7762 7763 2. s_waitcnt lgkmcnt(0) & 7764 vmcnt(0) 7765 7766 - If TgSplit execution mode, 7767 omit lgkmcnt(0). 7768 - If OpenCL, omit 7769 lgkmcnt(0). 7770 - Could be split into 7771 separate s_waitcnt 7772 vmcnt(0) and 7773 s_waitcnt 7774 lgkmcnt(0) to allow 7775 them to be 7776 independently moved 7777 according to the 7778 following rules. 7779 - s_waitcnt vmcnt(0) 7780 must happen after 7781 any preceding 7782 global/generic 7783 load/store/load 7784 atomic/store 7785 atomic/atomicrmw. 7786 - s_waitcnt lgkmcnt(0) 7787 must happen after 7788 any preceding 7789 local/generic 7790 load/store/load 7791 atomic/store 7792 atomic/atomicrmw. 7793 - Must happen before 7794 the following 7795 atomicrmw. 7796 - Ensures that all 7797 memory operations 7798 to global and L2 writeback 7799 have completed before 7800 performing the 7801 atomicrmw that is 7802 being released. 7803 7804 3. flat_atomic scc=1 7805 4. s_waitcnt vmcnt(0) & 7806 lgkmcnt(0) 7807 7808 - If TgSplit execution mode, 7809 omit lgkmcnt(0). 7810 - If OpenCL, omit 7811 lgkmcnt(0). 7812 - Must happen before 7813 following buffer_invl2 and 7814 buffer_wbinvl1_vol. 7815 - Ensures the 7816 atomicrmw has 7817 completed before 7818 invalidating the 7819 caches. 7820 7821 5. buffer_invl2; 7822 buffer_wbinvl1_vol 7823 7824 - Must happen before 7825 any following 7826 global/generic 7827 load/load 7828 atomic/atomicrmw. 7829 - Ensures that 7830 following loads 7831 will not see stale 7832 MTYPE NC global data. 7833 MTYPE RW and CC memory will 7834 never be stale due to the 7835 memory probes. 7836 7837 fence acq_rel - singlethread *none* *none* 7838 - wavefront 7839 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7840 7841 - Use lgkmcnt(0) if not 7842 TgSplit execution mode 7843 and vmcnt(0) if TgSplit 7844 execution mode. 7845 - If OpenCL and 7846 address space is 7847 not generic, omit 7848 lgkmcnt(0). 7849 - If OpenCL and 7850 address space is 7851 local, omit 7852 vmcnt(0). 7853 - However, 7854 since LLVM 7855 currently has no 7856 address space on 7857 the fence need to 7858 conservatively 7859 always generate 7860 (see comment for 7861 previous fence). 7862 - s_waitcnt vmcnt(0) 7863 must happen after 7864 any preceding 7865 global/generic 7866 load/store/ 7867 load atomic/store atomic/ 7868 atomicrmw. 7869 - s_waitcnt lgkmcnt(0) 7870 must happen after 7871 any preceding 7872 local/generic 7873 load/load 7874 atomic/store/store 7875 atomic/atomicrmw. 7876 - Must happen before 7877 any following 7878 global/generic 7879 load/load 7880 atomic/store/store 7881 atomic/atomicrmw. 7882 - Ensures that all 7883 memory operations 7884 have 7885 completed before 7886 performing any 7887 following global 7888 memory operations. 7889 - Ensures that the 7890 preceding 7891 local/generic load 7892 atomic/atomicrmw 7893 with an equal or 7894 wider sync scope 7895 and memory ordering 7896 stronger than 7897 unordered (this is 7898 termed the 7899 acquire-fence-paired-atomic) 7900 has completed 7901 before following 7902 global memory 7903 operations. This 7904 satisfies the 7905 requirements of 7906 acquire. 7907 - Ensures that all 7908 previous memory 7909 operations have 7910 completed before a 7911 following 7912 local/generic store 7913 atomic/atomicrmw 7914 with an equal or 7915 wider sync scope 7916 and memory ordering 7917 stronger than 7918 unordered (this is 7919 termed the 7920 release-fence-paired-atomic). 7921 This satisfies the 7922 requirements of 7923 release. 7924 - Must happen before 7925 the following 7926 buffer_wbinvl1_vol. 7927 - Ensures that the 7928 acquire-fence-paired 7929 atomic has completed 7930 before invalidating 7931 the 7932 cache. Therefore 7933 any following 7934 locations read must 7935 be no older than 7936 the value read by 7937 the 7938 acquire-fence-paired-atomic. 7939 7940 3. buffer_wbinvl1_vol 7941 7942 - If not TgSplit execution 7943 mode, omit. 7944 - Ensures that 7945 following 7946 loads will not see 7947 stale data. 7948 7949 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7950 vmcnt(0) 7951 7952 - If TgSplit execution mode, 7953 omit lgkmcnt(0). 7954 - If OpenCL and 7955 address space is 7956 not generic, omit 7957 lgkmcnt(0). 7958 - However, since LLVM 7959 currently has no 7960 address space on 7961 the fence need to 7962 conservatively 7963 always generate 7964 (see comment for 7965 previous fence). 7966 - Could be split into 7967 separate s_waitcnt 7968 vmcnt(0) and 7969 s_waitcnt 7970 lgkmcnt(0) to allow 7971 them to be 7972 independently moved 7973 according to the 7974 following rules. 7975 - s_waitcnt vmcnt(0) 7976 must happen after 7977 any preceding 7978 global/generic 7979 load/store/load 7980 atomic/store 7981 atomic/atomicrmw. 7982 - s_waitcnt lgkmcnt(0) 7983 must happen after 7984 any preceding 7985 local/generic 7986 load/store/load 7987 atomic/store 7988 atomic/atomicrmw. 7989 - Must happen before 7990 the following 7991 buffer_wbinvl1_vol. 7992 - Ensures that the 7993 preceding 7994 global/local/generic 7995 load 7996 atomic/atomicrmw 7997 with an equal or 7998 wider sync scope 7999 and memory ordering 8000 stronger than 8001 unordered (this is 8002 termed the 8003 acquire-fence-paired-atomic) 8004 has completed 8005 before invalidating 8006 the cache. This 8007 satisfies the 8008 requirements of 8009 acquire. 8010 - Ensures that all 8011 previous memory 8012 operations have 8013 completed before a 8014 following 8015 global/local/generic 8016 store 8017 atomic/atomicrmw 8018 with an equal or 8019 wider sync scope 8020 and memory ordering 8021 stronger than 8022 unordered (this is 8023 termed the 8024 release-fence-paired-atomic). 8025 This satisfies the 8026 requirements of 8027 release. 8028 8029 2. buffer_wbinvl1_vol 8030 8031 - Must happen before 8032 any following 8033 global/generic 8034 load/load 8035 atomic/store/store 8036 atomic/atomicrmw. 8037 - Ensures that 8038 following loads 8039 will not see stale 8040 global data. This 8041 satisfies the 8042 requirements of 8043 acquire. 8044 8045 fence acq_rel - system *none* 1. buffer_wbl2 8046 8047 - If OpenCL and 8048 address space is 8049 local, omit. 8050 - Must happen before 8051 following s_waitcnt. 8052 - Performs L2 writeback to 8053 ensure previous 8054 global/generic 8055 store/atomicrmw are 8056 visible at system scope. 8057 8058 2. s_waitcnt lgkmcnt(0) & 8059 vmcnt(0) 8060 8061 - If TgSplit execution mode, 8062 omit lgkmcnt(0). 8063 - If OpenCL and 8064 address space is 8065 not generic, omit 8066 lgkmcnt(0). 8067 - However, since LLVM 8068 currently has no 8069 address space on 8070 the fence need to 8071 conservatively 8072 always generate 8073 (see comment for 8074 previous fence). 8075 - Could be split into 8076 separate s_waitcnt 8077 vmcnt(0) and 8078 s_waitcnt 8079 lgkmcnt(0) to allow 8080 them to be 8081 independently moved 8082 according to the 8083 following rules. 8084 - s_waitcnt vmcnt(0) 8085 must happen after 8086 any preceding 8087 global/generic 8088 load/store/load 8089 atomic/store 8090 atomic/atomicrmw. 8091 - s_waitcnt lgkmcnt(0) 8092 must happen after 8093 any preceding 8094 local/generic 8095 load/store/load 8096 atomic/store 8097 atomic/atomicrmw. 8098 - Must happen before 8099 the following buffer_invl2 and 8100 buffer_wbinvl1_vol. 8101 - Ensures that the 8102 preceding 8103 global/local/generic 8104 load 8105 atomic/atomicrmw 8106 with an equal or 8107 wider sync scope 8108 and memory ordering 8109 stronger than 8110 unordered (this is 8111 termed the 8112 acquire-fence-paired-atomic) 8113 has completed 8114 before invalidating 8115 the cache. This 8116 satisfies the 8117 requirements of 8118 acquire. 8119 - Ensures that all 8120 previous memory 8121 operations have 8122 completed before a 8123 following 8124 global/local/generic 8125 store 8126 atomic/atomicrmw 8127 with an equal or 8128 wider sync scope 8129 and memory ordering 8130 stronger than 8131 unordered (this is 8132 termed the 8133 release-fence-paired-atomic). 8134 This satisfies the 8135 requirements of 8136 release. 8137 8138 3. buffer_invl2; 8139 buffer_wbinvl1_vol 8140 8141 - Must happen before 8142 any following 8143 global/generic 8144 load/load 8145 atomic/store/store 8146 atomic/atomicrmw. 8147 - Ensures that 8148 following loads 8149 will not see stale 8150 MTYPE NC global data. 8151 MTYPE RW and CC memory will 8152 never be stale due to the 8153 memory probes. 8154 8155 **Sequential Consistent Atomic** 8156 ------------------------------------------------------------------------------------ 8157 load atomic seq_cst - singlethread - global *Same as corresponding 8158 - wavefront - local load atomic acquire, 8159 - generic except must generated 8160 all instructions even 8161 for OpenCL.* 8162 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8163 - generic 8164 - Use lgkmcnt(0) if not 8165 TgSplit execution mode 8166 and vmcnt(0) if TgSplit 8167 execution mode. 8168 - s_waitcnt lgkmcnt(0) must 8169 happen after 8170 preceding 8171 local/generic load 8172 atomic/store 8173 atomic/atomicrmw 8174 with memory 8175 ordering of seq_cst 8176 and with equal or 8177 wider sync scope. 8178 (Note that seq_cst 8179 fences have their 8180 own s_waitcnt 8181 lgkmcnt(0) and so do 8182 not need to be 8183 considered.) 8184 - s_waitcnt vmcnt(0) 8185 must happen after 8186 preceding 8187 global/generic load 8188 atomic/store 8189 atomic/atomicrmw 8190 with memory 8191 ordering of seq_cst 8192 and with equal or 8193 wider sync scope. 8194 (Note that seq_cst 8195 fences have their 8196 own s_waitcnt 8197 vmcnt(0) and so do 8198 not need to be 8199 considered.) 8200 - Ensures any 8201 preceding 8202 sequential 8203 consistent global/local 8204 memory instructions 8205 have completed 8206 before executing 8207 this sequentially 8208 consistent 8209 instruction. This 8210 prevents reordering 8211 a seq_cst store 8212 followed by a 8213 seq_cst load. (Note 8214 that seq_cst is 8215 stronger than 8216 acquire/release as 8217 the reordering of 8218 load acquire 8219 followed by a store 8220 release is 8221 prevented by the 8222 s_waitcnt of 8223 the release, but 8224 there is nothing 8225 preventing a store 8226 release followed by 8227 load acquire from 8228 completing out of 8229 order. The s_waitcnt 8230 could be placed after 8231 seq_store or before 8232 the seq_load. We 8233 choose the load to 8234 make the s_waitcnt be 8235 as late as possible 8236 so that the store 8237 may have already 8238 completed.) 8239 8240 2. *Following 8241 instructions same as 8242 corresponding load 8243 atomic acquire, 8244 except must generated 8245 all instructions even 8246 for OpenCL.* 8247 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8248 local address space cannot 8249 be used.* 8250 8251 *Same as corresponding 8252 load atomic acquire, 8253 except must generated 8254 all instructions even 8255 for OpenCL.* 8256 8257 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8258 - system - generic vmcnt(0) 8259 8260 - If TgSplit execution mode, 8261 omit lgkmcnt(0). 8262 - Could be split into 8263 separate s_waitcnt 8264 vmcnt(0) 8265 and s_waitcnt 8266 lgkmcnt(0) to allow 8267 them to be 8268 independently moved 8269 according to the 8270 following rules. 8271 - s_waitcnt lgkmcnt(0) 8272 must happen after 8273 preceding 8274 global/generic load 8275 atomic/store 8276 atomic/atomicrmw 8277 with memory 8278 ordering of seq_cst 8279 and with equal or 8280 wider sync scope. 8281 (Note that seq_cst 8282 fences have their 8283 own s_waitcnt 8284 lgkmcnt(0) and so do 8285 not need to be 8286 considered.) 8287 - s_waitcnt vmcnt(0) 8288 must happen after 8289 preceding 8290 global/generic load 8291 atomic/store 8292 atomic/atomicrmw 8293 with memory 8294 ordering of seq_cst 8295 and with equal or 8296 wider sync scope. 8297 (Note that seq_cst 8298 fences have their 8299 own s_waitcnt 8300 vmcnt(0) and so do 8301 not need to be 8302 considered.) 8303 - Ensures any 8304 preceding 8305 sequential 8306 consistent global 8307 memory instructions 8308 have completed 8309 before executing 8310 this sequentially 8311 consistent 8312 instruction. This 8313 prevents reordering 8314 a seq_cst store 8315 followed by a 8316 seq_cst load. (Note 8317 that seq_cst is 8318 stronger than 8319 acquire/release as 8320 the reordering of 8321 load acquire 8322 followed by a store 8323 release is 8324 prevented by the 8325 s_waitcnt of 8326 the release, but 8327 there is nothing 8328 preventing a store 8329 release followed by 8330 load acquire from 8331 completing out of 8332 order. The s_waitcnt 8333 could be placed after 8334 seq_store or before 8335 the seq_load. We 8336 choose the load to 8337 make the s_waitcnt be 8338 as late as possible 8339 so that the store 8340 may have already 8341 completed.) 8342 8343 2. *Following 8344 instructions same as 8345 corresponding load 8346 atomic acquire, 8347 except must generated 8348 all instructions even 8349 for OpenCL.* 8350 store atomic seq_cst - singlethread - global *Same as corresponding 8351 - wavefront - local store atomic release, 8352 - workgroup - generic except must generated 8353 - agent all instructions even 8354 - system for OpenCL.* 8355 atomicrmw seq_cst - singlethread - global *Same as corresponding 8356 - wavefront - local atomicrmw acq_rel, 8357 - workgroup - generic except must generated 8358 - agent all instructions even 8359 - system for OpenCL.* 8360 fence seq_cst - singlethread *none* *Same as corresponding 8361 - wavefront fence acq_rel, 8362 - workgroup except must generated 8363 - agent all instructions even 8364 - system for OpenCL.* 8365 ============ ============ ============== ========== ================================ 8366 8367.. _amdgpu-amdhsa-memory-model-gfx10: 8368 8369Memory Model GFX10 8370++++++++++++++++++ 8371 8372For GFX10: 8373 8374* Each agent has multiple shader arrays (SA). 8375* Each SA has multiple work-group processors (WGP). 8376* Each WGP has multiple compute units (CU). 8377* Each CU has multiple SIMDs that execute wavefronts. 8378* The wavefronts for a single work-group are executed in the same 8379 WGP. In CU wavefront execution mode the wavefronts may be executed by 8380 different SIMDs in the same CU. In WGP wavefront execution mode the 8381 wavefronts may be executed by different SIMDs in different CUs in the same 8382 WGP. 8383* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8384 executing on it. 8385* All LDS operations of a WGP are performed as wavefront wide operations in a 8386 global order and involve no caching. Completion is reported to a wavefront in 8387 execution order. 8388* The LDS memory has multiple request queues shared by the SIMDs of a 8389 WGP. Therefore, the LDS operations performed by different wavefronts of a 8390 work-group can be reordered relative to each other, which can result in 8391 reordering the visibility of vector memory operations with respect to LDS 8392 operations of other wavefronts in the same work-group. A ``s_waitcnt 8393 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8394 vector memory operations between wavefronts of a work-group, but not between 8395 operations performed by the same wavefront. 8396* The vector memory operations are performed as wavefront wide operations. 8397 Completion of load/store/sample operations are reported to a wavefront in 8398 execution order of other load/store/sample operations performed by that 8399 wavefront. 8400* The vector memory operations access a vector L0 cache. There is a single L0 8401 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8402 special action is required for coherence between the lanes of a single 8403 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8404 wavefronts executing in the same work-group as they may be executing on SIMDs 8405 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8406 required for coherence between wavefronts executing in different work-groups 8407 as they may be executing on different WGPs. 8408* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8409 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8410 operations are used in a restricted way so do not impact the memory model. See 8411 :ref:`amdgpu-amdhsa-memory-spaces`. 8412* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8413 the same SA. Therefore, no special action is required for coherence between 8414 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8415 required for coherence between wavefronts executing in different work-groups 8416 as they may be executing on different SAs that access different L1s. 8417* The L1 caches have independent quadrants to service disjoint ranges of virtual 8418 addresses. 8419* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8420 vector and scalar memory operations performed by different wavefronts, whether 8421 executing in the same or different work-groups (which may be executing on 8422 different CUs accessing different L0s), can be reordered relative to each 8423 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8424 synchronization between vector memory operations of different wavefronts. It 8425 ensures a previous vector memory operation has completed before executing a 8426 subsequent vector memory or LDS operation and so can be used to meet the 8427 requirements of acquire, release and sequential consistency. 8428* The L1 caches use an L2 cache shared by all SAs on the same agent. 8429* The L2 cache has independent channels to service disjoint ranges of virtual 8430 addresses. 8431* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8432 quadrant has a separate request queue per L2 channel. Therefore, the vector 8433 and scalar memory operations performed by wavefronts executing in different 8434 work-groups (which may be executing on different SAs) of an agent can be 8435 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8436 required to ensure synchronization between vector memory operations of 8437 different SAs. It ensures a previous vector memory operation has completed 8438 before executing a subsequent vector memory and so can be used to meet the 8439 requirements of acquire, release and sequential consistency. 8440* The L2 cache can be kept coherent with other agents on some targets, or ranges 8441 of virtual addresses can be set up to bypass it to ensure system coherence. 8442 8443Scalar memory operations are only used to access memory that is proven to not 8444change during the execution of the kernel dispatch. This includes constant 8445address space and global address space for program scope ``const`` variables. 8446Therefore, the kernel machine code does not have to maintain the scalar cache to 8447ensure it is coherent with the vector caches. The scalar and vector caches are 8448invalidated between kernel dispatches by CP since constant address space data 8449may change between kernel dispatch executions. See 8450:ref:`amdgpu-amdhsa-memory-spaces`. 8451 8452The one exception is if scalar writes are used to spill SGPR registers. In this 8453case the AMDGPU backend ensures the memory location used to spill is never 8454accessed by vector memory operations at the same time. If scalar writes are used 8455then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8456return since the locations may be used for vector memory instructions by a 8457future wavefront that uses the same scratch area, or a function call that 8458creates a frame at the same address, respectively. There is no need for a 8459``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8460 8461For kernarg backing memory: 8462 8463* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8464* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8465 needing to invalidate the L2 cache. 8466* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8467 so the L2 cache will be coherent with the CPU and other agents. 8468 8469Scratch backing memory (which is used for the private address space) is accessed 8470with MTYPE NC (non-coherent). Since the private address space is only accessed 8471by a single thread, and is always write-before-read, there is never a need to 8472invalidate these entries from the L0 or L1 caches. 8473 8474Wavefronts are executed in native mode with in-order reporting of loads and 8475sample instructions. In this mode vmcnt reports completion of load, atomic with 8476return and sample instructions in order, and the vscnt reports the completion of 8477store and atomic without return in order. See ``MEM_ORDERED`` field in 8478:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8479 8480Wavefronts can be executed in WGP or CU wavefront execution mode: 8481 8482* In WGP wavefront execution mode the wavefronts of a work-group are executed 8483 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8484 CU L0 caches is required for work-group synchronization. Also accesses to L1 8485 at work-group scope need to be explicitly ordered as the accesses from 8486 different CUs are not ordered. 8487* In CU wavefront execution mode the wavefronts of a work-group are executed on 8488 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8489 the work-group access the same L0 which in turn ensures L1 accesses are 8490 ordered and so do not require explicit management of the caches for 8491 work-group synchronization. 8492 8493See ``WGP_MODE`` field in 8494:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8495:ref:`amdgpu-target-features`. 8496 8497The code sequences used to implement the memory model for GFX10 are defined in 8498table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8499 8500 .. table:: AMDHSA Memory Model Code Sequences GFX10 8501 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8502 8503 ============ ============ ============== ========== ================================ 8504 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8505 Ordering Sync Scope Address GFX10 8506 Space 8507 ============ ============ ============== ========== ================================ 8508 **Non-Atomic** 8509 ------------------------------------------------------------------------------------ 8510 load *none* *none* - global - !volatile & !nontemporal 8511 - generic 8512 - private 1. buffer/global/flat_load 8513 - constant 8514 - !volatile & nontemporal 8515 8516 1. buffer/global/flat_load 8517 slc=1 8518 8519 - volatile 8520 8521 1. buffer/global/flat_load 8522 glc=1 dlc=1 8523 2. s_waitcnt vmcnt(0) 8524 8525 - Must happen before 8526 any following volatile 8527 global/generic 8528 load/store. 8529 - Ensures that 8530 volatile 8531 operations to 8532 different 8533 addresses will not 8534 be reordered by 8535 hardware. 8536 8537 load *none* *none* - local 1. ds_load 8538 store *none* *none* - global - !volatile & !nontemporal 8539 - generic 8540 - private 1. buffer/global/flat_store 8541 - constant 8542 - !volatile & nontemporal 8543 8544 1. buffer/global/flat_store 8545 slc=1 8546 8547 - volatile 8548 8549 1. buffer/global/flat_store 8550 2. s_waitcnt vscnt(0) 8551 8552 - Must happen before 8553 any following volatile 8554 global/generic 8555 load/store. 8556 - Ensures that 8557 volatile 8558 operations to 8559 different 8560 addresses will not 8561 be reordered by 8562 hardware. 8563 8564 store *none* *none* - local 1. ds_store 8565 **Unordered Atomic** 8566 ------------------------------------------------------------------------------------ 8567 load atomic unordered *any* *any* *Same as non-atomic*. 8568 store atomic unordered *any* *any* *Same as non-atomic*. 8569 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8570 **Monotonic Atomic** 8571 ------------------------------------------------------------------------------------ 8572 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8573 - wavefront - generic 8574 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8575 - generic glc=1 8576 8577 - If CU wavefront execution 8578 mode, omit glc=1. 8579 8580 load atomic monotonic - singlethread - local 1. ds_load 8581 - wavefront 8582 - workgroup 8583 load atomic monotonic - agent - global 1. buffer/global/flat_load 8584 - system - generic glc=1 dlc=1 8585 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8586 - wavefront - generic 8587 - workgroup 8588 - agent 8589 - system 8590 store atomic monotonic - singlethread - local 1. ds_store 8591 - wavefront 8592 - workgroup 8593 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8594 - wavefront - generic 8595 - workgroup 8596 - agent 8597 - system 8598 atomicrmw monotonic - singlethread - local 1. ds_atomic 8599 - wavefront 8600 - workgroup 8601 **Acquire Atomic** 8602 ------------------------------------------------------------------------------------ 8603 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8604 - wavefront - local 8605 - generic 8606 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8607 8608 - If CU wavefront execution 8609 mode, omit glc=1. 8610 8611 2. s_waitcnt vmcnt(0) 8612 8613 - If CU wavefront execution 8614 mode, omit. 8615 - Must happen before 8616 the following buffer_gl0_inv 8617 and before any following 8618 global/generic 8619 load/load 8620 atomic/store/store 8621 atomic/atomicrmw. 8622 8623 3. buffer_gl0_inv 8624 8625 - If CU wavefront execution 8626 mode, omit. 8627 - Ensures that 8628 following 8629 loads will not see 8630 stale data. 8631 8632 load atomic acquire - workgroup - local 1. ds_load 8633 2. s_waitcnt lgkmcnt(0) 8634 8635 - If OpenCL, omit. 8636 - Must happen before 8637 the following buffer_gl0_inv 8638 and before any following 8639 global/generic load/load 8640 atomic/store/store 8641 atomic/atomicrmw. 8642 - Ensures any 8643 following global 8644 data read is no 8645 older than the local load 8646 atomic value being 8647 acquired. 8648 8649 3. buffer_gl0_inv 8650 8651 - If CU wavefront execution 8652 mode, omit. 8653 - If OpenCL, omit. 8654 - Ensures that 8655 following 8656 loads will not see 8657 stale data. 8658 8659 load atomic acquire - workgroup - generic 1. flat_load glc=1 8660 8661 - If CU wavefront execution 8662 mode, omit glc=1. 8663 8664 2. s_waitcnt lgkmcnt(0) & 8665 vmcnt(0) 8666 8667 - If CU wavefront execution 8668 mode, omit vmcnt(0). 8669 - If OpenCL, omit 8670 lgkmcnt(0). 8671 - Must happen before 8672 the following 8673 buffer_gl0_inv and any 8674 following global/generic 8675 load/load 8676 atomic/store/store 8677 atomic/atomicrmw. 8678 - Ensures any 8679 following global 8680 data read is no 8681 older than a local load 8682 atomic value being 8683 acquired. 8684 8685 3. buffer_gl0_inv 8686 8687 - If CU wavefront execution 8688 mode, omit. 8689 - Ensures that 8690 following 8691 loads will not see 8692 stale data. 8693 8694 load atomic acquire - agent - global 1. buffer/global_load 8695 - system glc=1 dlc=1 8696 2. s_waitcnt vmcnt(0) 8697 8698 - Must happen before 8699 following 8700 buffer_gl*_inv. 8701 - Ensures the load 8702 has completed 8703 before invalidating 8704 the caches. 8705 8706 3. buffer_gl0_inv; 8707 buffer_gl1_inv 8708 8709 - Must happen before 8710 any following 8711 global/generic 8712 load/load 8713 atomic/atomicrmw. 8714 - Ensures that 8715 following 8716 loads will not see 8717 stale global data. 8718 8719 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8720 - system 2. s_waitcnt vmcnt(0) & 8721 lgkmcnt(0) 8722 8723 - If OpenCL omit 8724 lgkmcnt(0). 8725 - Must happen before 8726 following 8727 buffer_gl*_invl. 8728 - Ensures the flat_load 8729 has completed 8730 before invalidating 8731 the caches. 8732 8733 3. buffer_gl0_inv; 8734 buffer_gl1_inv 8735 8736 - Must happen before 8737 any following 8738 global/generic 8739 load/load 8740 atomic/atomicrmw. 8741 - Ensures that 8742 following loads 8743 will not see stale 8744 global data. 8745 8746 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8747 - wavefront - local 8748 - generic 8749 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8750 2. s_waitcnt vm/vscnt(0) 8751 8752 - If CU wavefront execution 8753 mode, omit. 8754 - Use vmcnt(0) if atomic with 8755 return and vscnt(0) if 8756 atomic with no-return. 8757 - Must happen before 8758 the following buffer_gl0_inv 8759 and before any following 8760 global/generic 8761 load/load 8762 atomic/store/store 8763 atomic/atomicrmw. 8764 8765 3. buffer_gl0_inv 8766 8767 - If CU wavefront execution 8768 mode, omit. 8769 - Ensures that 8770 following 8771 loads will not see 8772 stale data. 8773 8774 atomicrmw acquire - workgroup - local 1. ds_atomic 8775 2. s_waitcnt lgkmcnt(0) 8776 8777 - If OpenCL, omit. 8778 - Must happen before 8779 the following 8780 buffer_gl0_inv. 8781 - Ensures any 8782 following global 8783 data read is no 8784 older than the local 8785 atomicrmw value 8786 being acquired. 8787 8788 3. buffer_gl0_inv 8789 8790 - If OpenCL omit. 8791 - Ensures that 8792 following 8793 loads will not see 8794 stale data. 8795 8796 atomicrmw acquire - workgroup - generic 1. flat_atomic 8797 2. s_waitcnt lgkmcnt(0) & 8798 vm/vscnt(0) 8799 8800 - If CU wavefront execution 8801 mode, omit vm/vscnt(0). 8802 - If OpenCL, omit lgkmcnt(0). 8803 - Use vmcnt(0) if atomic with 8804 return and vscnt(0) if 8805 atomic with no-return. 8806 - Must happen before 8807 the following 8808 buffer_gl0_inv. 8809 - Ensures any 8810 following global 8811 data read is no 8812 older than a local 8813 atomicrmw value 8814 being acquired. 8815 8816 3. buffer_gl0_inv 8817 8818 - If CU wavefront execution 8819 mode, omit. 8820 - Ensures that 8821 following 8822 loads will not see 8823 stale data. 8824 8825 atomicrmw acquire - agent - global 1. buffer/global_atomic 8826 - system 2. s_waitcnt vm/vscnt(0) 8827 8828 - Use vmcnt(0) if atomic with 8829 return and vscnt(0) if 8830 atomic with no-return. 8831 - Must happen before 8832 following 8833 buffer_gl*_inv. 8834 - Ensures the 8835 atomicrmw has 8836 completed before 8837 invalidating the 8838 caches. 8839 8840 3. buffer_gl0_inv; 8841 buffer_gl1_inv 8842 8843 - Must happen before 8844 any following 8845 global/generic 8846 load/load 8847 atomic/atomicrmw. 8848 - Ensures that 8849 following loads 8850 will not see stale 8851 global data. 8852 8853 atomicrmw acquire - agent - generic 1. flat_atomic 8854 - system 2. s_waitcnt vm/vscnt(0) & 8855 lgkmcnt(0) 8856 8857 - If OpenCL, omit 8858 lgkmcnt(0). 8859 - Use vmcnt(0) if atomic with 8860 return and vscnt(0) if 8861 atomic with no-return. 8862 - Must happen before 8863 following 8864 buffer_gl*_inv. 8865 - Ensures the 8866 atomicrmw has 8867 completed before 8868 invalidating the 8869 caches. 8870 8871 3. buffer_gl0_inv; 8872 buffer_gl1_inv 8873 8874 - Must happen before 8875 any following 8876 global/generic 8877 load/load 8878 atomic/atomicrmw. 8879 - Ensures that 8880 following loads 8881 will not see stale 8882 global data. 8883 8884 fence acquire - singlethread *none* *none* 8885 - wavefront 8886 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 8887 vmcnt(0) & vscnt(0) 8888 8889 - If CU wavefront execution 8890 mode, omit vmcnt(0) and 8891 vscnt(0). 8892 - If OpenCL and 8893 address space is 8894 not generic, omit 8895 lgkmcnt(0). 8896 - If OpenCL and 8897 address space is 8898 local, omit 8899 vmcnt(0) and vscnt(0). 8900 - However, since LLVM 8901 currently has no 8902 address space on 8903 the fence need to 8904 conservatively 8905 always generate. If 8906 fence had an 8907 address space then 8908 set to address 8909 space of OpenCL 8910 fence flag, or to 8911 generic if both 8912 local and global 8913 flags are 8914 specified. 8915 - Could be split into 8916 separate s_waitcnt 8917 vmcnt(0), s_waitcnt 8918 vscnt(0) and s_waitcnt 8919 lgkmcnt(0) to allow 8920 them to be 8921 independently moved 8922 according to the 8923 following rules. 8924 - s_waitcnt vmcnt(0) 8925 must happen after 8926 any preceding 8927 global/generic load 8928 atomic/ 8929 atomicrmw-with-return-value 8930 with an equal or 8931 wider sync scope 8932 and memory ordering 8933 stronger than 8934 unordered (this is 8935 termed the 8936 fence-paired-atomic). 8937 - s_waitcnt vscnt(0) 8938 must happen after 8939 any preceding 8940 global/generic 8941 atomicrmw-no-return-value 8942 with an equal or 8943 wider sync scope 8944 and memory ordering 8945 stronger than 8946 unordered (this is 8947 termed the 8948 fence-paired-atomic). 8949 - s_waitcnt lgkmcnt(0) 8950 must happen after 8951 any preceding 8952 local/generic load 8953 atomic/atomicrmw 8954 with an equal or 8955 wider sync scope 8956 and memory ordering 8957 stronger than 8958 unordered (this is 8959 termed the 8960 fence-paired-atomic). 8961 - Must happen before 8962 the following 8963 buffer_gl0_inv. 8964 - Ensures that the 8965 fence-paired atomic 8966 has completed 8967 before invalidating 8968 the 8969 cache. Therefore 8970 any following 8971 locations read must 8972 be no older than 8973 the value read by 8974 the 8975 fence-paired-atomic. 8976 8977 3. buffer_gl0_inv 8978 8979 - If CU wavefront execution 8980 mode, omit. 8981 - Ensures that 8982 following 8983 loads will not see 8984 stale data. 8985 8986 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 8987 - system vmcnt(0) & vscnt(0) 8988 8989 - If OpenCL and 8990 address space is 8991 not generic, omit 8992 lgkmcnt(0). 8993 - If OpenCL and 8994 address space is 8995 local, omit 8996 vmcnt(0) and vscnt(0). 8997 - However, since LLVM 8998 currently has no 8999 address space on 9000 the fence need to 9001 conservatively 9002 always generate 9003 (see comment for 9004 previous fence). 9005 - Could be split into 9006 separate s_waitcnt 9007 vmcnt(0), s_waitcnt 9008 vscnt(0) and s_waitcnt 9009 lgkmcnt(0) to allow 9010 them to be 9011 independently moved 9012 according to the 9013 following rules. 9014 - s_waitcnt vmcnt(0) 9015 must happen after 9016 any preceding 9017 global/generic load 9018 atomic/ 9019 atomicrmw-with-return-value 9020 with an equal or 9021 wider sync scope 9022 and memory ordering 9023 stronger than 9024 unordered (this is 9025 termed the 9026 fence-paired-atomic). 9027 - s_waitcnt vscnt(0) 9028 must happen after 9029 any preceding 9030 global/generic 9031 atomicrmw-no-return-value 9032 with an equal or 9033 wider sync scope 9034 and memory ordering 9035 stronger than 9036 unordered (this is 9037 termed the 9038 fence-paired-atomic). 9039 - s_waitcnt lgkmcnt(0) 9040 must happen after 9041 any preceding 9042 local/generic load 9043 atomic/atomicrmw 9044 with an equal or 9045 wider sync scope 9046 and memory ordering 9047 stronger than 9048 unordered (this is 9049 termed the 9050 fence-paired-atomic). 9051 - Must happen before 9052 the following 9053 buffer_gl*_inv. 9054 - Ensures that the 9055 fence-paired atomic 9056 has completed 9057 before invalidating 9058 the 9059 caches. Therefore 9060 any following 9061 locations read must 9062 be no older than 9063 the value read by 9064 the 9065 fence-paired-atomic. 9066 9067 2. buffer_gl0_inv; 9068 buffer_gl1_inv 9069 9070 - Must happen before any 9071 following global/generic 9072 load/load 9073 atomic/store/store 9074 atomic/atomicrmw. 9075 - Ensures that 9076 following loads 9077 will not see stale 9078 global data. 9079 9080 **Release Atomic** 9081 ------------------------------------------------------------------------------------ 9082 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9083 - wavefront - local 9084 - generic 9085 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9086 - generic vmcnt(0) & vscnt(0) 9087 9088 - If CU wavefront execution 9089 mode, omit vmcnt(0) and 9090 vscnt(0). 9091 - If OpenCL, omit 9092 lgkmcnt(0). 9093 - Could be split into 9094 separate s_waitcnt 9095 vmcnt(0), s_waitcnt 9096 vscnt(0) and s_waitcnt 9097 lgkmcnt(0) to allow 9098 them to be 9099 independently moved 9100 according to the 9101 following rules. 9102 - s_waitcnt vmcnt(0) 9103 must happen after 9104 any preceding 9105 global/generic load/load 9106 atomic/ 9107 atomicrmw-with-return-value. 9108 - s_waitcnt vscnt(0) 9109 must happen after 9110 any preceding 9111 global/generic 9112 store/store 9113 atomic/ 9114 atomicrmw-no-return-value. 9115 - s_waitcnt lgkmcnt(0) 9116 must happen after 9117 any preceding 9118 local/generic 9119 load/store/load 9120 atomic/store 9121 atomic/atomicrmw. 9122 - Must happen before 9123 the following 9124 store. 9125 - Ensures that all 9126 memory operations 9127 have 9128 completed before 9129 performing the 9130 store that is being 9131 released. 9132 9133 2. buffer/global/flat_store 9134 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9135 9136 - If CU wavefront execution 9137 mode, omit. 9138 - If OpenCL, omit. 9139 - Could be split into 9140 separate s_waitcnt 9141 vmcnt(0) and s_waitcnt 9142 vscnt(0) to allow 9143 them to be 9144 independently moved 9145 according to the 9146 following rules. 9147 - s_waitcnt vmcnt(0) 9148 must happen after 9149 any preceding 9150 global/generic load/load 9151 atomic/ 9152 atomicrmw-with-return-value. 9153 - s_waitcnt vscnt(0) 9154 must happen after 9155 any preceding 9156 global/generic 9157 store/store atomic/ 9158 atomicrmw-no-return-value. 9159 - Must happen before 9160 the following 9161 store. 9162 - Ensures that all 9163 global memory 9164 operations have 9165 completed before 9166 performing the 9167 store that is being 9168 released. 9169 9170 2. ds_store 9171 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9172 - system - generic vmcnt(0) & vscnt(0) 9173 9174 - If OpenCL and 9175 address space is 9176 not generic, omit 9177 lgkmcnt(0). 9178 - Could be split into 9179 separate s_waitcnt 9180 vmcnt(0), s_waitcnt vscnt(0) 9181 and s_waitcnt 9182 lgkmcnt(0) to allow 9183 them to be 9184 independently moved 9185 according to the 9186 following rules. 9187 - s_waitcnt vmcnt(0) 9188 must happen after 9189 any preceding 9190 global/generic 9191 load/load 9192 atomic/ 9193 atomicrmw-with-return-value. 9194 - s_waitcnt vscnt(0) 9195 must happen after 9196 any preceding 9197 global/generic 9198 store/store atomic/ 9199 atomicrmw-no-return-value. 9200 - s_waitcnt lgkmcnt(0) 9201 must happen after 9202 any preceding 9203 local/generic 9204 load/store/load 9205 atomic/store 9206 atomic/atomicrmw. 9207 - Must happen before 9208 the following 9209 store. 9210 - Ensures that all 9211 memory operations 9212 have 9213 completed before 9214 performing the 9215 store that is being 9216 released. 9217 9218 2. buffer/global/flat_store 9219 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9220 - wavefront - local 9221 - generic 9222 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9223 - generic vmcnt(0) & vscnt(0) 9224 9225 - If CU wavefront execution 9226 mode, omit vmcnt(0) and 9227 vscnt(0). 9228 - If OpenCL, omit lgkmcnt(0). 9229 - Could be split into 9230 separate s_waitcnt 9231 vmcnt(0), s_waitcnt 9232 vscnt(0) and s_waitcnt 9233 lgkmcnt(0) to allow 9234 them to be 9235 independently moved 9236 according to the 9237 following rules. 9238 - s_waitcnt vmcnt(0) 9239 must happen after 9240 any preceding 9241 global/generic load/load 9242 atomic/ 9243 atomicrmw-with-return-value. 9244 - s_waitcnt vscnt(0) 9245 must happen after 9246 any preceding 9247 global/generic 9248 store/store 9249 atomic/ 9250 atomicrmw-no-return-value. 9251 - s_waitcnt lgkmcnt(0) 9252 must happen after 9253 any preceding 9254 local/generic 9255 load/store/load 9256 atomic/store 9257 atomic/atomicrmw. 9258 - Must happen before 9259 the following 9260 atomicrmw. 9261 - Ensures that all 9262 memory operations 9263 have 9264 completed before 9265 performing the 9266 atomicrmw that is 9267 being released. 9268 9269 2. buffer/global/flat_atomic 9270 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9271 9272 - If CU wavefront execution 9273 mode, omit. 9274 - If OpenCL, omit. 9275 - Could be split into 9276 separate s_waitcnt 9277 vmcnt(0) and s_waitcnt 9278 vscnt(0) to allow 9279 them to be 9280 independently moved 9281 according to the 9282 following rules. 9283 - s_waitcnt vmcnt(0) 9284 must happen after 9285 any preceding 9286 global/generic load/load 9287 atomic/ 9288 atomicrmw-with-return-value. 9289 - s_waitcnt vscnt(0) 9290 must happen after 9291 any preceding 9292 global/generic 9293 store/store atomic/ 9294 atomicrmw-no-return-value. 9295 - Must happen before 9296 the following 9297 store. 9298 - Ensures that all 9299 global memory 9300 operations have 9301 completed before 9302 performing the 9303 store that is being 9304 released. 9305 9306 2. ds_atomic 9307 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9308 - system - generic vmcnt(0) & vscnt(0) 9309 9310 - If OpenCL, omit 9311 lgkmcnt(0). 9312 - Could be split into 9313 separate s_waitcnt 9314 vmcnt(0), s_waitcnt 9315 vscnt(0) and s_waitcnt 9316 lgkmcnt(0) to allow 9317 them to be 9318 independently moved 9319 according to the 9320 following rules. 9321 - s_waitcnt vmcnt(0) 9322 must happen after 9323 any preceding 9324 global/generic 9325 load/load atomic/ 9326 atomicrmw-with-return-value. 9327 - s_waitcnt vscnt(0) 9328 must happen after 9329 any preceding 9330 global/generic 9331 store/store atomic/ 9332 atomicrmw-no-return-value. 9333 - s_waitcnt lgkmcnt(0) 9334 must happen after 9335 any preceding 9336 local/generic 9337 load/store/load 9338 atomic/store 9339 atomic/atomicrmw. 9340 - Must happen before 9341 the following 9342 atomicrmw. 9343 - Ensures that all 9344 memory operations 9345 to global and local 9346 have completed 9347 before performing 9348 the atomicrmw that 9349 is being released. 9350 9351 2. buffer/global/flat_atomic 9352 fence release - singlethread *none* *none* 9353 - wavefront 9354 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9355 vmcnt(0) & vscnt(0) 9356 9357 - If CU wavefront execution 9358 mode, omit vmcnt(0) and 9359 vscnt(0). 9360 - If OpenCL and 9361 address space is 9362 not generic, omit 9363 lgkmcnt(0). 9364 - If OpenCL and 9365 address space is 9366 local, omit 9367 vmcnt(0) and vscnt(0). 9368 - However, since LLVM 9369 currently has no 9370 address space on 9371 the fence need to 9372 conservatively 9373 always generate. If 9374 fence had an 9375 address space then 9376 set to address 9377 space of OpenCL 9378 fence flag, or to 9379 generic if both 9380 local and global 9381 flags are 9382 specified. 9383 - Could be split into 9384 separate s_waitcnt 9385 vmcnt(0), s_waitcnt 9386 vscnt(0) and s_waitcnt 9387 lgkmcnt(0) to allow 9388 them to be 9389 independently moved 9390 according to the 9391 following rules. 9392 - s_waitcnt vmcnt(0) 9393 must happen after 9394 any preceding 9395 global/generic 9396 load/load 9397 atomic/ 9398 atomicrmw-with-return-value. 9399 - s_waitcnt vscnt(0) 9400 must happen after 9401 any preceding 9402 global/generic 9403 store/store atomic/ 9404 atomicrmw-no-return-value. 9405 - s_waitcnt lgkmcnt(0) 9406 must happen after 9407 any preceding 9408 local/generic 9409 load/store/load 9410 atomic/store atomic/ 9411 atomicrmw. 9412 - Must happen before 9413 any following store 9414 atomic/atomicrmw 9415 with an equal or 9416 wider sync scope 9417 and memory ordering 9418 stronger than 9419 unordered (this is 9420 termed the 9421 fence-paired-atomic). 9422 - Ensures that all 9423 memory operations 9424 have 9425 completed before 9426 performing the 9427 following 9428 fence-paired-atomic. 9429 9430 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9431 - system vmcnt(0) & vscnt(0) 9432 9433 - If OpenCL and 9434 address space is 9435 not generic, omit 9436 lgkmcnt(0). 9437 - If OpenCL and 9438 address space is 9439 local, omit 9440 vmcnt(0) and vscnt(0). 9441 - However, since LLVM 9442 currently has no 9443 address space on 9444 the fence need to 9445 conservatively 9446 always generate. If 9447 fence had an 9448 address space then 9449 set to address 9450 space of OpenCL 9451 fence flag, or to 9452 generic if both 9453 local and global 9454 flags are 9455 specified. 9456 - Could be split into 9457 separate s_waitcnt 9458 vmcnt(0), s_waitcnt 9459 vscnt(0) and s_waitcnt 9460 lgkmcnt(0) to allow 9461 them to be 9462 independently moved 9463 according to the 9464 following rules. 9465 - s_waitcnt vmcnt(0) 9466 must happen after 9467 any preceding 9468 global/generic 9469 load/load atomic/ 9470 atomicrmw-with-return-value. 9471 - s_waitcnt vscnt(0) 9472 must happen after 9473 any preceding 9474 global/generic 9475 store/store atomic/ 9476 atomicrmw-no-return-value. 9477 - s_waitcnt lgkmcnt(0) 9478 must happen after 9479 any preceding 9480 local/generic 9481 load/store/load 9482 atomic/store 9483 atomic/atomicrmw. 9484 - Must happen before 9485 any following store 9486 atomic/atomicrmw 9487 with an equal or 9488 wider sync scope 9489 and memory ordering 9490 stronger than 9491 unordered (this is 9492 termed the 9493 fence-paired-atomic). 9494 - Ensures that all 9495 memory operations 9496 have 9497 completed before 9498 performing the 9499 following 9500 fence-paired-atomic. 9501 9502 **Acquire-Release Atomic** 9503 ------------------------------------------------------------------------------------ 9504 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9505 - wavefront - local 9506 - generic 9507 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9508 vmcnt(0) & vscnt(0) 9509 9510 - If CU wavefront execution 9511 mode, omit vmcnt(0) and 9512 vscnt(0). 9513 - If OpenCL, omit 9514 lgkmcnt(0). 9515 - Must happen after 9516 any preceding 9517 local/generic 9518 load/store/load 9519 atomic/store 9520 atomic/atomicrmw. 9521 - Could be split into 9522 separate s_waitcnt 9523 vmcnt(0), s_waitcnt 9524 vscnt(0), and s_waitcnt 9525 lgkmcnt(0) to allow 9526 them to be 9527 independently moved 9528 according to the 9529 following rules. 9530 - s_waitcnt vmcnt(0) 9531 must happen after 9532 any preceding 9533 global/generic load/load 9534 atomic/ 9535 atomicrmw-with-return-value. 9536 - s_waitcnt vscnt(0) 9537 must happen after 9538 any preceding 9539 global/generic 9540 store/store 9541 atomic/ 9542 atomicrmw-no-return-value. 9543 - s_waitcnt lgkmcnt(0) 9544 must happen after 9545 any preceding 9546 local/generic 9547 load/store/load 9548 atomic/store 9549 atomic/atomicrmw. 9550 - Must happen before 9551 the following 9552 atomicrmw. 9553 - Ensures that all 9554 memory operations 9555 have 9556 completed before 9557 performing the 9558 atomicrmw that is 9559 being released. 9560 9561 2. buffer/global_atomic 9562 3. s_waitcnt vm/vscnt(0) 9563 9564 - If CU wavefront execution 9565 mode, omit. 9566 - Use vmcnt(0) if atomic with 9567 return and vscnt(0) if 9568 atomic with no-return. 9569 - Must happen before 9570 the following 9571 buffer_gl0_inv. 9572 - Ensures any 9573 following global 9574 data read is no 9575 older than the 9576 atomicrmw value 9577 being acquired. 9578 9579 4. buffer_gl0_inv 9580 9581 - If CU wavefront execution 9582 mode, omit. 9583 - Ensures that 9584 following 9585 loads will not see 9586 stale data. 9587 9588 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9589 9590 - If CU wavefront execution 9591 mode, omit. 9592 - If OpenCL, omit. 9593 - Could be split into 9594 separate s_waitcnt 9595 vmcnt(0) and s_waitcnt 9596 vscnt(0) to allow 9597 them to be 9598 independently moved 9599 according to the 9600 following rules. 9601 - s_waitcnt vmcnt(0) 9602 must happen after 9603 any preceding 9604 global/generic load/load 9605 atomic/ 9606 atomicrmw-with-return-value. 9607 - s_waitcnt vscnt(0) 9608 must happen after 9609 any preceding 9610 global/generic 9611 store/store atomic/ 9612 atomicrmw-no-return-value. 9613 - Must happen before 9614 the following 9615 store. 9616 - Ensures that all 9617 global memory 9618 operations have 9619 completed before 9620 performing the 9621 store that is being 9622 released. 9623 9624 2. ds_atomic 9625 3. s_waitcnt lgkmcnt(0) 9626 9627 - If OpenCL, omit. 9628 - Must happen before 9629 the following 9630 buffer_gl0_inv. 9631 - Ensures any 9632 following global 9633 data read is no 9634 older than the local load 9635 atomic value being 9636 acquired. 9637 9638 4. buffer_gl0_inv 9639 9640 - If CU wavefront execution 9641 mode, omit. 9642 - If OpenCL omit. 9643 - Ensures that 9644 following 9645 loads will not see 9646 stale data. 9647 9648 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9649 vmcnt(0) & vscnt(0) 9650 9651 - If CU wavefront execution 9652 mode, omit vmcnt(0) and 9653 vscnt(0). 9654 - If OpenCL, omit lgkmcnt(0). 9655 - Could be split into 9656 separate s_waitcnt 9657 vmcnt(0), s_waitcnt 9658 vscnt(0) and s_waitcnt 9659 lgkmcnt(0) to allow 9660 them to be 9661 independently moved 9662 according to the 9663 following rules. 9664 - s_waitcnt vmcnt(0) 9665 must happen after 9666 any preceding 9667 global/generic load/load 9668 atomic/ 9669 atomicrmw-with-return-value. 9670 - s_waitcnt vscnt(0) 9671 must happen after 9672 any preceding 9673 global/generic 9674 store/store 9675 atomic/ 9676 atomicrmw-no-return-value. 9677 - s_waitcnt lgkmcnt(0) 9678 must happen after 9679 any preceding 9680 local/generic 9681 load/store/load 9682 atomic/store 9683 atomic/atomicrmw. 9684 - Must happen before 9685 the following 9686 atomicrmw. 9687 - Ensures that all 9688 memory operations 9689 have 9690 completed before 9691 performing the 9692 atomicrmw that is 9693 being released. 9694 9695 2. flat_atomic 9696 3. s_waitcnt lgkmcnt(0) & 9697 vmcnt(0) & vscnt(0) 9698 9699 - If CU wavefront execution 9700 mode, omit vmcnt(0) and 9701 vscnt(0). 9702 - If OpenCL, omit lgkmcnt(0). 9703 - Must happen before 9704 the following 9705 buffer_gl0_inv. 9706 - Ensures any 9707 following global 9708 data read is no 9709 older than the load 9710 atomic value being 9711 acquired. 9712 9713 3. buffer_gl0_inv 9714 9715 - If CU wavefront execution 9716 mode, omit. 9717 - Ensures that 9718 following 9719 loads will not see 9720 stale data. 9721 9722 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9723 - system vmcnt(0) & vscnt(0) 9724 9725 - If OpenCL, omit 9726 lgkmcnt(0). 9727 - Could be split into 9728 separate s_waitcnt 9729 vmcnt(0), s_waitcnt 9730 vscnt(0) and s_waitcnt 9731 lgkmcnt(0) to allow 9732 them to be 9733 independently moved 9734 according to the 9735 following rules. 9736 - s_waitcnt vmcnt(0) 9737 must happen after 9738 any preceding 9739 global/generic 9740 load/load atomic/ 9741 atomicrmw-with-return-value. 9742 - s_waitcnt vscnt(0) 9743 must happen after 9744 any preceding 9745 global/generic 9746 store/store atomic/ 9747 atomicrmw-no-return-value. 9748 - s_waitcnt lgkmcnt(0) 9749 must happen after 9750 any preceding 9751 local/generic 9752 load/store/load 9753 atomic/store 9754 atomic/atomicrmw. 9755 - Must happen before 9756 the following 9757 atomicrmw. 9758 - Ensures that all 9759 memory operations 9760 to global have 9761 completed before 9762 performing the 9763 atomicrmw that is 9764 being released. 9765 9766 2. buffer/global_atomic 9767 3. s_waitcnt vm/vscnt(0) 9768 9769 - Use vmcnt(0) if atomic with 9770 return and vscnt(0) if 9771 atomic with no-return. 9772 - Must happen before 9773 following 9774 buffer_gl*_inv. 9775 - Ensures the 9776 atomicrmw has 9777 completed before 9778 invalidating the 9779 caches. 9780 9781 4. buffer_gl0_inv; 9782 buffer_gl1_inv 9783 9784 - Must happen before 9785 any following 9786 global/generic 9787 load/load 9788 atomic/atomicrmw. 9789 - Ensures that 9790 following loads 9791 will not see stale 9792 global data. 9793 9794 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9795 - system vmcnt(0) & vscnt(0) 9796 9797 - If OpenCL, omit 9798 lgkmcnt(0). 9799 - Could be split into 9800 separate s_waitcnt 9801 vmcnt(0), s_waitcnt 9802 vscnt(0), and s_waitcnt 9803 lgkmcnt(0) to allow 9804 them to be 9805 independently moved 9806 according to the 9807 following rules. 9808 - s_waitcnt vmcnt(0) 9809 must happen after 9810 any preceding 9811 global/generic 9812 load/load atomic 9813 atomicrmw-with-return-value. 9814 - s_waitcnt vscnt(0) 9815 must happen after 9816 any preceding 9817 global/generic 9818 store/store atomic/ 9819 atomicrmw-no-return-value. 9820 - s_waitcnt lgkmcnt(0) 9821 must happen after 9822 any preceding 9823 local/generic 9824 load/store/load 9825 atomic/store 9826 atomic/atomicrmw. 9827 - Must happen before 9828 the following 9829 atomicrmw. 9830 - Ensures that all 9831 memory operations 9832 have 9833 completed before 9834 performing the 9835 atomicrmw that is 9836 being released. 9837 9838 2. flat_atomic 9839 3. s_waitcnt vm/vscnt(0) & 9840 lgkmcnt(0) 9841 9842 - If OpenCL, omit 9843 lgkmcnt(0). 9844 - Use vmcnt(0) if atomic with 9845 return and vscnt(0) if 9846 atomic with no-return. 9847 - Must happen before 9848 following 9849 buffer_gl*_inv. 9850 - Ensures the 9851 atomicrmw has 9852 completed before 9853 invalidating the 9854 caches. 9855 9856 4. buffer_gl0_inv; 9857 buffer_gl1_inv 9858 9859 - Must happen before 9860 any following 9861 global/generic 9862 load/load 9863 atomic/atomicrmw. 9864 - Ensures that 9865 following loads 9866 will not see stale 9867 global data. 9868 9869 fence acq_rel - singlethread *none* *none* 9870 - wavefront 9871 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9872 vmcnt(0) & vscnt(0) 9873 9874 - If CU wavefront execution 9875 mode, omit vmcnt(0) and 9876 vscnt(0). 9877 - If OpenCL and 9878 address space is 9879 not generic, omit 9880 lgkmcnt(0). 9881 - If OpenCL and 9882 address space is 9883 local, omit 9884 vmcnt(0) and vscnt(0). 9885 - However, 9886 since LLVM 9887 currently has no 9888 address space on 9889 the fence need to 9890 conservatively 9891 always generate 9892 (see comment for 9893 previous fence). 9894 - Could be split into 9895 separate s_waitcnt 9896 vmcnt(0), s_waitcnt 9897 vscnt(0) and s_waitcnt 9898 lgkmcnt(0) to allow 9899 them to be 9900 independently moved 9901 according to the 9902 following rules. 9903 - s_waitcnt vmcnt(0) 9904 must happen after 9905 any preceding 9906 global/generic 9907 load/load 9908 atomic/ 9909 atomicrmw-with-return-value. 9910 - s_waitcnt vscnt(0) 9911 must happen after 9912 any preceding 9913 global/generic 9914 store/store atomic/ 9915 atomicrmw-no-return-value. 9916 - s_waitcnt lgkmcnt(0) 9917 must happen after 9918 any preceding 9919 local/generic 9920 load/store/load 9921 atomic/store atomic/ 9922 atomicrmw. 9923 - Must happen before 9924 any following 9925 global/generic 9926 load/load 9927 atomic/store/store 9928 atomic/atomicrmw. 9929 - Ensures that all 9930 memory operations 9931 have 9932 completed before 9933 performing any 9934 following global 9935 memory operations. 9936 - Ensures that the 9937 preceding 9938 local/generic load 9939 atomic/atomicrmw 9940 with an equal or 9941 wider sync scope 9942 and memory ordering 9943 stronger than 9944 unordered (this is 9945 termed the 9946 acquire-fence-paired-atomic) 9947 has completed 9948 before following 9949 global memory 9950 operations. This 9951 satisfies the 9952 requirements of 9953 acquire. 9954 - Ensures that all 9955 previous memory 9956 operations have 9957 completed before a 9958 following 9959 local/generic store 9960 atomic/atomicrmw 9961 with an equal or 9962 wider sync scope 9963 and memory ordering 9964 stronger than 9965 unordered (this is 9966 termed the 9967 release-fence-paired-atomic). 9968 This satisfies the 9969 requirements of 9970 release. 9971 - Must happen before 9972 the following 9973 buffer_gl0_inv. 9974 - Ensures that the 9975 acquire-fence-paired 9976 atomic has completed 9977 before invalidating 9978 the 9979 cache. Therefore 9980 any following 9981 locations read must 9982 be no older than 9983 the value read by 9984 the 9985 acquire-fence-paired-atomic. 9986 9987 3. buffer_gl0_inv 9988 9989 - If CU wavefront execution 9990 mode, omit. 9991 - Ensures that 9992 following 9993 loads will not see 9994 stale data. 9995 9996 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 9997 - system vmcnt(0) & vscnt(0) 9998 9999 - If OpenCL and 10000 address space is 10001 not generic, omit 10002 lgkmcnt(0). 10003 - If OpenCL and 10004 address space is 10005 local, omit 10006 vmcnt(0) and vscnt(0). 10007 - However, since LLVM 10008 currently has no 10009 address space on 10010 the fence need to 10011 conservatively 10012 always generate 10013 (see comment for 10014 previous fence). 10015 - Could be split into 10016 separate s_waitcnt 10017 vmcnt(0), s_waitcnt 10018 vscnt(0) and s_waitcnt 10019 lgkmcnt(0) to allow 10020 them to be 10021 independently moved 10022 according to the 10023 following rules. 10024 - s_waitcnt vmcnt(0) 10025 must happen after 10026 any preceding 10027 global/generic 10028 load/load 10029 atomic/ 10030 atomicrmw-with-return-value. 10031 - s_waitcnt vscnt(0) 10032 must happen after 10033 any preceding 10034 global/generic 10035 store/store atomic/ 10036 atomicrmw-no-return-value. 10037 - s_waitcnt lgkmcnt(0) 10038 must happen after 10039 any preceding 10040 local/generic 10041 load/store/load 10042 atomic/store 10043 atomic/atomicrmw. 10044 - Must happen before 10045 the following 10046 buffer_gl*_inv. 10047 - Ensures that the 10048 preceding 10049 global/local/generic 10050 load 10051 atomic/atomicrmw 10052 with an equal or 10053 wider sync scope 10054 and memory ordering 10055 stronger than 10056 unordered (this is 10057 termed the 10058 acquire-fence-paired-atomic) 10059 has completed 10060 before invalidating 10061 the caches. This 10062 satisfies the 10063 requirements of 10064 acquire. 10065 - Ensures that all 10066 previous memory 10067 operations have 10068 completed before a 10069 following 10070 global/local/generic 10071 store 10072 atomic/atomicrmw 10073 with an equal or 10074 wider sync scope 10075 and memory ordering 10076 stronger than 10077 unordered (this is 10078 termed the 10079 release-fence-paired-atomic). 10080 This satisfies the 10081 requirements of 10082 release. 10083 10084 2. buffer_gl0_inv; 10085 buffer_gl1_inv 10086 10087 - Must happen before 10088 any following 10089 global/generic 10090 load/load 10091 atomic/store/store 10092 atomic/atomicrmw. 10093 - Ensures that 10094 following loads 10095 will not see stale 10096 global data. This 10097 satisfies the 10098 requirements of 10099 acquire. 10100 10101 **Sequential Consistent Atomic** 10102 ------------------------------------------------------------------------------------ 10103 load atomic seq_cst - singlethread - global *Same as corresponding 10104 - wavefront - local load atomic acquire, 10105 - generic except must generated 10106 all instructions even 10107 for OpenCL.* 10108 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10109 - generic vmcnt(0) & vscnt(0) 10110 10111 - If CU wavefront execution 10112 mode, omit vmcnt(0) and 10113 vscnt(0). 10114 - Could be split into 10115 separate s_waitcnt 10116 vmcnt(0), s_waitcnt 10117 vscnt(0), and s_waitcnt 10118 lgkmcnt(0) to allow 10119 them to be 10120 independently moved 10121 according to the 10122 following rules. 10123 - s_waitcnt lgkmcnt(0) must 10124 happen after 10125 preceding 10126 local/generic load 10127 atomic/store 10128 atomic/atomicrmw 10129 with memory 10130 ordering of seq_cst 10131 and with equal or 10132 wider sync scope. 10133 (Note that seq_cst 10134 fences have their 10135 own s_waitcnt 10136 lgkmcnt(0) and so do 10137 not need to be 10138 considered.) 10139 - s_waitcnt vmcnt(0) 10140 must happen after 10141 preceding 10142 global/generic load 10143 atomic/ 10144 atomicrmw-with-return-value 10145 with memory 10146 ordering of seq_cst 10147 and with equal or 10148 wider sync scope. 10149 (Note that seq_cst 10150 fences have their 10151 own s_waitcnt 10152 vmcnt(0) and so do 10153 not need to be 10154 considered.) 10155 - s_waitcnt vscnt(0) 10156 Must happen after 10157 preceding 10158 global/generic store 10159 atomic/ 10160 atomicrmw-no-return-value 10161 with memory 10162 ordering of seq_cst 10163 and with equal or 10164 wider sync scope. 10165 (Note that seq_cst 10166 fences have their 10167 own s_waitcnt 10168 vscnt(0) and so do 10169 not need to be 10170 considered.) 10171 - Ensures any 10172 preceding 10173 sequential 10174 consistent global/local 10175 memory instructions 10176 have completed 10177 before executing 10178 this sequentially 10179 consistent 10180 instruction. This 10181 prevents reordering 10182 a seq_cst store 10183 followed by a 10184 seq_cst load. (Note 10185 that seq_cst is 10186 stronger than 10187 acquire/release as 10188 the reordering of 10189 load acquire 10190 followed by a store 10191 release is 10192 prevented by the 10193 s_waitcnt of 10194 the release, but 10195 there is nothing 10196 preventing a store 10197 release followed by 10198 load acquire from 10199 completing out of 10200 order. The s_waitcnt 10201 could be placed after 10202 seq_store or before 10203 the seq_load. We 10204 choose the load to 10205 make the s_waitcnt be 10206 as late as possible 10207 so that the store 10208 may have already 10209 completed.) 10210 10211 2. *Following 10212 instructions same as 10213 corresponding load 10214 atomic acquire, 10215 except must generated 10216 all instructions even 10217 for OpenCL.* 10218 load atomic seq_cst - workgroup - local 10219 10220 1. s_waitcnt vmcnt(0) & vscnt(0) 10221 10222 - If CU wavefront execution 10223 mode, omit. 10224 - Could be split into 10225 separate s_waitcnt 10226 vmcnt(0) and s_waitcnt 10227 vscnt(0) to allow 10228 them to be 10229 independently moved 10230 according to the 10231 following rules. 10232 - s_waitcnt vmcnt(0) 10233 Must happen after 10234 preceding 10235 global/generic load 10236 atomic/ 10237 atomicrmw-with-return-value 10238 with memory 10239 ordering of seq_cst 10240 and with equal or 10241 wider sync scope. 10242 (Note that seq_cst 10243 fences have their 10244 own s_waitcnt 10245 vmcnt(0) and so do 10246 not need to be 10247 considered.) 10248 - s_waitcnt vscnt(0) 10249 Must happen after 10250 preceding 10251 global/generic store 10252 atomic/ 10253 atomicrmw-no-return-value 10254 with memory 10255 ordering of seq_cst 10256 and with equal or 10257 wider sync scope. 10258 (Note that seq_cst 10259 fences have their 10260 own s_waitcnt 10261 vscnt(0) and so do 10262 not need to be 10263 considered.) 10264 - Ensures any 10265 preceding 10266 sequential 10267 consistent global 10268 memory instructions 10269 have completed 10270 before executing 10271 this sequentially 10272 consistent 10273 instruction. This 10274 prevents reordering 10275 a seq_cst store 10276 followed by a 10277 seq_cst load. (Note 10278 that seq_cst is 10279 stronger than 10280 acquire/release as 10281 the reordering of 10282 load acquire 10283 followed by a store 10284 release is 10285 prevented by the 10286 s_waitcnt of 10287 the release, but 10288 there is nothing 10289 preventing a store 10290 release followed by 10291 load acquire from 10292 completing out of 10293 order. The s_waitcnt 10294 could be placed after 10295 seq_store or before 10296 the seq_load. We 10297 choose the load to 10298 make the s_waitcnt be 10299 as late as possible 10300 so that the store 10301 may have already 10302 completed.) 10303 10304 2. *Following 10305 instructions same as 10306 corresponding load 10307 atomic acquire, 10308 except must generated 10309 all instructions even 10310 for OpenCL.* 10311 10312 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10313 - system - generic vmcnt(0) & vscnt(0) 10314 10315 - Could be split into 10316 separate s_waitcnt 10317 vmcnt(0), s_waitcnt 10318 vscnt(0) and s_waitcnt 10319 lgkmcnt(0) to allow 10320 them to be 10321 independently moved 10322 according to the 10323 following rules. 10324 - s_waitcnt lgkmcnt(0) 10325 must happen after 10326 preceding 10327 local load 10328 atomic/store 10329 atomic/atomicrmw 10330 with memory 10331 ordering of seq_cst 10332 and with equal or 10333 wider sync scope. 10334 (Note that seq_cst 10335 fences have their 10336 own s_waitcnt 10337 lgkmcnt(0) and so do 10338 not need to be 10339 considered.) 10340 - s_waitcnt vmcnt(0) 10341 must happen after 10342 preceding 10343 global/generic load 10344 atomic/ 10345 atomicrmw-with-return-value 10346 with memory 10347 ordering of seq_cst 10348 and with equal or 10349 wider sync scope. 10350 (Note that seq_cst 10351 fences have their 10352 own s_waitcnt 10353 vmcnt(0) and so do 10354 not need to be 10355 considered.) 10356 - s_waitcnt vscnt(0) 10357 Must happen after 10358 preceding 10359 global/generic store 10360 atomic/ 10361 atomicrmw-no-return-value 10362 with memory 10363 ordering of seq_cst 10364 and with equal or 10365 wider sync scope. 10366 (Note that seq_cst 10367 fences have their 10368 own s_waitcnt 10369 vscnt(0) and so do 10370 not need to be 10371 considered.) 10372 - Ensures any 10373 preceding 10374 sequential 10375 consistent global 10376 memory instructions 10377 have completed 10378 before executing 10379 this sequentially 10380 consistent 10381 instruction. This 10382 prevents reordering 10383 a seq_cst store 10384 followed by a 10385 seq_cst load. (Note 10386 that seq_cst is 10387 stronger than 10388 acquire/release as 10389 the reordering of 10390 load acquire 10391 followed by a store 10392 release is 10393 prevented by the 10394 s_waitcnt of 10395 the release, but 10396 there is nothing 10397 preventing a store 10398 release followed by 10399 load acquire from 10400 completing out of 10401 order. The s_waitcnt 10402 could be placed after 10403 seq_store or before 10404 the seq_load. We 10405 choose the load to 10406 make the s_waitcnt be 10407 as late as possible 10408 so that the store 10409 may have already 10410 completed.) 10411 10412 2. *Following 10413 instructions same as 10414 corresponding load 10415 atomic acquire, 10416 except must generated 10417 all instructions even 10418 for OpenCL.* 10419 store atomic seq_cst - singlethread - global *Same as corresponding 10420 - wavefront - local store atomic release, 10421 - workgroup - generic except must generated 10422 - agent all instructions even 10423 - system for OpenCL.* 10424 atomicrmw seq_cst - singlethread - global *Same as corresponding 10425 - wavefront - local atomicrmw acq_rel, 10426 - workgroup - generic except must generated 10427 - agent all instructions even 10428 - system for OpenCL.* 10429 fence seq_cst - singlethread *none* *Same as corresponding 10430 - wavefront fence acq_rel, 10431 - workgroup except must generated 10432 - agent all instructions even 10433 - system for OpenCL.* 10434 ============ ============ ============== ========== ================================ 10435 10436Trap Handler ABI 10437~~~~~~~~~~~~~~~~ 10438 10439For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10440runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10441supports the ``s_trap`` instruction. For usage see: 10442 10443- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10444- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10445- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10446 10447 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10448 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10449 10450 =================== =============== =============== ======================================= 10451 Usage Code Sequence Trap Handler Description 10452 Inputs 10453 =================== =============== =============== ======================================= 10454 reserved ``s_trap 0x00`` Reserved by hardware. 10455 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10456 ``queue_ptr`` intrinsic (not implemented). 10457 ``VGPR0``: 10458 ``arg`` 10459 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10460 ``queue_ptr`` the trap instruction. The associated 10461 queue is signalled to put it into the 10462 error state. When the queue is put in 10463 the error state, the waves executing 10464 dispatches on the queue will be 10465 terminated. 10466 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10467 as a no-operation. The trap handler 10468 is entered and immediately returns to 10469 continue execution of the wavefront. 10470 - If the debugger is enabled, causes 10471 the debug trap to be reported by the 10472 debugger and the wavefront is put in 10473 the halt state with the PC at the 10474 instruction. The debugger must 10475 increment the PC and resume the wave. 10476 reserved ``s_trap 0x04`` Reserved. 10477 reserved ``s_trap 0x05`` Reserved. 10478 reserved ``s_trap 0x06`` Reserved. 10479 reserved ``s_trap 0x07`` Reserved. 10480 reserved ``s_trap 0x08`` Reserved. 10481 reserved ``s_trap 0xfe`` Reserved. 10482 reserved ``s_trap 0xff`` Reserved. 10483 =================== =============== =============== ======================================= 10484 10485.. 10486 10487 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10488 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10489 10490 =================== =============== =============== ======================================= 10491 Usage Code Sequence Trap Handler Description 10492 Inputs 10493 =================== =============== =============== ======================================= 10494 reserved ``s_trap 0x00`` Reserved by hardware. 10495 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10496 breakpoints. Causes wave to be halted 10497 with the PC at the trap instruction. 10498 The debugger is responsible to resume 10499 the wave, including the instruction 10500 that the breakpoint overwrote. 10501 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10502 ``queue_ptr`` the trap instruction. The associated 10503 queue is signalled to put it into the 10504 error state. When the queue is put in 10505 the error state, the waves executing 10506 dispatches on the queue will be 10507 terminated. 10508 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10509 as a no-operation. The trap handler 10510 is entered and immediately returns to 10511 continue execution of the wavefront. 10512 - If the debugger is enabled, causes 10513 the debug trap to be reported by the 10514 debugger and the wavefront is put in 10515 the halt state with the PC at the 10516 instruction. The debugger must 10517 increment the PC and resume the wave. 10518 reserved ``s_trap 0x04`` Reserved. 10519 reserved ``s_trap 0x05`` Reserved. 10520 reserved ``s_trap 0x06`` Reserved. 10521 reserved ``s_trap 0x07`` Reserved. 10522 reserved ``s_trap 0x08`` Reserved. 10523 reserved ``s_trap 0xfe`` Reserved. 10524 reserved ``s_trap 0xff`` Reserved. 10525 =================== =============== =============== ======================================= 10526 10527.. 10528 10529 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10530 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10531 10532 =================== =============== ================ ================= ======================================= 10533 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10534 =================== =============== ================ ================= ======================================= 10535 reserved ``s_trap 0x00`` Reserved by hardware. 10536 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10537 breakpoints. Causes wave to be halted 10538 with the PC at the trap instruction. 10539 The debugger is responsible to resume 10540 the wave, including the instruction 10541 that the breakpoint overwrote. 10542 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10543 ``queue_ptr`` the trap instruction. The associated 10544 queue is signalled to put it into the 10545 error state. When the queue is put in 10546 the error state, the waves executing 10547 dispatches on the queue will be 10548 terminated. 10549 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10550 as a no-operation. The trap handler 10551 is entered and immediately returns to 10552 continue execution of the wavefront. 10553 - If the debugger is enabled, causes 10554 the debug trap to be reported by the 10555 debugger and the wavefront is put in 10556 the halt state with the PC at the 10557 instruction. The debugger must 10558 increment the PC and resume the wave. 10559 reserved ``s_trap 0x04`` Reserved. 10560 reserved ``s_trap 0x05`` Reserved. 10561 reserved ``s_trap 0x06`` Reserved. 10562 reserved ``s_trap 0x07`` Reserved. 10563 reserved ``s_trap 0x08`` Reserved. 10564 reserved ``s_trap 0xfe`` Reserved. 10565 reserved ``s_trap 0xff`` Reserved. 10566 =================== =============== ================ ================= ======================================= 10567 10568.. _amdgpu-amdhsa-function-call-convention: 10569 10570Call Convention 10571~~~~~~~~~~~~~~~ 10572 10573.. note:: 10574 10575 This section is currently incomplete and has inaccuracies. It is WIP that will 10576 be updated as information is determined. 10577 10578See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10579addresses. Unswizzled addresses are normal linear addresses. 10580 10581.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10582 10583Kernel Functions 10584++++++++++++++++ 10585 10586This section describes the call convention ABI for the outer kernel function. 10587 10588See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10589convention. 10590 10591The following is not part of the AMDGPU kernel calling convention but describes 10592how the AMDGPU implements function calls: 10593 105941. Clang decides the kernarg layout to match the *HSA Programmer's Language 10595 Reference* [HSA]_. 10596 10597 - All structs are passed directly. 10598 - Lambda values are passed *TBA*. 10599 10600 .. TODO:: 10601 10602 - Does this really follow HSA rules? Or are structs >16 bytes passed 10603 by-value struct? 10604 - What is ABI for lambda values? 10605 106064. The kernel performs certain setup in its prolog, as described in 10607 :ref:`amdgpu-amdhsa-kernel-prolog`. 10608 10609.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10610 10611Non-Kernel Functions 10612++++++++++++++++++++ 10613 10614This section describes the call convention ABI for functions other than the 10615outer kernel function. 10616 10617If a kernel has function calls then scratch is always allocated and used for 10618the call stack which grows from low address to high address using the swizzled 10619scratch address space. 10620 10621On entry to a function: 10622 106231. SGPR0-3 contain a V# with the following properties (see 10624 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10625 10626 * Base address pointing to the beginning of the wavefront scratch backing 10627 memory. 10628 * Swizzled with dword element size and stride of wavefront size elements. 10629 106302. The FLAT_SCRATCH register pair is setup. See 10631 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 106323. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10633 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 106344. The EXEC register is set to the lanes active on entry to the function. 106355. MODE register: *TBD* 106366. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10637 below. 106387. SGPR30-31 return address (RA). The code address that the function must 10639 return to when it completes. The value is undefined if the function is *no 10640 return*. 106418. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10642 offset relative to the beginning of the wavefront scratch backing memory. 10643 10644 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10645 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10646 manner. 10647 10648 The unswizzled SP value can be converted into the swizzled SP value by: 10649 10650 | swizzled SP = unswizzled SP / wavefront size 10651 10652 This may be used to obtain the private address space address of stack 10653 objects and to convert this address to a flat address by adding the flat 10654 scratch aperture base address. 10655 10656 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10657 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10658 10659 .. note:: 10660 10661 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10662 OpenCL language which has the largest base type defined as 16 bytes. 10663 10664 On entry, the swizzled SP value is the address of the first function 10665 argument passed on the stack. Other stack passed arguments are positive 10666 offsets from the entry swizzled SP value. 10667 10668 The function may use positive offsets beyond the last stack passed argument 10669 for stack allocated local variables and register spill slots. If necessary, 10670 the function may align these to greater alignment than 16 bytes. After these 10671 the function may dynamically allocate space for such things as runtime sized 10672 ``alloca`` local allocations. 10673 10674 If the function calls another function, it will place any stack allocated 10675 arguments after the last local allocation and adjust SGPR32 to the address 10676 after the last local allocation. 10677 106789. All other registers are unspecified. 1067910. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10680 to the function. 10681 10682On exit from a function: 10683 106841. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10685 described below. Any registers used are considered clobbered registers. 106862. The following registers are preserved and have the same value as on entry: 10687 10688 * FLAT_SCRATCH 10689 * EXEC 10690 * GFX6-GFX8: M0 10691 * All SGPR registers except the clobbered registers of SGPR4-31. 10692 * VGPR40-47 10693 * VGPR56-63 10694 * VGPR72-79 10695 * VGPR88-95 10696 * VGPR104-111 10697 * VGPR120-127 10698 * VGPR136-143 10699 * VGPR152-159 10700 * VGPR168-175 10701 * VGPR184-191 10702 * VGPR200-207 10703 * VGPR216-223 10704 * VGPR232-239 10705 * VGPR248-255 10706 10707 .. note:: 10708 10709 Except the argument registers, the VGPRs clobbered and the preserved 10710 registers are intermixed at regular intervals in order to keep a 10711 similar ratio independent of the number of allocated VGPRs. 10712 10713 * Lanes of all VGPRs that are inactive at the call site. 10714 10715 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10716 optimization may mark some of clobbered SGPR and VGPR registers as 10717 preserved if it can be determined that the called function does not change 10718 their value. 10719 107202. The PC is set to the RA provided on entry. 107213. MODE register: *TBD*. 107224. All other registers are clobbered. 107235. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10724 function is available to the caller. 10725 10726.. TODO:: 10727 10728 - On gfx908 are all ACC registers clobbered? 10729 10730 - How are function results returned? The address of structured types is passed 10731 by reference, but what about other types? 10732 10733The function input arguments are made up of the formal arguments explicitly 10734declared by the source language function plus the implicit input arguments used 10735by the implementation. 10736 10737The source language input arguments are: 10738 107391. Any source language implicit ``this`` or ``self`` argument comes first as a 10740 pointer type. 107412. Followed by the function formal arguments in left to right source order. 10742 10743The source language result arguments are: 10744 107451. The function result argument. 10746 10747The source language input or result struct type arguments that are less than or 10748equal to 16 bytes, are decomposed recursively into their base type fields, and 10749each field is passed as if a separate argument. For input arguments, if the 10750called function requires the struct to be in memory, for example because its 10751address is taken, then the function body is responsible for allocating a stack 10752location and copying the field arguments into it. Clang terms this *direct 10753struct*. 10754 10755The source language input struct type arguments that are greater than 16 bytes, 10756are passed by reference. The caller is responsible for allocating a stack 10757location to make a copy of the struct value and pass the address as the input 10758argument. The called function is responsible to perform the dereference when 10759accessing the input argument. Clang terms this *by-value struct*. 10760 10761A source language result struct type argument that is greater than 16 bytes, is 10762returned by reference. The caller is responsible for allocating a stack location 10763to hold the result value and passes the address as the last input argument 10764(before the implicit input arguments). In this case there are no result 10765arguments. The called function is responsible to perform the dereference when 10766storing the result value. Clang terms this *structured return (sret)*. 10767 10768*TODO: correct the ``sret`` definition.* 10769 10770.. TODO:: 10771 10772 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10773 pass as non-decomposed struct as stack argument? Or something else? Is the 10774 memory location in the caller stack frame, or a stack memory argument and so 10775 no address is passed as the caller can directly write to the argument stack 10776 location? But then the stack location is still live after return. If an 10777 argument stack location is it the first stack argument or the last one? 10778 10779Lambda argument types are treated as struct types with an implementation defined 10780set of fields. 10781 10782.. TODO:: 10783 10784 Need to specify the ABI for lambda types for AMDGPU. 10785 10786For AMDGPU backend all source language arguments (including the decomposed 10787struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10788they are passed in SGPRs. 10789 10790The AMDGPU backend walks the function call graph from the leaves to determine 10791which implicit input arguments are used, propagating to each caller of the 10792function. The used implicit arguments are appended to the function arguments 10793after the source language arguments in the following order: 10794 10795.. TODO:: 10796 10797 Is recursion or external functions supported? 10798 107991. Work-Item ID (1 VGPR) 10800 10801 The X, Y and Z work-item ID are packed into a single VGRP with the following 10802 layout. Only fields actually used by the function are set. The other bits 10803 are undefined. 10804 10805 The values come from the initial kernel execution state. See 10806 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10807 10808 .. table:: Work-item implicit argument layout 10809 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10810 10811 ======= ======= ============== 10812 Bits Size Field Name 10813 ======= ======= ============== 10814 9:0 10 bits X Work-Item ID 10815 19:10 10 bits Y Work-Item ID 10816 29:20 10 bits Z Work-Item ID 10817 31:30 2 bits Unused 10818 ======= ======= ============== 10819 108202. Dispatch Ptr (2 SGPRs) 10821 10822 The value comes from the initial kernel execution state. See 10823 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10824 108253. Queue Ptr (2 SGPRs) 10826 10827 The value comes from the initial kernel execution state. See 10828 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10829 108304. Kernarg Segment Ptr (2 SGPRs) 10831 10832 The value comes from the initial kernel execution state. See 10833 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10834 108355. Dispatch id (2 SGPRs) 10836 10837 The value comes from the initial kernel execution state. See 10838 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10839 108406. Work-Group ID X (1 SGPR) 10841 10842 The value comes from the initial kernel execution state. See 10843 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10844 108457. Work-Group ID Y (1 SGPR) 10846 10847 The value comes from the initial kernel execution state. See 10848 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10849 108508. Work-Group ID Z (1 SGPR) 10851 10852 The value comes from the initial kernel execution state. See 10853 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10854 108559. Implicit Argument Ptr (2 SGPRs) 10856 10857 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10858 global address space pointer to the first kernarg implicit argument. 10859 10860The input and result arguments are assigned in order in the following manner: 10861 10862.. note:: 10863 10864 There are likely some errors and omissions in the following description that 10865 need correction. 10866 10867 .. TODO:: 10868 10869 Check the Clang source code to decipher how function arguments and return 10870 results are handled. Also see the AMDGPU specific values used. 10871 10872* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 10873 VGPR31. 10874 10875 If there are more arguments than will fit in these registers, the remaining 10876 arguments are allocated on the stack in order on naturally aligned 10877 addresses. 10878 10879 .. TODO:: 10880 10881 How are overly aligned structures allocated on the stack? 10882 10883* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 10884 SGPR29. 10885 10886 If there are more arguments than will fit in these registers, the remaining 10887 arguments are allocated on the stack in order on naturally aligned 10888 addresses. 10889 10890Note that decomposed struct type arguments may have some fields passed in 10891registers and some in memory. 10892 10893.. TODO:: 10894 10895 So, a struct which can pass some fields as decomposed register arguments, will 10896 pass the rest as decomposed stack elements? But an argument that will not start 10897 in registers will not be decomposed and will be passed as a non-decomposed 10898 stack value? 10899 10900The following is not part of the AMDGPU function calling convention but 10901describes how the AMDGPU implements function calls: 10902 109031. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 10904 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 10905 are used, or for the reasons defined in ``SIFrameLowering``. 109062. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 10907 to access the incoming stack arguments in the function. The BP is needed 10908 only when the function requires the runtime stack alignment. 10909 109103. Allocating SGPR arguments on the stack are not supported. 10911 109124. No CFI is currently generated. See 10913 :ref:`amdgpu-dwarf-call-frame-information`. 10914 10915 .. note:: 10916 10917 CFI will be generated that defines the CFA as the unswizzled address 10918 relative to the wave scratch base in the unswizzled private address space 10919 of the lowest address stack allocated local variable. 10920 10921 ``DW_AT_frame_base`` will be defined as the swizzled address in the 10922 swizzled private address space by dividing the CFA by the wavefront size 10923 (since CFA is always at least dword aligned which matches the scratch 10924 swizzle element size). 10925 10926 If no dynamic stack alignment was performed, the stack allocated arguments 10927 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 10928 local variables and register spill slots are accessed as positive offsets 10929 relative to ``DW_AT_frame_base``. 10930 109315. Function argument passing is implemented by copying the input physical 10932 registers to virtual registers on entry. The register allocator can spill if 10933 necessary. These are copied back to physical registers at call sites. The 10934 net effect is that each function call can have these values in entirely 10935 distinct locations. The IPRA can help avoid shuffling argument registers. 109366. Call sites are implemented by setting up the arguments at positive offsets 10937 from SP. Then SP is incremented to account for the known frame size before 10938 the call and decremented after the call. 10939 10940 .. note:: 10941 10942 The CFI will reflect the changed calculation needed to compute the CFA 10943 from SP. 10944 109457. 4 byte spill slots are used in the stack frame. One slot is allocated for an 10946 emergency spill slot. Buffer instructions are used for stack accesses and 10947 not the ``flat_scratch`` instruction. 10948 10949 .. TODO:: 10950 10951 Explain when the emergency spill slot is used. 10952 10953.. TODO:: 10954 10955 Possible broken issues: 10956 10957 - Stack arguments must be aligned to required alignment. 10958 - Stack is aligned to max(16, max formal argument alignment) 10959 - Direct argument < 64 bits should check register budget. 10960 - Register budget calculation should respect ``inreg`` for SGPR. 10961 - SGPR overflow is not handled. 10962 - struct with 1 member unpeeling is not checking size of member. 10963 - ``sret`` is after ``this`` pointer. 10964 - Caller is not implementing stack realignment: need an extra pointer. 10965 - Should say AMDGPU passes FP rather than SP. 10966 - Should CFI define CFA as address of locals or arguments. Difference is 10967 apparent when have implemented dynamic alignment. 10968 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 10969 highest address of stack frame and use negative offset for locals. Would 10970 allow SP to be the same as FP and could support signal-handler-like as now 10971 have a real SP for the top of the stack. 10972 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 10973 arguments? 10974 10975AMDPAL 10976------ 10977 10978This section provides code conventions used when the target triple OS is 10979``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 10980from the application/runtime to each invocation of a hardware shader. These 10981parameters include both generic, application-controlled parameters called 10982*user data* as well as system-generated parameters that are a product of the 10983draw or dispatch execution. 10984 10985User Data 10986~~~~~~~~~ 10987 10988Each hardware stage has a set of 32-bit *user data registers* which can be 10989written from a command buffer and then loaded into SGPRs when waves are launched 10990via a subsequent dispatch or draw operation. This is the way most arguments are 10991passed from the application/runtime to a hardware shader. 10992 10993Compute User Data 10994~~~~~~~~~~~~~~~~~ 10995 10996Compute shader user data mappings are simpler than graphics shaders and have a 10997fixed mapping. 10998 10999Note that there are always 10 available *user data entries* in registers - 11000entries beyond that limit must be fetched from memory (via the spill table 11001pointer) by the shader. 11002 11003 .. table:: PAL Compute Shader User Data Registers 11004 :name: pal-compute-user-data-registers 11005 11006 ============= ================================ 11007 User Register Description 11008 ============= ================================ 11009 0 Global Internal Table (32-bit pointer) 11010 1 Per-Shader Internal Table (32-bit pointer) 11011 2 - 11 Application-Controlled User Data (10 32-bit values) 11012 12 Spill Table (32-bit pointer) 11013 13 - 14 Thread Group Count (64-bit pointer) 11014 15 GDS Range 11015 ============= ================================ 11016 11017Graphics User Data 11018~~~~~~~~~~~~~~~~~~ 11019 11020Graphics pipelines support a much more flexible user data mapping: 11021 11022 .. table:: PAL Graphics Shader User Data Registers 11023 :name: pal-graphics-user-data-registers 11024 11025 ============= ================================ 11026 User Register Description 11027 ============= ================================ 11028 0 Global Internal Table (32-bit pointer) 11029 + Per-Shader Internal Table (32-bit pointer) 11030 + 1-15 Application Controlled User Data 11031 (1-15 Contiguous 32-bit Values in Registers) 11032 + Spill Table (32-bit pointer) 11033 + Draw Index (First Stage Only) 11034 + Vertex Offset (First Stage Only) 11035 + Instance Offset (First Stage Only) 11036 ============= ================================ 11037 11038 The placement of the global internal table remains fixed in the first *user 11039 data SGPR register*. Otherwise all parameters are optional, and can be mapped 11040 to any desired *user data SGPR register*, with the following restrictions: 11041 11042 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 11043 active hardware stage in a graphics pipeline (i.e. where the API vertex 11044 shader runs). 11045 11046 * Application-controlled user data must be mapped into a contiguous range of 11047 user data registers. 11048 11049 * The application-controlled user data range supports compaction remapping, so 11050 only *entries* that are actually consumed by the shader must be assigned to 11051 corresponding *registers*. Note that in order to support an efficient runtime 11052 implementation, the remapping must pack *registers* in the same order as 11053 *entries*, with unused *entries* removed. 11054 11055.. _pal_global_internal_table: 11056 11057Global Internal Table 11058~~~~~~~~~~~~~~~~~~~~~ 11059 11060The global internal table is a table of *shader resource descriptors* (SRDs) 11061that define how certain engine-wide, runtime-managed resources should be 11062accessed from a shader. The majority of these resources have HW-defined formats, 11063and it is up to the compiler to write/read data as required by the target 11064hardware. 11065 11066The following table illustrates the required format: 11067 11068 .. table:: PAL Global Internal Table 11069 :name: pal-git-table 11070 11071 ============= ================================ 11072 Offset Description 11073 ============= ================================ 11074 0-3 Graphics Scratch SRD 11075 4-7 Compute Scratch SRD 11076 8-11 ES/GS Ring Output SRD 11077 12-15 ES/GS Ring Input SRD 11078 16-19 GS/VS Ring Output #0 11079 20-23 GS/VS Ring Output #1 11080 24-27 GS/VS Ring Output #2 11081 28-31 GS/VS Ring Output #3 11082 32-35 GS/VS Ring Input SRD 11083 36-39 Tessellation Factor Buffer SRD 11084 40-43 Off-Chip LDS Buffer SRD 11085 44-47 Off-Chip Param Cache Buffer SRD 11086 48-51 Sample Position Buffer SRD 11087 52 vaRange::ShadowDescriptorTable High Bits 11088 ============= ================================ 11089 11090 The pointer to the global internal table passed to the shader as user data 11091 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 11092 the top 32 bits of the pipeline, so the shader may use the program 11093 counter's top 32 bits. 11094 11095.. _pal_call-convention: 11096 11097Call Convention 11098~~~~~~~~~~~~~~~ 11099 11100For graphics use cases, the calling convention is `amdgpu_gfx`. 11101 11102.. note:: 11103 11104 `amdgpu_gfx` Function calls are currently in development and are 11105 subject to major changes. 11106 11107This calling convention shares most properties with calling non-kernel 11108functions (see 11109:ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions`). 11110Differences are: 11111 11112 - Currently there are none, differences will be listed here 11113 11114Unspecified OS 11115-------------- 11116 11117This section provides code conventions used when the target triple OS is 11118empty (see :ref:`amdgpu-target-triples`). 11119 11120Trap Handler ABI 11121~~~~~~~~~~~~~~~~ 11122 11123For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11124not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11125instructions are handled as follows: 11126 11127 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11128 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11129 11130 =============== =============== =========================================== 11131 Usage Code Sequence Description 11132 =============== =============== =========================================== 11133 llvm.trap s_endpgm Causes wavefront to be terminated. 11134 llvm.debugtrap *none* Compiler warning given that there is no 11135 trap handler installed. 11136 =============== =============== =========================================== 11137 11138Source Languages 11139================ 11140 11141.. _amdgpu-opencl: 11142 11143OpenCL 11144------ 11145 11146When the language is OpenCL the following differences occur: 11147 111481. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 111492. The AMDGPU backend appends additional arguments to the kernel's explicit 11150 arguments for the AMDHSA OS (see 11151 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 111523. Additional metadata is generated 11153 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11154 11155 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11156 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11157 11158 ======== ==== ========= =========================================== 11159 Position Byte Byte Description 11160 Size Alignment 11161 ======== ==== ========= =========================================== 11162 1 8 8 OpenCL Global Offset X 11163 2 8 8 OpenCL Global Offset Y 11164 3 8 8 OpenCL Global Offset Z 11165 4 8 8 OpenCL address of printf buffer 11166 5 8 8 OpenCL address of virtual queue used by 11167 enqueue_kernel. 11168 6 8 8 OpenCL address of AqlWrap struct used by 11169 enqueue_kernel. 11170 7 8 8 Pointer argument used for Multi-gird 11171 synchronization. 11172 ======== ==== ========= =========================================== 11173 11174.. _amdgpu-hcc: 11175 11176HCC 11177--- 11178 11179When the language is HCC the following differences occur: 11180 111811. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11182 11183.. _amdgpu-assembler: 11184 11185Assembler 11186--------- 11187 11188AMDGPU backend has LLVM-MC based assembler which is currently in development. 11189It supports AMDGCN GFX6-GFX10. 11190 11191This section describes general syntax for instructions and operands. 11192 11193Instructions 11194~~~~~~~~~~~~ 11195 11196An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11197 11198 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11199 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11200 11201:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11202:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11203 11204The order of operands and modifiers is fixed. 11205Most modifiers are optional and may be omitted. 11206 11207Links to detailed instruction syntax description may be found in the following 11208table. Note that features under development are not included 11209in this description. 11210 11211 =================================== ======================================= 11212 Core ISA ISA Extensions 11213 =================================== ======================================= 11214 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11215 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11216 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11217 11218 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11219 11220 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11221 11222 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11223 11224 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11225 11226 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11227 11228 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11229 11230 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11231 =================================== ======================================= 11232 11233For more information about instructions, their semantics and supported 11234combinations of operands, refer to one of instruction set architecture manuals 11235[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_, 11236[AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11237 11238Operands 11239~~~~~~~~ 11240 11241Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11242 11243Modifiers 11244~~~~~~~~~ 11245 11246Detailed description of modifiers may be found 11247:doc:`here<AMDGPUModifierSyntax>`. 11248 11249Instruction Examples 11250~~~~~~~~~~~~~~~~~~~~ 11251 11252DS 11253++ 11254 11255.. code-block:: nasm 11256 11257 ds_add_u32 v2, v4 offset:16 11258 ds_write_src2_b64 v2 offset0:4 offset1:8 11259 ds_cmpst_f32 v2, v4, v6 11260 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11261 11262For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11263Manual. 11264 11265FLAT 11266++++ 11267 11268.. code-block:: nasm 11269 11270 flat_load_dword v1, v[3:4] 11271 flat_store_dwordx3 v[3:4], v[5:7] 11272 flat_atomic_swap v1, v[3:4], v5 glc 11273 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11274 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11275 11276For full list of supported instructions, refer to "FLAT instructions" in ISA 11277Manual. 11278 11279MUBUF 11280+++++ 11281 11282.. code-block:: nasm 11283 11284 buffer_load_dword v1, off, s[4:7], s1 11285 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11286 buffer_store_format_xy v[1:2], off, s[4:7], s1 11287 buffer_wbinvl1 11288 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11289 11290For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11291Manual. 11292 11293SMRD/SMEM 11294+++++++++ 11295 11296.. code-block:: nasm 11297 11298 s_load_dword s1, s[2:3], 0xfc 11299 s_load_dwordx8 s[8:15], s[2:3], s4 11300 s_load_dwordx16 s[88:103], s[2:3], s4 11301 s_dcache_inv_vol 11302 s_memtime s[4:5] 11303 11304For full list of supported instructions, refer to "Scalar Memory Operations" in 11305ISA Manual. 11306 11307SOP1 11308++++ 11309 11310.. code-block:: nasm 11311 11312 s_mov_b32 s1, s2 11313 s_mov_b64 s[0:1], 0x80000000 11314 s_cmov_b32 s1, 200 11315 s_wqm_b64 s[2:3], s[4:5] 11316 s_bcnt0_i32_b64 s1, s[2:3] 11317 s_swappc_b64 s[2:3], s[4:5] 11318 s_cbranch_join s[4:5] 11319 11320For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11321Manual. 11322 11323SOP2 11324++++ 11325 11326.. code-block:: nasm 11327 11328 s_add_u32 s1, s2, s3 11329 s_and_b64 s[2:3], s[4:5], s[6:7] 11330 s_cselect_b32 s1, s2, s3 11331 s_andn2_b32 s2, s4, s6 11332 s_lshr_b64 s[2:3], s[4:5], s6 11333 s_ashr_i32 s2, s4, s6 11334 s_bfm_b64 s[2:3], s4, s6 11335 s_bfe_i64 s[2:3], s[4:5], s6 11336 s_cbranch_g_fork s[4:5], s[6:7] 11337 11338For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11339Manual. 11340 11341SOPC 11342++++ 11343 11344.. code-block:: nasm 11345 11346 s_cmp_eq_i32 s1, s2 11347 s_bitcmp1_b32 s1, s2 11348 s_bitcmp0_b64 s[2:3], s4 11349 s_setvskip s3, s5 11350 11351For full list of supported instructions, refer to "SOPC Instructions" in ISA 11352Manual. 11353 11354SOPP 11355++++ 11356 11357.. code-block:: nasm 11358 11359 s_barrier 11360 s_nop 2 11361 s_endpgm 11362 s_waitcnt 0 ; Wait for all counters to be 0 11363 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11364 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11365 s_sethalt 9 11366 s_sleep 10 11367 s_sendmsg 0x1 11368 s_sendmsg sendmsg(MSG_INTERRUPT) 11369 s_trap 1 11370 11371For full list of supported instructions, refer to "SOPP Instructions" in ISA 11372Manual. 11373 11374Unless otherwise mentioned, little verification is performed on the operands 11375of SOPP Instructions, so it is up to the programmer to be familiar with the 11376range or acceptable values. 11377 11378VALU 11379++++ 11380 11381For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11382the assembler will automatically use optimal encoding based on its operands. To 11383force specific encoding, one can add a suffix to the opcode of the instruction: 11384 11385* _e32 for 32-bit VOP1/VOP2/VOPC 11386* _e64 for 64-bit VOP3 11387* _dpp for VOP_DPP 11388* _sdwa for VOP_SDWA 11389 11390VOP1/VOP2/VOP3/VOPC examples: 11391 11392.. code-block:: nasm 11393 11394 v_mov_b32 v1, v2 11395 v_mov_b32_e32 v1, v2 11396 v_nop 11397 v_cvt_f64_i32_e32 v[1:2], v2 11398 v_floor_f32_e32 v1, v2 11399 v_bfrev_b32_e32 v1, v2 11400 v_add_f32_e32 v1, v2, v3 11401 v_mul_i32_i24_e64 v1, v2, 3 11402 v_mul_i32_i24_e32 v1, -3, v3 11403 v_mul_i32_i24_e32 v1, -100, v3 11404 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11405 v_max_f16_e32 v1, v2, v3 11406 11407VOP_DPP examples: 11408 11409.. code-block:: nasm 11410 11411 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11412 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11413 v_mov_b32 v0, v0 wave_shl:1 11414 v_mov_b32 v0, v0 row_mirror 11415 v_mov_b32 v0, v0 row_bcast:31 11416 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11417 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11418 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11419 11420VOP_SDWA examples: 11421 11422.. code-block:: nasm 11423 11424 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11425 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11426 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11427 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11428 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11429 11430For full list of supported instructions, refer to "Vector ALU instructions". 11431 11432.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11433 11434Code Object V2 Predefined Symbols 11435~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11436 11437.. warning:: 11438 Code object V2 is not the default code object version emitted by 11439 this version of LLVM. 11440 11441The AMDGPU assembler defines and updates some symbols automatically. These 11442symbols do not affect code generation. 11443 11444.option.machine_version_major 11445+++++++++++++++++++++++++++++ 11446 11447Set to the GFX major generation number of the target being assembled for. For 11448example, when assembling for a "GFX9" target this will be set to the integer 11449value "9". The possible GFX major generation numbers are presented in 11450:ref:`amdgpu-processors`. 11451 11452.option.machine_version_minor 11453+++++++++++++++++++++++++++++ 11454 11455Set to the GFX minor generation number of the target being assembled for. For 11456example, when assembling for a "GFX810" target this will be set to the integer 11457value "1". The possible GFX minor generation numbers are presented in 11458:ref:`amdgpu-processors`. 11459 11460.option.machine_version_stepping 11461++++++++++++++++++++++++++++++++ 11462 11463Set to the GFX stepping generation number of the target being assembled for. 11464For example, when assembling for a "GFX704" target this will be set to the 11465integer value "4". The possible GFX stepping generation numbers are presented 11466in :ref:`amdgpu-processors`. 11467 11468.kernel.vgpr_count 11469++++++++++++++++++ 11470 11471Set to zero each time a 11472:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11473encountered. At each instruction, if the current value of this symbol is less 11474than or equal to the maximum VGPR number explicitly referenced within that 11475instruction then the symbol value is updated to equal that VGPR number plus 11476one. 11477 11478.kernel.sgpr_count 11479++++++++++++++++++ 11480 11481Set to zero each time a 11482:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11483encountered. At each instruction, if the current value of this symbol is less 11484than or equal to the maximum VGPR number explicitly referenced within that 11485instruction then the symbol value is updated to equal that SGPR number plus 11486one. 11487 11488.. _amdgpu-amdhsa-assembler-directives-v2: 11489 11490Code Object V2 Directives 11491~~~~~~~~~~~~~~~~~~~~~~~~~ 11492 11493.. warning:: 11494 Code object V2 is not the default code object version emitted by 11495 this version of LLVM. 11496 11497AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11498one can specify them with assembler directives. 11499 11500.hsa_code_object_version major, minor 11501+++++++++++++++++++++++++++++++++++++ 11502 11503*major* and *minor* are integers that specify the version of the HSA code 11504object that will be generated by the assembler. 11505 11506.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11507+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11508 11509 11510*major*, *minor*, and *stepping* are all integers that describe the instruction 11511set architecture (ISA) version of the assembly program. 11512 11513*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11514"AMD" and *arch* should always be equal to "AMDGPU". 11515 11516By default, the assembler will derive the ISA version, *vendor*, and *arch* 11517from the value of the -mcpu option that is passed to the assembler. 11518 11519.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11520 11521.amdgpu_hsa_kernel (name) 11522+++++++++++++++++++++++++ 11523 11524This directives specifies that the symbol with given name is a kernel entry 11525point (label) and the object should contain corresponding symbol of type 11526STT_AMDGPU_HSA_KERNEL. 11527 11528.amd_kernel_code_t 11529++++++++++++++++++ 11530 11531This directive marks the beginning of a list of key / value pairs that are used 11532to specify the amd_kernel_code_t object that will be emitted by the assembler. 11533The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11534amd_kernel_code_t values that are unspecified a default value will be used. The 11535default value for all keys is 0, with the following exceptions: 11536 11537- *amd_code_version_major* defaults to 1. 11538- *amd_kernel_code_version_minor* defaults to 2. 11539- *amd_machine_kind* defaults to 1. 11540- *amd_machine_version_major*, *machine_version_minor*, and 11541 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11542 that is passed to the assembler. 11543- *kernel_code_entry_byte_offset* defaults to 256. 11544- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11545 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11546 Note that wavefront size is specified as a power of two, so a value of **n** 11547 means a size of 2^ **n**. 11548- *call_convention* defaults to -1. 11549- *kernarg_segment_alignment*, *group_segment_alignment*, and 11550 *private_segment_alignment* default to 4. Note that alignments are specified 11551 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11552- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11553 GFX90A onwards. 11554- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11555 GFX10 onwards. 11556- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11557 11558The *.amd_kernel_code_t* directive must be placed immediately after the 11559function label and before any instructions. 11560 11561For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11562comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11563 11564.. _amdgpu-amdhsa-assembler-example-v2: 11565 11566Code Object V2 Example Source Code 11567~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11568 11569.. warning:: 11570 Code Object V2 is not the default code object version emitted by 11571 this version of LLVM. 11572 11573Here is an example of a minimal assembly source file, defining one HSA kernel: 11574 11575.. code:: 11576 :number-lines: 11577 11578 .hsa_code_object_version 1,0 11579 .hsa_code_object_isa 11580 11581 .hsatext 11582 .globl hello_world 11583 .p2align 8 11584 .amdgpu_hsa_kernel hello_world 11585 11586 hello_world: 11587 11588 .amd_kernel_code_t 11589 enable_sgpr_kernarg_segment_ptr = 1 11590 is_ptr64 = 1 11591 compute_pgm_rsrc1_vgprs = 0 11592 compute_pgm_rsrc1_sgprs = 0 11593 compute_pgm_rsrc2_user_sgpr = 2 11594 compute_pgm_rsrc1_wgp_mode = 0 11595 compute_pgm_rsrc1_mem_ordered = 0 11596 compute_pgm_rsrc1_fwd_progress = 1 11597 .end_amd_kernel_code_t 11598 11599 s_load_dwordx2 s[0:1], s[0:1] 0x0 11600 v_mov_b32 v0, 3.14159 11601 s_waitcnt lgkmcnt(0) 11602 v_mov_b32 v1, s0 11603 v_mov_b32 v2, s1 11604 flat_store_dword v[1:2], v0 11605 s_endpgm 11606 .Lfunc_end0: 11607 .size hello_world, .Lfunc_end0-hello_world 11608 11609.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11610 11611Code Object V3 to V4 Predefined Symbols 11612~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11613 11614The AMDGPU assembler defines and updates some symbols automatically. These 11615symbols do not affect code generation. 11616 11617.amdgcn.gfx_generation_number 11618+++++++++++++++++++++++++++++ 11619 11620Set to the GFX major generation number of the target being assembled for. For 11621example, when assembling for a "GFX9" target this will be set to the integer 11622value "9". The possible GFX major generation numbers are presented in 11623:ref:`amdgpu-processors`. 11624 11625.amdgcn.gfx_generation_minor 11626++++++++++++++++++++++++++++ 11627 11628Set to the GFX minor generation number of the target being assembled for. For 11629example, when assembling for a "GFX810" target this will be set to the integer 11630value "1". The possible GFX minor generation numbers are presented in 11631:ref:`amdgpu-processors`. 11632 11633.amdgcn.gfx_generation_stepping 11634+++++++++++++++++++++++++++++++ 11635 11636Set to the GFX stepping generation number of the target being assembled for. 11637For example, when assembling for a "GFX704" target this will be set to the 11638integer value "4". The possible GFX stepping generation numbers are presented 11639in :ref:`amdgpu-processors`. 11640 11641.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 11642 11643.amdgcn.next_free_vgpr 11644++++++++++++++++++++++ 11645 11646Set to zero before assembly begins. At each instruction, if the current value 11647of this symbol is less than or equal to the maximum VGPR number explicitly 11648referenced within that instruction then the symbol value is updated to equal 11649that VGPR number plus one. 11650 11651May be used to set the `.amdhsa_next_free_vgpr` directive in 11652:ref:`amdhsa-kernel-directives-table`. 11653 11654May be set at any time, e.g. manually set to zero at the start of each kernel. 11655 11656.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 11657 11658.amdgcn.next_free_sgpr 11659++++++++++++++++++++++ 11660 11661Set to zero before assembly begins. At each instruction, if the current value 11662of this symbol is less than or equal the maximum SGPR number explicitly 11663referenced within that instruction then the symbol value is updated to equal 11664that SGPR number plus one. 11665 11666May be used to set the `.amdhsa_next_free_spgr` directive in 11667:ref:`amdhsa-kernel-directives-table`. 11668 11669May be set at any time, e.g. manually set to zero at the start of each kernel. 11670 11671.. _amdgpu-amdhsa-assembler-directives-v3-v4: 11672 11673Code Object V3 to V4 Directives 11674~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11675 11676Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 11677architecture processors, and are not OS-specific. Directives which begin with 11678``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 11679``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 11680:ref:`amdgpu-processors`. 11681 11682.. _amdgpu-assembler-directive-amdgcn-target: 11683 11684.amdgcn_target <target-triple> "-" <target-id> 11685++++++++++++++++++++++++++++++++++++++++++++++ 11686 11687Optional directive which declares the ``<target-triple>-<target-id>`` supported 11688by the containing assembler source file. Used by the assembler to validate 11689command-line options such as ``-triple``, ``-mcpu``, and 11690``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 11691:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 11692 11693.. note:: 11694 11695 The target ID syntax used for code object V2 to V3 for this directive differs 11696 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 11697 11698.amdhsa_kernel <name> 11699+++++++++++++++++++++ 11700 11701Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 11702``<name>.kd``, in the current location of the current section. Only valid when 11703the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 11704instruction to execute, and does not need to be previously defined. 11705 11706Marks the beginning of a list of directives used to generate the bytes of a 11707kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 11708Directives which may appear in this list are described in 11709:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 11710be valid for the target being assembled for, and cannot be repeated. Directives 11711support the range of values specified by the field they reference in 11712:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 11713assumed to have its default value, unless it is marked as "Required", in which 11714case it is an error to omit the directive. This list of directives is 11715terminated by an ``.end_amdhsa_kernel`` directive. 11716 11717 .. table:: AMDHSA Kernel Assembler Directives 11718 :name: amdhsa-kernel-directives-table 11719 11720 ======================================================== =================== ============ =================== 11721 Directive Default Supported On Description 11722 ======================================================== =================== ============ =================== 11723 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 11724 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11725 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 11726 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11727 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 11728 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11729 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 11730 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11731 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 11732 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11733 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 11734 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11735 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 11736 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11737 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 11738 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11739 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 11740 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11741 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 11742 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11743 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 11744 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11745 Specific 11746 (wavefrontsize64) 11747 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 11748 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11749 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 11750 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11751 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 11752 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11753 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 11754 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11755 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 11756 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11757 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 11758 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11759 Possible values are defined in 11760 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 11761 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 11762 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 11763 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11764 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 11765 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11766 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11767 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 11768 Used to calculate ACCUM_OFFSET in 11769 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11770 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 11771 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11772 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11773 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 11774 scratch memory. Used to calculate 11775 GRANULATED_WAVEFRONT_SGPR_COUNT in 11776 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11777 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 11778 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11779 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11780 (xnack) 11781 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 11782 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11783 Possible values are defined in 11784 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11785 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 11786 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11787 Possible values are defined in 11788 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11789 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 11790 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11791 Possible values are defined in 11792 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11793 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 11794 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11795 Possible values are defined in 11796 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11797 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 11798 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11799 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 11800 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11801 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 11802 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11803 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 11804 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11805 Specific 11806 (tgsplit) 11807 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 11808 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11809 Specific 11810 (cumode) 11811 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 11812 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11813 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 11814 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11815 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 11816 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11817 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 11818 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11819 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 11820 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11821 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 11822 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11823 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 11824 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11825 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 11826 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11827 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 11828 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11829 ======================================================== =================== ============ =================== 11830 11831.amdgpu_metadata 11832++++++++++++++++ 11833 11834Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 11835note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 11836 11837The contents must be in the [YAML]_ markup format, with the same structure and 11838semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 11839:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 11840 11841This directive is terminated by an ``.end_amdgpu_metadata`` directive. 11842 11843.. _amdgpu-amdhsa-assembler-example-v3-v4: 11844 11845Code Object V3 to V4 Example Source Code 11846~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11847 11848Here is an example of a minimal assembly source file, defining one HSA kernel: 11849 11850.. code:: 11851 :number-lines: 11852 11853 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 11854 11855 .text 11856 .globl hello_world 11857 .p2align 8 11858 .type hello_world,@function 11859 hello_world: 11860 s_load_dwordx2 s[0:1], s[0:1] 0x0 11861 v_mov_b32 v0, 3.14159 11862 s_waitcnt lgkmcnt(0) 11863 v_mov_b32 v1, s0 11864 v_mov_b32 v2, s1 11865 flat_store_dword v[1:2], v0 11866 s_endpgm 11867 .Lfunc_end0: 11868 .size hello_world, .Lfunc_end0-hello_world 11869 11870 .rodata 11871 .p2align 6 11872 .amdhsa_kernel hello_world 11873 .amdhsa_user_sgpr_kernarg_segment_ptr 1 11874 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11875 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11876 .end_amdhsa_kernel 11877 11878 .amdgpu_metadata 11879 --- 11880 amdhsa.version: 11881 - 1 11882 - 0 11883 amdhsa.kernels: 11884 - .name: hello_world 11885 .symbol: hello_world.kd 11886 .kernarg_segment_size: 48 11887 .group_segment_fixed_size: 0 11888 .private_segment_fixed_size: 0 11889 .kernarg_segment_align: 4 11890 .wavefront_size: 64 11891 .sgpr_count: 2 11892 .vgpr_count: 3 11893 .max_flat_workgroup_size: 256 11894 ... 11895 .end_amdgpu_metadata 11896 11897If an assembly source file contains multiple kernels and/or functions, the 11898:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 11899:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 11900the ``.set <symbol>, <expression>`` directive. For example, in the case of two 11901kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 11902to group the function with the kernel that calls it and reset the symbols 11903between the two connected components: 11904 11905.. code:: 11906 :number-lines: 11907 11908 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 11909 11910 // gpr tracking symbols are implicitly set to zero 11911 11912 .text 11913 .globl kern0 11914 .p2align 8 11915 .type kern0,@function 11916 kern0: 11917 // ... 11918 s_endpgm 11919 .Lkern0_end: 11920 .size kern0, .Lkern0_end-kern0 11921 11922 .rodata 11923 .p2align 6 11924 .amdhsa_kernel kern0 11925 // ... 11926 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11927 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11928 .end_amdhsa_kernel 11929 11930 // reset symbols to begin tracking usage in func1 and kern1 11931 .set .amdgcn.next_free_vgpr, 0 11932 .set .amdgcn.next_free_sgpr, 0 11933 11934 .text 11935 .hidden func1 11936 .global func1 11937 .p2align 2 11938 .type func1,@function 11939 func1: 11940 // ... 11941 s_setpc_b64 s[30:31] 11942 .Lfunc1_end: 11943 .size func1, .Lfunc1_end-func1 11944 11945 .globl kern1 11946 .p2align 8 11947 .type kern1,@function 11948 kern1: 11949 // ... 11950 s_getpc_b64 s[4:5] 11951 s_add_u32 s4, s4, func1@rel32@lo+4 11952 s_addc_u32 s5, s5, func1@rel32@lo+4 11953 s_swappc_b64 s[30:31], s[4:5] 11954 // ... 11955 s_endpgm 11956 .Lkern1_end: 11957 .size kern1, .Lkern1_end-kern1 11958 11959 .rodata 11960 .p2align 6 11961 .amdhsa_kernel kern1 11962 // ... 11963 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 11964 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 11965 .end_amdhsa_kernel 11966 11967These symbols cannot identify connected components in order to automatically 11968track the usage for each kernel. However, in some cases careful organization of 11969the kernels and functions in the source file means there is minimal additional 11970effort required to accurately calculate GPR usage. 11971 11972Additional Documentation 11973======================== 11974 11975.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 11976.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 11977.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 11978.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 11979.. [AMD-GCN-GFX10-RDNA1] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 11980.. [AMD-GCN-GFX10-RDNA2] `AMD "RDNA 2" Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 11981.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 11982.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 11983.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 11984.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 11985.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 11986.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 11987.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 11988.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 11989.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 11990.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 11991.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 11992.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 11993.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 11994.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 11995.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 11996.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 11997