1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPUModifierSyntax 22 AMDGPUOperandSyntax 23 AMDGPUInstructionSyntax 24 AMDGPUInstructionNotation 25 AMDGPUDwarfExtensionsForHeterogeneousDebugging 26 27Introduction 28============ 29 30The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 31R600 family up until the current GCN families. It lives in the 32``llvm/lib/Target/AMDGPU`` directory. 33 34LLVM 35==== 36 37.. _amdgpu-target-triples: 38 39Target Triples 40-------------- 41 42Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 43to specify the target triple: 44 45 .. table:: AMDGPU Architectures 46 :name: amdgpu-architecture-table 47 48 ============ ============================================================== 49 Architecture Description 50 ============ ============================================================== 51 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 52 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 53 ============ ============================================================== 54 55 .. table:: AMDGPU Vendors 56 :name: amdgpu-vendor-table 57 58 ============ ============================================================== 59 Vendor Description 60 ============ ============================================================== 61 ``amd`` Can be used for all AMD GPU usage. 62 ``mesa3d`` Can be used if the OS is ``mesa3d``. 63 ============ ============================================================== 64 65 .. table:: AMDGPU Operating Systems 66 :name: amdgpu-os 67 68 ============== ============================================================ 69 OS Description 70 ============== ============================================================ 71 *<empty>* Defaults to the *unknown* OS. 72 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 73 such as: 74 75 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 76 loader on Linux. See *AMD ROCm Platform Release Notes* 77 [AMD-ROCm-Release-Notes]_ for supported hardware and 78 software. 79 - AMD's PAL runtime using the *pal-amdhsa* loader on 80 Windows. 81 82 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 83 runtime using the *pal-amdpal* loader on Windows and Linux 84 Pro. 85 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 86 3D runtime using the *mesa-mesa3d* loader on Linux. 87 ============== ============================================================ 88 89 .. table:: AMDGPU Environments 90 :name: amdgpu-environment-table 91 92 ============ ============================================================== 93 Environment Description 94 ============ ============================================================== 95 *<empty>* Default. 96 ============ ============================================================== 97 98.. _amdgpu-processors: 99 100Processors 101---------- 102 103Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 104specify the AMDGPU processor together with optional target features. See 105:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 106specific information. 107 108Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 109 110* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 111 112 113 .. table:: AMDGPU Processors 114 :name: amdgpu-processor-table 115 116 =========== =============== ============ ===== ================= =============== =============== ====================== 117 Processor Alternative Target dGPU/ Target Target OS Support Example 118 Processor Triple APU Features Properties *(see* Products 119 Architecture Supported `amdgpu-os`_ 120 *and 121 corresponding 122 runtime release 123 notes for 124 current 125 information and 126 level of 127 support)* 128 =========== =============== ============ ===== ================= =============== =============== ====================== 129 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 130 ----------------------------------------------------------------------------------------------------------------------- 131 ``r600`` ``r600`` dGPU - Does not 132 support 133 generic 134 address 135 space 136 ``r630`` ``r600`` dGPU - Does not 137 support 138 generic 139 address 140 space 141 ``rs880`` ``r600`` dGPU - Does not 142 support 143 generic 144 address 145 space 146 ``rv670`` ``r600`` dGPU - Does not 147 support 148 generic 149 address 150 space 151 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 152 ----------------------------------------------------------------------------------------------------------------------- 153 ``rv710`` ``r600`` dGPU - Does not 154 support 155 generic 156 address 157 space 158 ``rv730`` ``r600`` dGPU - Does not 159 support 160 generic 161 address 162 space 163 ``rv770`` ``r600`` dGPU - Does not 164 support 165 generic 166 address 167 space 168 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 169 ----------------------------------------------------------------------------------------------------------------------- 170 ``cedar`` ``r600`` dGPU - Does not 171 support 172 generic 173 address 174 space 175 ``cypress`` ``r600`` dGPU - Does not 176 support 177 generic 178 address 179 space 180 ``juniper`` ``r600`` dGPU - Does not 181 support 182 generic 183 address 184 space 185 ``redwood`` ``r600`` dGPU - Does not 186 support 187 generic 188 address 189 space 190 ``sumo`` ``r600`` dGPU - Does not 191 support 192 generic 193 address 194 space 195 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 196 ----------------------------------------------------------------------------------------------------------------------- 197 ``barts`` ``r600`` dGPU - Does not 198 support 199 generic 200 address 201 space 202 ``caicos`` ``r600`` dGPU - Does not 203 support 204 generic 205 address 206 space 207 ``cayman`` ``r600`` dGPU - Does not 208 support 209 generic 210 address 211 space 212 ``turks`` ``r600`` dGPU - Does not 213 support 214 generic 215 address 216 space 217 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 218 ----------------------------------------------------------------------------------------------------------------------- 219 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 220 support 221 generic 222 address 223 space 224 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 225 - ``verde`` support 226 generic 227 address 228 space 229 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 230 - ``oland`` support 231 generic 232 address 233 space 234 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 235 ----------------------------------------------------------------------------------------------------------------------- 236 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 237 flat - *pal-amdhsa* - A6 Pro-7050B 238 scratch - *pal-amdpal* - A8-7100 239 - A8 Pro-7150B 240 - A10-7300 241 - A10 Pro-7350B 242 - FX-7500 243 - A8-7200P 244 - A10-7400P 245 - FX-7600P 246 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 247 flat - *pal-amdhsa* - FirePro W9100 248 scratch - *pal-amdpal* - FirePro S9150 249 - FirePro S9170 250 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 251 flat - *pal-amdhsa* - Radeon R9 290x 252 scratch - *pal-amdpal* - Radeon R390 253 - Radeon R390x 254 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 255 - ``mullins`` flat - *pal-amdpal* - E1-2200 256 scratch - E1-2500 257 - E2-3000 258 - E2-3800 259 - A4-5000 260 - A4-5100 261 - A6-5200 262 - A4 Pro-3340B 263 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 264 flat - *pal-amdpal* - Radeon HD 8770 265 scratch - R7 260 266 - R7 260X 267 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 268 flat - *pal-amdpal* 269 scratch .. TODO:: 270 271 Add product 272 names. 273 274 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 275 ----------------------------------------------------------------------------------------------------------------------- 276 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 277 flat - *pal-amdhsa* - Pro A6-8500B 278 scratch - *pal-amdpal* - A8-8600P 279 - Pro A8-8600B 280 - FX-8800P 281 - Pro A12-8800B 282 - A10-8700P 283 - Pro A10-8700B 284 - A10-8780P 285 - A10-9600P 286 - A10-9630P 287 - A12-9700P 288 - A12-9730P 289 - FX-9800P 290 - FX-9830P 291 - E2-9010 292 - A6-9210 293 - A9-9410 294 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 295 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 296 scratch - *pal-amdpal* - Radeon R9 385 297 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 298 - *pal-amdhsa* - Radeon R9 Fury 299 - *pal-amdpal* - Radeon R9 FuryX 300 - Radeon Pro Duo 301 - FirePro S9300x2 302 - Radeon Instinct MI8 303 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 304 flat - *pal-amdhsa* - Radeon RX 480 305 scratch - *pal-amdpal* - Radeon Instinct MI6 306 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 307 flat - *pal-amdhsa* 308 scratch - *pal-amdpal* 309 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 310 flat - *pal-amdhsa* - FirePro S7100 311 scratch - *pal-amdpal* - FirePro W7100 312 - Mobile FirePro 313 M7170 314 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 315 flat - *pal-amdhsa* 316 scratch - *pal-amdpal* .. TODO:: 317 318 Add product 319 names. 320 321 **GCN GFX9 (Vega)** [AMD-GCN-GFX9]_ 322 ----------------------------------------------------------------------------------------------------------------------- 323 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 324 flat - *pal-amdhsa* Frontier Edition 325 scratch - *pal-amdpal* - Radeon RX Vega 56 326 - Radeon RX Vega 64 327 - Radeon RX Vega 64 328 Liquid 329 - Radeon Instinct MI25 330 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 331 flat - *pal-amdhsa* - Ryzen 5 2400G 332 scratch - *pal-amdpal* 333 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 334 - *pal-amdhsa* 335 - *pal-amdpal* .. TODO:: 336 337 Add product 338 names. 339 340 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 341 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 342 scratch - *pal-amdpal* - Radeon VII 343 - Radeon Pro VII 344 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 345 - xnack - Absolute 346 flat 347 scratch 348 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 349 flat 350 scratch .. TODO:: 351 352 Add product 353 names. 354 355 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 356 - tgsplit flat 357 - xnack scratch .. TODO:: 358 - Packed 359 work-item Add product 360 IDs names. 361 362 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 363 flat - Ryzen 7 4700GE 364 scratch - Ryzen 5 4600G 365 - Ryzen 5 4600GE 366 - Ryzen 3 4300G 367 - Ryzen 3 4300GE 368 - Ryzen Pro 4000G 369 - Ryzen 7 Pro 4700G 370 - Ryzen 7 Pro 4750GE 371 - Ryzen 5 Pro 4650G 372 - Ryzen 5 Pro 4650GE 373 - Ryzen 3 Pro 4350G 374 - Ryzen 3 Pro 4350GE 375 376 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 377 ----------------------------------------------------------------------------------------------------------------------- 378 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 379 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 380 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 381 - Radeon Pro 5600M 382 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 383 - wavefrontsize64 - Absolute - *pal-amdhsa* 384 - xnack flat - *pal-amdpal* 385 scratch 386 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 387 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 388 - xnack scratch - *pal-amdpal* 389 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 390 ----------------------------------------------------------------------------------------------------------------------- 391 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 392 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 393 scratch - *pal-amdpal* - Radeon RX 6900 XT 394 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 395 - wavefrontsize64 flat - *pal-amdhsa* 396 scratch - *pal-amdpal* 397 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 398 - wavefrontsize64 flat - *pal-amdhsa* 399 scratch - *pal-amdpal* .. TODO:: 400 401 Add product 402 names. 403 404 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 405 - wavefrontsize64 flat 406 scratch .. TODO:: 407 408 Add product 409 names. 410 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 411 - wavefrontsize64 flat 412 scratch .. TODO:: 413 414 Add product 415 names. 416 417 =========== =============== ============ ===== ================= =============== =============== ====================== 418 419.. _amdgpu-target-features: 420 421Target Features 422--------------- 423 424Target features control how code is generated to support certain 425processor specific features. Not all target features are supported by 426all processors. The runtime must ensure that the features supported by 427the device used to execute the code match the features enabled when 428generating the code. A mismatch of features may result in incorrect 429execution, or a reduction in performance. 430 431The target features supported by each processor is listed in 432:ref:`amdgpu-processor-table`. 433 434Target features are controlled by exactly one of the following Clang 435options: 436 437``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 438 439 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 440 optional components of the target ID. If omitted, the target feature has the 441 ``any`` value. See :ref:`amdgpu-target-id`. 442 443``-m[no-]<target-feature>`` 444 445 Target features not specified by the target ID are specified using a 446 separate option. These target features can have an ``on`` or ``off`` 447 value. ``on`` is specified by omitting the ``no-`` prefix, and 448 ``off`` is specified by including the ``no-`` prefix. The default 449 if not specified is ``off``. 450 451For example: 452 453``-mcpu=gfx908:xnack+`` 454 Enable the ``xnack`` feature. 455``-mcpu=gfx908:xnack-`` 456 Disable the ``xnack`` feature. 457``-mcumode`` 458 Enable the ``cumode`` feature. 459``-mno-cumode`` 460 Disable the ``cumode`` feature. 461 462 .. table:: AMDGPU Target Features 463 :name: amdgpu-target-features-table 464 465 =============== ============================ ================================================== 466 Target Feature Clang Option to Control Description 467 Name 468 =============== ============================ ================================================== 469 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 470 when generating code for kernels. When disabled 471 native WGP wavefront execution mode is used, 472 when enabled CU wavefront execution mode is used 473 (see :ref:`amdgpu-amdhsa-memory-model`). 474 475 sramecc - ``-mcpu`` If specified, generate code that can only be 476 - ``--offload-arch`` loaded and executed in a process that has a 477 matching setting for SRAMECC. 478 479 If not specified for code object V2 to V3, generate 480 code that can be loaded and executed in a process 481 with SRAMECC enabled. 482 483 If not specified for code object V4, generate 484 code that can be loaded and executed in a process 485 with either setting of SRAMECC. 486 487 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 488 work-groups are launched in threadgroup split mode. 489 When enabled the waves of a work-group may be 490 launched in different CUs. 491 492 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 493 generating code for kernels. When disabled 494 native wavefront size 32 is used, when enabled 495 wavefront size 64 is used. 496 497 xnack - ``-mcpu`` If specified, generate code that can only be 498 - ``--offload-arch`` loaded and executed in a process that has a 499 matching setting for XNACK replay. 500 501 If not specified for code object V2 to V3, generate 502 code that can be loaded and executed in a process 503 with XNACK replay enabled. 504 505 If not specified for code object V4, generate 506 code that can be loaded and executed in a process 507 with either setting of XNACK replay. 508 509 XNACK replay can be used for demand paging and 510 page migration. If enabled in the device, then if 511 a page fault occurs the code may execute 512 incorrectly unless generated with XNACK replay 513 enabled, or generated for code object V4 without 514 specifying XNACK replay. Executing code that was 515 generated with XNACK replay enabled, or generated 516 for code object V4 without specifying XNACK replay, 517 on a device that does not have XNACK replay 518 enabled will execute correctly but may be less 519 performant than code generated for XNACK replay 520 disabled. 521 =============== ============================ ================================================== 522 523.. _amdgpu-target-id: 524 525Target ID 526--------- 527 528AMDGPU supports target IDs. See `Clang Offload Bundler 529<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 530description. The AMDGPU target specific information is: 531 532**processor** 533 Is an AMDGPU processor or alternative processor name specified in 534 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 535 the primary processor and alternative processor names. The canonical form 536 target ID only allow the primary processor name. 537 538**target-feature** 539 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 540 is supported by the processor. The target features supported by each processor 541 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 542 a target ID are marked as being controlled by ``-mcpu`` and 543 ``--offload-arch``. Each target feature must appear at most once in a target 544 ID. The non-canonical form target ID allows the target features to be 545 specified in any order. The canonical form target ID requires the target 546 features to be specified in alphabetic order. 547 548.. _amdgpu-target-id-v2-v3: 549 550Code Object V2 to V3 Target ID 551~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 552 553The target ID syntax for code object V2 to V3 is the same as defined in `Clang 554Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 555when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 556directive and the bundle entry ID. In those cases it has the following BNF 557syntax: 558 559.. code:: 560 561 <target-id> ::== <processor> ( "+" <target-feature> )* 562 563Where a target feature is omitted if *Off* and present if *On* or *Any*. 564 565.. note:: 566 567 The code object V2 to V3 cannot represent *Any* and treats it the same as 568 *On*. 569 570.. _amdgpu-embedding-bundled-objects: 571 572Embedding Bundled Code Objects 573------------------------------ 574 575AMDGPU supports the HIP and OpenMP languages that perform code object embedding 576as described in `Clang Offload Bundler 577<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 578 579.. note:: 580 581 The target ID syntax used for code object V2 to V3 for a bundle entry ID 582 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 583 584.. _amdgpu-address-spaces: 585 586Address Spaces 587-------------- 588 589The AMDGPU architecture supports a number of memory address spaces. The address 590space names use the OpenCL standard names, with some additions. 591 592The AMDGPU address spaces correspond to target architecture specific LLVM 593address space numbers used in LLVM IR. 594 595The AMDGPU address spaces are described in 596:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 597supported for the ``amdgcn`` target. 598 599 .. table:: AMDGPU Address Spaces 600 :name: amdgpu-address-spaces-table 601 602 ================================= =============== =========== ================ ======= ============================ 603 .. 64-Bit Process Address Space 604 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 605 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 606 Space Number Name Name Size 607 ================================= =============== =========== ================ ======= ============================ 608 Generic 0 flat flat 64 0x0000000000000000 609 Global 1 global global 64 0x0000000000000000 610 Region 2 N/A GDS 32 *not implemented for AMDHSA* 611 Local 3 group LDS 32 0xFFFFFFFF 612 Constant 4 constant *same as global* 64 0x0000000000000000 613 Private 5 private scratch 32 0xFFFFFFFF 614 Constant 32-bit 6 *TODO* 0x00000000 615 Buffer Fat Pointer (experimental) 7 *TODO* 616 ================================= =============== =========== ================ ======= ============================ 617 618**Generic** 619 The generic address space is supported unless the *Target Properties* column 620 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 621 space*. 622 623 The generic address space uses the hardware flat address support for two fixed 624 ranges of virtual addresses (the private and local apertures), that are 625 outside the range of addressable global memory, to map from a flat address to 626 a private or local address. This uses FLAT instructions that can take a flat 627 address and access global, private (scratch), and group (LDS) memory depending 628 on if the address is within one of the aperture ranges. 629 630 Flat access to scratch requires hardware aperture setup and setup in the 631 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 632 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 633 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 634 635 To convert between a private or group address space address (termed a segment 636 address) and a flat address the base address of the corresponding aperture 637 can be used. For GFX7-GFX8 these are available in the 638 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 639 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 640 GFX9-GFX10 the aperture base addresses are directly available as inline 641 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 642 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 643 aligned to 2^32 which makes it easier to convert from flat to segment or 644 segment to flat. 645 646 A global address space address has the same value when used as a flat address 647 so no conversion is needed. 648 649**Global and Constant** 650 The global and constant address spaces both use global virtual addresses, 651 which are the same virtual address space used by the CPU. However, some 652 virtual addresses may only be accessible to the CPU, some only accessible 653 by the GPU, and some by both. 654 655 Using the constant address space indicates that the data will not change 656 during the execution of the kernel. This allows scalar read instructions to 657 be used. As the constant address space could only be modified on the host 658 side, a generic pointer loaded from the constant address space is safe to be 659 assumed as a global pointer since only the device global memory is visible 660 and managed on the host side. The vector and scalar L1 caches are invalidated 661 of volatile data before each kernel dispatch execution to allow constant 662 memory to change values between kernel dispatches. 663 664**Region** 665 The region address space uses the hardware Global Data Store (GDS). All 666 wavefronts executing on the same device will access the same memory for any 667 given region address. However, the same region address accessed by wavefronts 668 executing on different devices will access different memory. It is higher 669 performance than global memory. It is allocated by the runtime. The data 670 store (DS) instructions can be used to access it. 671 672**Local** 673 The local address space uses the hardware Local Data Store (LDS) which is 674 automatically allocated when the hardware creates the wavefronts of a 675 work-group, and freed when all the wavefronts of a work-group have 676 terminated. All wavefronts belonging to the same work-group will access the 677 same memory for any given local address. However, the same local address 678 accessed by wavefronts belonging to different work-groups will access 679 different memory. It is higher performance than global memory. The data store 680 (DS) instructions can be used to access it. 681 682**Private** 683 The private address space uses the hardware scratch memory support which 684 automatically allocates memory when it creates a wavefront and frees it when 685 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 686 given private address will be different to the memory accessed by another lane 687 of the same or different wavefront for the same private address. 688 689 If a kernel dispatch uses scratch, then the hardware allocates memory from a 690 pool of backing memory allocated by the runtime for each wavefront. The lanes 691 of the wavefront access this using dword (4 byte) interleaving. The mapping 692 used from private address to backing memory address is: 693 694 ``wavefront-scratch-base + 695 ((private-address / 4) * wavefront-size * 4) + 696 (wavefront-lane-id * 4) + (private-address % 4)`` 697 698 If each lane of a wavefront accesses the same private address, the 699 interleaving results in adjacent dwords being accessed and hence requires 700 fewer cache lines to be fetched. 701 702 There are different ways that the wavefront scratch base address is 703 determined by a wavefront (see 704 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 705 706 Scratch memory can be accessed in an interleaved manner using buffer 707 instructions with the scratch buffer descriptor and per wavefront scratch 708 offset, by the scratch instructions, or by flat instructions. Multi-dword 709 access is not supported except by flat and scratch instructions in 710 GFX9-GFX10. 711 712**Constant 32-bit** 713 *TODO* 714 715**Buffer Fat Pointer** 716 The buffer fat pointer is an experimental address space that is currently 717 unsupported in the backend. It exposes a non-integral pointer that is in 718 the future intended to support the modelling of 128-bit buffer descriptors 719 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 720 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 721 model the buffer descriptors used heavily in graphics workloads targeting 722 the backend. 723 724.. _amdgpu-memory-scopes: 725 726Memory Scopes 727------------- 728 729This section provides LLVM memory synchronization scopes supported by the AMDGPU 730backend memory model when the target triple OS is ``amdhsa`` (see 731:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 732 733The memory model supported is based on the HSA memory model [HSA]_ which is 734based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 735relation is transitive over the synchronizes-with relation independent of scope 736and synchronizes-with allows the memory scope instances to be inclusive (see 737table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 738 739This is different to the OpenCL [OpenCL]_ memory model which does not have scope 740inclusion and requires the memory scopes to exactly match. However, this 741is conservatively correct for OpenCL. 742 743 .. table:: AMDHSA LLVM Sync Scopes 744 :name: amdgpu-amdhsa-llvm-sync-scopes-table 745 746 ======================= =================================================== 747 LLVM Sync Scope Description 748 ======================= =================================================== 749 *none* The default: ``system``. 750 751 Synchronizes with, and participates in modification 752 and seq_cst total orderings with, other operations 753 (except image operations) for all address spaces 754 (except private, or generic that accesses private) 755 provided the other operation's sync scope is: 756 757 - ``system``. 758 - ``agent`` and executed by a thread on the same 759 agent. 760 - ``workgroup`` and executed by a thread in the 761 same work-group. 762 - ``wavefront`` and executed by a thread in the 763 same wavefront. 764 765 ``agent`` Synchronizes with, and participates in modification 766 and seq_cst total orderings with, other operations 767 (except image operations) for all address spaces 768 (except private, or generic that accesses private) 769 provided the other operation's sync scope is: 770 771 - ``system`` or ``agent`` and executed by a thread 772 on the same agent. 773 - ``workgroup`` and executed by a thread in the 774 same work-group. 775 - ``wavefront`` and executed by a thread in the 776 same wavefront. 777 778 ``workgroup`` Synchronizes with, and participates in modification 779 and seq_cst total orderings with, other operations 780 (except image operations) for all address spaces 781 (except private, or generic that accesses private) 782 provided the other operation's sync scope is: 783 784 - ``system``, ``agent`` or ``workgroup`` and 785 executed by a thread in the same work-group. 786 - ``wavefront`` and executed by a thread in the 787 same wavefront. 788 789 ``wavefront`` Synchronizes with, and participates in modification 790 and seq_cst total orderings with, other operations 791 (except image operations) for all address spaces 792 (except private, or generic that accesses private) 793 provided the other operation's sync scope is: 794 795 - ``system``, ``agent``, ``workgroup`` or 796 ``wavefront`` and executed by a thread in the 797 same wavefront. 798 799 ``singlethread`` Only synchronizes with and participates in 800 modification and seq_cst total orderings with, 801 other operations (except image operations) running 802 in the same thread for all address spaces (for 803 example, in signal handlers). 804 805 ``one-as`` Same as ``system`` but only synchronizes with other 806 operations within the same address space. 807 808 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 809 operations within the same address space. 810 811 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 812 other operations within the same address space. 813 814 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 815 other operations within the same address space. 816 817 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 818 other operations within the same address space. 819 ======================= =================================================== 820 821LLVM IR Intrinsics 822------------------ 823 824The AMDGPU backend implements the following LLVM IR intrinsics. 825 826*This section is WIP.* 827 828.. TODO:: 829 830 List AMDGPU intrinsics. 831 832LLVM IR Attributes 833------------------ 834 835The AMDGPU backend supports the following LLVM IR attributes. 836 837 .. table:: AMDGPU LLVM IR Attributes 838 :name: amdgpu-llvm-ir-attributes-table 839 840 ======================================= ========================================================== 841 LLVM Attribute Description 842 ======================================= ========================================================== 843 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 844 will be specified when the kernel is dispatched. Generated 845 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 846 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 847 argument block size for the implicit arguments. This 848 varies by OS and language (for OpenCL see 849 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 850 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 851 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 852 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 853 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 854 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 855 execution unit. Generated by the ``amdgpu_waves_per_eu`` 856 CLANG attribute [CLANG-ATTR]_. 857 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 858 mode register to be set on entry. Overrides the default for 859 the calling convention. 860 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 861 the mode register to be set on entry. Overrides the default 862 for the calling convention. 863 ======================================= ========================================================== 864 865.. _amdgpu-elf-code-object: 866 867ELF Code Object 868=============== 869 870The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 871can be linked by ``lld`` to produce a standard ELF shared code object which can 872be loaded and executed on an AMDGPU target. 873 874.. _amdgpu-elf-header: 875 876Header 877------ 878 879The AMDGPU backend uses the following ELF header: 880 881 .. table:: AMDGPU ELF Header 882 :name: amdgpu-elf-header-table 883 884 ========================== =============================== 885 Field Value 886 ========================== =============================== 887 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 888 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 889 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 890 - ``ELFOSABI_AMDGPU_HSA`` 891 - ``ELFOSABI_AMDGPU_PAL`` 892 - ``ELFOSABI_AMDGPU_MESA3D`` 893 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 894 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 895 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 896 - ``ELFABIVERSION_AMDGPU_PAL`` 897 - ``ELFABIVERSION_AMDGPU_MESA3D`` 898 ``e_type`` - ``ET_REL`` 899 - ``ET_DYN`` 900 ``e_machine`` ``EM_AMDGPU`` 901 ``e_entry`` 0 902 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 903 :ref:`amdgpu-elf-header-e_flags-table-v3`, 904 and :ref:`amdgpu-elf-header-e_flags-table-v4` 905 ========================== =============================== 906 907.. 908 909 .. table:: AMDGPU ELF Header Enumeration Values 910 :name: amdgpu-elf-header-enumeration-values-table 911 912 =============================== ===== 913 Name Value 914 =============================== ===== 915 ``EM_AMDGPU`` 224 916 ``ELFOSABI_NONE`` 0 917 ``ELFOSABI_AMDGPU_HSA`` 64 918 ``ELFOSABI_AMDGPU_PAL`` 65 919 ``ELFOSABI_AMDGPU_MESA3D`` 66 920 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 921 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 922 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 923 ``ELFABIVERSION_AMDGPU_PAL`` 0 924 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 925 =============================== ===== 926 927``e_ident[EI_CLASS]`` 928 The ELF class is: 929 930 * ``ELFCLASS32`` for ``r600`` architecture. 931 932 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 933 process address space applications. 934 935``e_ident[EI_DATA]`` 936 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 937 938``e_ident[EI_OSABI]`` 939 One of the following AMDGPU target architecture specific OS ABIs 940 (see :ref:`amdgpu-os`): 941 942 * ``ELFOSABI_NONE`` for *unknown* OS. 943 944 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 945 946 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 947 948 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 949 950``e_ident[EI_ABIVERSION]`` 951 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 952 object conforms: 953 954 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 955 runtime ABI for code object V2. Specify using the Clang option 956 ``-mcode-object-version=2``. 957 958 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 959 runtime ABI for code object V3. Specify using the Clang option 960 ``-mcode-object-version=3``. This is the default code object 961 version if not specified. 962 963 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 964 runtime ABI for code object V4. Specify using the Clang option 965 ``-mcode-object-version=4``. 966 967 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 968 runtime ABI. 969 970 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 971 3D runtime ABI. 972 973``e_type`` 974 Can be one of the following values: 975 976 977 ``ET_REL`` 978 The type produced by the AMDGPU backend compiler as it is relocatable code 979 object. 980 981 ``ET_DYN`` 982 The type produced by the linker as it is a shared code object. 983 984 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 985 986``e_machine`` 987 The value ``EM_AMDGPU`` is used for the machine for all processors supported 988 by the ``r600`` and ``amdgcn`` architectures (see 989 :ref:`amdgpu-processor-table`). The specific processor is specified in the 990 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 991 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 992 ``e_flags`` for code object V3 to V4 (see 993 :ref:`amdgpu-elf-header-e_flags-table-v3` and 994 :ref:`amdgpu-elf-header-e_flags-table-v4`). 995 996``e_entry`` 997 The entry point is 0 as the entry points for individual kernels must be 998 selected in order to invoke them through AQL packets. 999 1000``e_flags`` 1001 The AMDGPU backend uses the following ELF header flags: 1002 1003 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1004 :name: amdgpu-elf-header-e_flags-v2-table 1005 1006 ===================================== ===== ============================= 1007 Name Value Description 1008 ===================================== ===== ============================= 1009 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1010 target feature is 1011 enabled for all code 1012 contained in the code object. 1013 If the processor 1014 does not support the 1015 ``xnack`` target 1016 feature then must 1017 be 0. 1018 See 1019 :ref:`amdgpu-target-features`. 1020 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1021 handler is enabled for all 1022 code contained in the code 1023 object. If the processor 1024 does not support a trap 1025 handler then must be 0. 1026 See 1027 :ref:`amdgpu-target-features`. 1028 ===================================== ===== ============================= 1029 1030 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1031 :name: amdgpu-elf-header-e_flags-table-v3 1032 1033 ================================= ===== ============================= 1034 Name Value Description 1035 ================================= ===== ============================= 1036 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1037 mask for 1038 ``EF_AMDGPU_MACH_xxx`` values 1039 defined in 1040 :ref:`amdgpu-ef-amdgpu-mach-table`. 1041 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1042 target feature is 1043 enabled for all code 1044 contained in the code object. 1045 If the processor 1046 does not support the 1047 ``xnack`` target 1048 feature then must 1049 be 0. 1050 See 1051 :ref:`amdgpu-target-features`. 1052 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1053 target feature is 1054 enabled for all code 1055 contained in the code object. 1056 If the processor 1057 does not support the 1058 ``sramecc`` target 1059 feature then must 1060 be 0. 1061 See 1062 :ref:`amdgpu-target-features`. 1063 ================================= ===== ============================= 1064 1065 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1066 :name: amdgpu-elf-header-e_flags-table-v4 1067 1068 ============================================ ===== =================================== 1069 Name Value Description 1070 ============================================ ===== =================================== 1071 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1072 mask for 1073 ``EF_AMDGPU_MACH_xxx`` values 1074 defined in 1075 :ref:`amdgpu-ef-amdgpu-mach-table`. 1076 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1077 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1078 values. 1079 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1080 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1081 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1082 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1083 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1084 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1085 values. 1086 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1087 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1088 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1089 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1090 ============================================ ===== =================================== 1091 1092 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1093 :name: amdgpu-ef-amdgpu-mach-table 1094 1095 ==================================== ========== ============================= 1096 Name Value Description (see 1097 :ref:`amdgpu-processor-table`) 1098 ==================================== ========== ============================= 1099 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1100 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1101 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1102 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1103 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1104 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1105 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1106 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1107 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1108 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1109 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1110 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1111 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1112 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1113 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1114 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1115 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1116 *reserved* 0x011 - Reserved for ``r600`` 1117 0x01f architecture processors. 1118 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1119 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1120 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1121 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1122 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1123 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1124 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1125 *reserved* 0x027 Reserved. 1126 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1127 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1128 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1129 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1130 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1131 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1132 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1133 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1134 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1135 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1136 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1137 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1138 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1139 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1140 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1141 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1142 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1143 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1144 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1145 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1146 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1147 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1148 *reserved* 0x03d Reserved. 1149 *reserved* 0x03e Reserved. 1150 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1151 *reserved* 0x040 Reserved. 1152 *reserved* 0x041 Reserved. 1153 ==================================== ========== ============================= 1154 1155Sections 1156-------- 1157 1158An AMDGPU target ELF code object has the standard ELF sections which include: 1159 1160 .. table:: AMDGPU ELF Sections 1161 :name: amdgpu-elf-sections-table 1162 1163 ================== ================ ================================= 1164 Name Type Attributes 1165 ================== ================ ================================= 1166 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1167 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1168 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1169 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1170 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1171 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1172 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1173 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1174 ``.note`` ``SHT_NOTE`` *none* 1175 ``.rela``\ *name* ``SHT_RELA`` *none* 1176 ``.rela.dyn`` ``SHT_RELA`` *none* 1177 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1178 ``.shstrtab`` ``SHT_STRTAB`` *none* 1179 ``.strtab`` ``SHT_STRTAB`` *none* 1180 ``.symtab`` ``SHT_SYMTAB`` *none* 1181 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1182 ================== ================ ================================= 1183 1184These sections have their standard meanings (see [ELF]_) and are only generated 1185if needed. 1186 1187``.debug``\ *\** 1188 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1189 information on the DWARF produced by the AMDGPU backend. 1190 1191``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1192 The standard sections used by a dynamic loader. 1193 1194``.note`` 1195 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1196 backend. 1197 1198``.rela``\ *name*, ``.rela.dyn`` 1199 For relocatable code objects, *name* is the name of the section that the 1200 relocation records apply. For example, ``.rela.text`` is the section name for 1201 relocation records associated with the ``.text`` section. 1202 1203 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1204 records from each of the relocatable code object's ``.rela``\ *name* sections. 1205 1206 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1207 the AMDGPU backend. 1208 1209``.text`` 1210 The executable machine code for the kernels and functions they call. Generated 1211 as position independent code. See :ref:`amdgpu-code-conventions` for 1212 information on conventions used in the isa generation. 1213 1214.. _amdgpu-note-records: 1215 1216Note Records 1217------------ 1218 1219The AMDGPU backend code object contains ELF note records in the ``.note`` 1220section. The set of generated notes and their semantics depend on the code 1221object version; see :ref:`amdgpu-note-records-v2` and 1222:ref:`amdgpu-note-records-v3-v4`. 1223 1224As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1225must be generated after the ``name`` field to ensure the ``desc`` field is 4 1226byte aligned. In addition, minimal zero-byte padding must be generated to 1227ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1228field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1229alignment. 1230 1231.. _amdgpu-note-records-v2: 1232 1233Code Object V2 Note Records 1234~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1235 1236.. warning:: 1237 Code object V2 is not the default code object version emitted by 1238 this version of LLVM. 1239 1240The AMDGPU backend code object uses the following ELF note record in the 1241``.note`` section when compiling for code object V2. 1242 1243The note record vendor field is "AMD". 1244 1245Additional note records may be present, but any which are not documented here 1246are deprecated and should not be used. 1247 1248 .. table:: AMDGPU Code Object V2 ELF Note Records 1249 :name: amdgpu-elf-note-records-v2-table 1250 1251 ===== ===================================== ====================================== 1252 Name Type Description 1253 ===== ===================================== ====================================== 1254 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1255 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1256 Finalizer and not the LLVM compiler. 1257 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1258 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1259 YAML [YAML]_ textual format. 1260 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1261 ===== ===================================== ====================================== 1262 1263.. 1264 1265 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1266 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1267 1268 ===================================== ===== 1269 Name Value 1270 ===================================== ===== 1271 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1272 ``NT_AMD_HSA_HSAIL`` 2 1273 ``NT_AMD_HSA_ISA_VERSION`` 3 1274 *reserved* 4-9 1275 ``NT_AMD_HSA_METADATA`` 10 1276 ``NT_AMD_HSA_ISA_NAME`` 11 1277 ===================================== ===== 1278 1279``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1280 Specifies the code object version number. The description field has the 1281 following layout: 1282 1283 .. code:: 1284 1285 struct amdgpu_hsa_note_code_object_version_s { 1286 uint32_t major_version; 1287 uint32_t minor_version; 1288 }; 1289 1290 The ``major_version`` has a value less than or equal to 2. 1291 1292``NT_AMD_HSA_HSAIL`` 1293 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1294 field has the following layout: 1295 1296 .. code:: 1297 1298 struct amdgpu_hsa_note_hsail_s { 1299 uint32_t hsail_major_version; 1300 uint32_t hsail_minor_version; 1301 uint8_t profile; 1302 uint8_t machine_model; 1303 uint8_t default_float_round; 1304 }; 1305 1306``NT_AMD_HSA_ISA_VERSION`` 1307 Specifies the target ISA version. The description field has the following layout: 1308 1309 .. code:: 1310 1311 struct amdgpu_hsa_note_isa_s { 1312 uint16_t vendor_name_size; 1313 uint16_t architecture_name_size; 1314 uint32_t major; 1315 uint32_t minor; 1316 uint32_t stepping; 1317 char vendor_and_architecture_name[1]; 1318 }; 1319 1320 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1321 vendor and architecture names respectively, including the NUL character. 1322 1323 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1324 vendor, immediately followed by the NUL terminated string for the 1325 architecture. 1326 1327 This note record is used by the HSA runtime loader. 1328 1329 Code object V2 only supports a limited number of processors and has fixed 1330 settings for target features. See 1331 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1332 processors and the corresponding target ID. In the table the note record ISA 1333 name is a concatenation of the vendor name, architecture name, major, minor, 1334 and stepping separated by a ":". 1335 1336 The target ID column shows the processor name and fixed target features used 1337 by the LLVM compiler. The LLVM compiler does not generate a 1338 ``NT_AMD_HSA_HSAIL`` note record. 1339 1340 A code object generated by the Finalizer also uses code object V2 and always 1341 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1342 ``sramecc`` target feature is as shown in 1343 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1344 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1345 bit. 1346 1347``NT_AMD_HSA_ISA_NAME`` 1348 Specifies the target ISA name as a non-NUL terminated string. 1349 1350 This note record is not used by the HSA runtime loader. 1351 1352 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1353 V2's limited support of processors and fixed settings for target features. 1354 1355 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1356 from the string to the corresponding target ID. If the ``xnack`` target 1357 feature is supported and enabled, the string produced by the LLVM compiler 1358 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1359 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1360 1361``NT_AMD_HSA_METADATA`` 1362 Specifies extensible metadata associated with the code objects executed on HSA 1363 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1364 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1365 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1366 metadata string. 1367 1368 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1369 :name: amdgpu-elf-note-record-supported_processors-v2-table 1370 1371 ===================== ========================== 1372 Note Record ISA Name Target ID 1373 ===================== ========================== 1374 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1375 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1376 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1377 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1378 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1379 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1380 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1381 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1382 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1383 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1384 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1385 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1386 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1387 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1388 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1389 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1390 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1391 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1392 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1393 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1394 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1395 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1396 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1397 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1398 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1399 ===================== ========================== 1400 1401.. _amdgpu-note-records-v3-v4: 1402 1403Code Object V3 to V4 Note Records 1404~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1405 1406The AMDGPU backend code object uses the following ELF note record in the 1407``.note`` section when compiling for code object V3 to V4. 1408 1409The note record vendor field is "AMDGPU". 1410 1411Additional note records may be present, but any which are not documented here 1412are deprecated and should not be used. 1413 1414 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1415 :name: amdgpu-elf-note-records-table-v3-v4 1416 1417 ======== ============================== ====================================== 1418 Name Type Description 1419 ======== ============================== ====================================== 1420 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1421 binary format. 1422 ======== ============================== ====================================== 1423 1424.. 1425 1426 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1427 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1428 1429 ============================== ===== 1430 Name Value 1431 ============================== ===== 1432 *reserved* 0-31 1433 ``NT_AMDGPU_METADATA`` 32 1434 ============================== ===== 1435 1436``NT_AMDGPU_METADATA`` 1437 Specifies extensible metadata associated with an AMDGPU code object. It is 1438 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1439 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1440 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1441 ``amdhsa`` OS. 1442 1443.. _amdgpu-symbols: 1444 1445Symbols 1446------- 1447 1448Symbols include the following: 1449 1450 .. table:: AMDGPU ELF Symbols 1451 :name: amdgpu-elf-symbols-table 1452 1453 ===================== ================== ================ ================== 1454 Name Type Section Description 1455 ===================== ================== ================ ================== 1456 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1457 - ``.rodata`` 1458 - ``.bss`` 1459 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1460 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1461 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1462 ===================== ================== ================ ================== 1463 1464Global variable 1465 Global variables both used and defined by the compilation unit. 1466 1467 If the symbol is defined in the compilation unit then it is allocated in the 1468 appropriate section according to if it has initialized data or is readonly. 1469 1470 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1471 will resolve relocations using the definition provided by another code object 1472 or explicitly defined by the runtime. 1473 1474 If the symbol resides in local/group memory (LDS) then its section is the 1475 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1476 ``st_value`` field describes alignment requirements as it does for common 1477 symbols. 1478 1479 .. TODO:: 1480 1481 Add description of linked shared object symbols. Seems undefined symbols 1482 are marked as STT_NOTYPE. 1483 1484Kernel descriptor 1485 Every HSA kernel has an associated kernel descriptor. It is the address of the 1486 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1487 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1488 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1489 1490Kernel entry point 1491 Every HSA kernel also has a symbol for its machine code entry point. 1492 1493.. _amdgpu-relocation-records: 1494 1495Relocation Records 1496------------------ 1497 1498AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1499relocatable fields are: 1500 1501``word32`` 1502 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1503 alignment. These values use the same byte order as other word values in the 1504 AMDGPU architecture. 1505 1506``word64`` 1507 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1508 alignment. These values use the same byte order as other word values in the 1509 AMDGPU architecture. 1510 1511Following notations are used for specifying relocation calculations: 1512 1513**A** 1514 Represents the addend used to compute the value of the relocatable field. 1515 1516**G** 1517 Represents the offset into the global offset table at which the relocation 1518 entry's symbol will reside during execution. 1519 1520**GOT** 1521 Represents the address of the global offset table. 1522 1523**P** 1524 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1525 of the storage unit being relocated (computed using ``r_offset``). 1526 1527**S** 1528 Represents the value of the symbol whose index resides in the relocation 1529 entry. Relocations not using this must specify a symbol index of 1530 ``STN_UNDEF``. 1531 1532**B** 1533 Represents the base address of a loaded executable or shared object which is 1534 the difference between the ELF address and the actual load address. 1535 Relocations using this are only valid in executable or shared objects. 1536 1537The following relocation types are supported: 1538 1539 .. table:: AMDGPU ELF Relocation Records 1540 :name: amdgpu-elf-relocation-records-table 1541 1542 ========================== ======= ===== ========== ============================== 1543 Relocation Type Kind Value Field Calculation 1544 ========================== ======= ===== ========== ============================== 1545 ``R_AMDGPU_NONE`` 0 *none* *none* 1546 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1547 Dynamic 1548 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1549 Dynamic 1550 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1551 Dynamic 1552 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1553 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1554 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1555 Dynamic 1556 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1557 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1558 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1559 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1560 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1561 *reserved* 12 1562 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1563 ========================== ======= ===== ========== ============================== 1564 1565``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1566the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1567 1568There is no current OS loader support for 32-bit programs and so 1569``R_AMDGPU_ABS32`` is not used. 1570 1571.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1572 1573Loaded Code Object Path Uniform Resource Identifier (URI) 1574--------------------------------------------------------- 1575 1576The AMD GPU code object loader represents the path of the ELF shared object from 1577which the code object was loaded as a textual Unifom Resource Identifier (URI). 1578Note that the code object is the in memory loaded relocated form of the ELF 1579shared object. Multiple code objects may be loaded at different memory 1580addresses in the same process from the same ELF shared object. 1581 1582The loaded code object path URI syntax is defined by the following BNF syntax: 1583 1584.. code:: 1585 1586 code_object_uri ::== file_uri | memory_uri 1587 file_uri ::== "file://" file_path [ range_specifier ] 1588 memory_uri ::== "memory://" process_id range_specifier 1589 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1590 file_path ::== URI_ENCODED_OS_FILE_PATH 1591 process_id ::== DECIMAL_NUMBER 1592 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1593 1594**number** 1595 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1596 and octal values by "0". 1597 1598**file_path** 1599 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1600 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1601 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1602 the path are separated by "/". 1603 1604**offset** 1605 Is a 0-based byte offset to the start of the code object. For a file URI, it 1606 is from the start of the file specified by the ``file_path``, and if omitted 1607 defaults to 0. For a memory URI, it is the memory address and is required. 1608 1609**size** 1610 Is the number of bytes in the code object. For a file URI, if omitted it 1611 defaults to the size of the file. It is required for a memory URI. 1612 1613**process_id** 1614 Is the identity of the process owning the memory. For Linux it is the C 1615 unsigned integral decimal literal for the process ID (PID). 1616 1617For example: 1618 1619.. code:: 1620 1621 file:///dir1/dir2/file1 1622 file:///dir3/dir4/file2#offset=0x2000&size=3000 1623 memory://1234#offset=0x20000&size=3000 1624 1625.. _amdgpu-dwarf-debug-information: 1626 1627DWARF Debug Information 1628======================= 1629 1630.. warning:: 1631 1632 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1633 is not currently fully implemented and is subject to change. 1634 1635AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1636:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1637object executable code and data to the source language constructs. It can be 1638used by tools such as debuggers and profilers. It uses features defined in 1639:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1640DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1641 1642This section defines the AMDGPU target architecture specific DWARF mappings. 1643 1644.. _amdgpu-dwarf-register-identifier: 1645 1646Register Identifier 1647------------------- 1648 1649This section defines the AMDGPU target architecture register numbers used in 1650DWARF operation expressions (see DWARF Version 5 section 2.5 and 1651:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1652instructions (see DWARF Version 5 section 6.4 and 1653:ref:`amdgpu-dwarf-call-frame-information`). 1654 1655A single code object can contain code for kernels that have different wavefront 1656sizes. The vector registers and some scalar registers are based on the wavefront 1657size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1658simplifies the consumer of the DWARF so that each register has a fixed size, 1659rather than being dynamic according to the wavefront size mode. Similarly, 1660distinct DWARF registers are defined for those registers that vary in size 1661according to the process address size. This allows a consumer to treat a 1662specific AMDGPU processor as a single architecture regardless of how it is 1663configured at run time. The compiler explicitly specifies the DWARF registers 1664that match the mode in which the code it is generating will be executed. 1665 1666DWARF registers are encoded as numbers, which are mapped to architecture 1667registers. The mapping for AMDGPU is defined in 1668:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1669mapping. 1670 1671.. table:: AMDGPU DWARF Register Mapping 1672 :name: amdgpu-dwarf-register-mapping-table 1673 1674 ============== ================= ======== ================================== 1675 DWARF Register AMDGPU Register Bit Size Description 1676 ============== ================= ======== ================================== 1677 0 PC_32 32 Program Counter (PC) when 1678 executing in a 32-bit process 1679 address space. Used in the CFI to 1680 describe the PC of the calling 1681 frame. 1682 1 EXEC_MASK_32 32 Execution Mask Register when 1683 executing in wavefront 32 mode. 1684 2-15 *Reserved* *Reserved for highly accessed 1685 registers using DWARF shortcut.* 1686 16 PC_64 64 Program Counter (PC) when 1687 executing in a 64-bit process 1688 address space. Used in the CFI to 1689 describe the PC of the calling 1690 frame. 1691 17 EXEC_MASK_64 64 Execution Mask Register when 1692 executing in wavefront 64 mode. 1693 18-31 *Reserved* *Reserved for highly accessed 1694 registers using DWARF shortcut.* 1695 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1696 Registers. 1697 96-127 *Reserved* *Reserved for frequently accessed 1698 registers using DWARF 1-byte ULEB.* 1699 128 STATUS 32 Status Register. 1700 129-511 *Reserved* *Reserved for future Scalar 1701 Architectural Registers.* 1702 512 VCC_32 32 Vector Condition Code Register 1703 when executing in wavefront 32 1704 mode. 1705 513-1023 *Reserved* *Reserved for future Vector 1706 Architectural Registers when 1707 executing in wavefront 32 mode.* 1708 768 VCC_64 64 Vector Condition Code Register 1709 when executing in wavefront 64 1710 mode. 1711 769-1023 *Reserved* *Reserved for future Vector 1712 Architectural Registers when 1713 executing in wavefront 64 mode.* 1714 1024-1087 *Reserved* *Reserved for padding.* 1715 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1716 1130-1535 *Reserved* *Reserved for future Scalar 1717 General Purpose Registers.* 1718 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1719 when executing in wavefront 32 1720 mode. 1721 1792-2047 *Reserved* *Reserved for future Vector 1722 General Purpose Registers when 1723 executing in wavefront 32 mode.* 1724 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1725 when executing in wavefront 32 1726 mode. 1727 2304-2559 *Reserved* *Reserved for future Vector 1728 Accumulation Registers when 1729 executing in wavefront 32 mode.* 1730 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1731 when executing in wavefront 64 1732 mode. 1733 2816-3071 *Reserved* *Reserved for future Vector 1734 General Purpose Registers when 1735 executing in wavefront 64 mode.* 1736 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1737 when executing in wavefront 64 1738 mode. 1739 3328-3583 *Reserved* *Reserved for future Vector 1740 Accumulation Registers when 1741 executing in wavefront 64 mode.* 1742 ============== ================= ======== ================================== 1743 1744The vector registers are represented as the full size for the wavefront. They 1745are organized as consecutive dwords (32-bits), one per lane, with the dword at 1746the least significant bit position corresponding to lane 0 and so forth. DWARF 1747location expressions involving the ``DW_OP_LLVM_offset`` and 1748``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1749register corresponding to the lane that is executing the current thread of 1750execution in languages that are implemented using a SIMD or SIMT execution 1751model. 1752 1753If the wavefront size is 32 lanes then the wavefront 32 mode register 1754definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1755mode register definitions are used. Some AMDGPU targets support executing in 1756both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1757to the wavefront mode of the generated code will be used. 1758 1759If code is generated to execute in a 32-bit process address space, then the 176032-bit process address space register definitions are used. If code is generated 1761to execute in a 64-bit process address space, then the 64-bit process address 1762space register definitions are used. The ``amdgcn`` target only supports the 176364-bit process address space. 1764 1765.. _amdgpu-dwarf-address-class-identifier: 1766 1767Address Class Identifier 1768------------------------ 1769 1770The DWARF address class represents the source language memory space. See DWARF 1771Version 5 section 2.12 which is updated by the *DWARF Extensions For 1772Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1773 1774The DWARF address class mapping used for AMDGPU is defined in 1775:ref:`amdgpu-dwarf-address-class-mapping-table`. 1776 1777.. table:: AMDGPU DWARF Address Class Mapping 1778 :name: amdgpu-dwarf-address-class-mapping-table 1779 1780 ========================= ====== ================= 1781 DWARF AMDGPU 1782 -------------------------------- ----------------- 1783 Address Class Name Value Address Space 1784 ========================= ====== ================= 1785 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1786 ``DW_ADDR_LLVM_global`` 0x0001 Global 1787 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1788 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1789 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1790 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1791 ========================= ====== ================= 1792 1793The DWARF address class values defined in the *DWARF Extensions For 1794Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1795 1796In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1797available for use for the AMD extension for access to the hardware GDS memory 1798which is scratchpad memory allocated per device. 1799 1800For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1801address class of ``DW_ADDR_none`` is used. 1802 1803See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1804mapping of DWARF address classes to DWARF address spaces, including address size 1805and NULL value. 1806 1807.. _amdgpu-dwarf-address-space-identifier: 1808 1809Address Space Identifier 1810------------------------ 1811 1812DWARF address spaces correspond to target architecture specific linear 1813addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1814For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1815 1816The DWARF address space mapping used for AMDGPU is defined in 1817:ref:`amdgpu-dwarf-address-space-mapping-table`. 1818 1819.. table:: AMDGPU DWARF Address Space Mapping 1820 :name: amdgpu-dwarf-address-space-mapping-table 1821 1822 ======================================= ===== ======= ======== ================= ======================= 1823 DWARF AMDGPU Notes 1824 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1825 Address Space Name Value Address Bit Size Address Space 1826 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1827 .. 64-bit 32-bit 1828 process process 1829 address address 1830 space space 1831 ======================================= ===== ======= ======== ================= ======================= 1832 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1833 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1834 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1835 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1836 *Reserved* 0x04 1837 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1838 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1839 ======================================= ===== ======= ======== ================= ======================= 1840 1841See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1842including address size and NULL value. 1843 1844The ``DW_ASPACE_none`` address space is the default target architecture address 1845space used in DWARF operations that do not specify an address space. It 1846therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1847related operations can refer to addresses in the program code. 1848 1849The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1850specify the flat address space. If the address corresponds to an address in the 1851local address space, then it corresponds to the wavefront that is executing the 1852focused thread of execution. If the address corresponds to an address in the 1853private address space, then it corresponds to the lane that is executing the 1854focused thread of execution for languages that are implemented using a SIMD or 1855SIMT execution model. 1856 1857.. note:: 1858 1859 CUDA-like languages such as HIP that do not have address spaces in the 1860 language type system, but do allow variables to be allocated in different 1861 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1862 address space in the DWARF expression operations as the default address space 1863 is the global address space. 1864 1865The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1866specify the local address space corresponding to the wavefront that is executing 1867the focused thread of execution. 1868 1869The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1870to specify the private address space corresponding to the lane that is executing 1871the focused thread of execution for languages that are implemented using a SIMD 1872or SIMT execution model. 1873 1874The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1875to specify the unswizzled private address space corresponding to the wavefront 1876that is executing the focused thread of execution. The wavefront view of private 1877memory is the per wavefront unswizzled backing memory layout defined in 1878:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1879location for the backing memory of the wavefront (namely the address is not 1880offset by ``wavefront-scratch-base``). The following formula can be used to 1881convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1882``DW_ASPACE_AMDGPU_private_wave`` address: 1883 1884:: 1885 1886 private-address-wavefront = 1887 ((private-address-lane / 4) * wavefront-size * 4) + 1888 (wavefront-lane-id * 4) + (private-address-lane % 4) 1889 1890If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1891of the dwords for each lane starting with lane 0 is required, then this 1892simplifies to: 1893 1894:: 1895 1896 private-address-wavefront = 1897 private-address-lane * wavefront-size 1898 1899A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1900complete spilled vector register back into a complete vector register in the 1901CFI. The frame pointer can be a private lane address which is dword aligned, 1902which can be shifted to multiply by the wavefront size, and then used to form a 1903private wavefront address that gives a location for a contiguous set of dwords, 1904one per lane, where the vector register dwords are spilled. The compiler knows 1905the wavefront size since it generates the code. Note that the type of the 1906address may have to be converted as the size of a 1907``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1908``DW_ASPACE_AMDGPU_private_wave`` address. 1909 1910.. _amdgpu-dwarf-lane-identifier: 1911 1912Lane identifier 1913--------------- 1914 1915DWARF lane identifies specify a target architecture lane position for hardware 1916that executes in a SIMD or SIMT manner, and on which a source language maps its 1917threads of execution onto those lanes. The DWARF lane identifier is pushed by 1918the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1919section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1920section :ref:`amdgpu-dwarf-operation-expressions`. 1921 1922For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1923wavefront. It is numbered from 0 to the wavefront size minus 1. 1924 1925Operation Expressions 1926--------------------- 1927 1928DWARF expressions are used to compute program values and the locations of 1929program objects. See DWARF Version 5 section 2.5 and 1930:ref:`amdgpu-dwarf-operation-expressions`. 1931 1932DWARF location descriptions describe how to access storage which includes memory 1933and registers. When accessing storage on AMDGPU, bytes are ordered with least 1934significant bytes first, and bits are ordered within bytes with least 1935significant bits first. 1936 1937For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1938unwinding vector registers that are spilled under the execution mask to memory: 1939the zero-single location description is the vector register, and the one-single 1940location description is the spilled memory location description. The 1941``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1942memory location description. 1943 1944In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1945``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1946controlled by the execution mask. An undefined location description together 1947with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1948to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1949 1950Debugger Information Entry Attributes 1951------------------------------------- 1952 1953This section describes how certain debugger information entry attributes are 1954used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1955by *DWARF Extensions For Heterogeneous Debugging* section 1956:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1957 1958.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1959 1960``DW_AT_LLVM_lane_pc`` 1961~~~~~~~~~~~~~~~~~~~~~~ 1962 1963For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1964location of the separate lanes of a SIMT thread. 1965 1966If the lane is an active lane then this will be the same as the current program 1967location. 1968 1969If the lane is inactive, but was active on entry to the subprogram, then this is 1970the program location in the subprogram at which execution of the lane is 1971conceptual positioned. 1972 1973If the lane was not active on entry to the subprogram, then this will be the 1974undefined location. A client debugger can check if the lane is part of a valid 1975work-group by checking that the lane is in the range of the associated 1976work-group within the grid, accounting for partial work-groups. If it is not, 1977then the debugger can omit any information for the lane. Otherwise, the debugger 1978may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1979calling subprogram until it finds a non-undefined location. Conceptually the 1980lane only has the call frames that it has a non-undefined 1981``DW_AT_LLVM_lane_pc``. 1982 1983The following example illustrates how the AMDGPU backend can generate a DWARF 1984location list expression for the nested ``IF/THEN/ELSE`` structures of the 1985following subprogram pseudo code for a target with 64 lanes per wavefront. 1986 1987.. code:: 1988 :number-lines: 1989 1990 SUBPROGRAM X 1991 BEGIN 1992 a; 1993 IF (c1) THEN 1994 b; 1995 IF (c2) THEN 1996 c; 1997 ELSE 1998 d; 1999 ENDIF 2000 e; 2001 ELSE 2002 f; 2003 ENDIF 2004 g; 2005 END 2006 2007The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2008execution mask (``EXEC``) to linearize the control flow. The condition is 2009evaluated to make a mask of the lanes for which the condition evaluates to true. 2010First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2011logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2012``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2013the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2014region the ``EXEC`` mask is restored to the value it had at the beginning of the 2015region. This is shown below. Other approaches are possible, but the basic 2016concept is the same. 2017 2018.. code:: 2019 :number-lines: 2020 2021 $lex_start: 2022 a; 2023 %1 = EXEC 2024 %2 = c1 2025 $lex_1_start: 2026 EXEC = %1 & %2 2027 $if_1_then: 2028 b; 2029 %3 = EXEC 2030 %4 = c2 2031 $lex_1_1_start: 2032 EXEC = %3 & %4 2033 $lex_1_1_then: 2034 c; 2035 EXEC = ~EXEC & %3 2036 $lex_1_1_else: 2037 d; 2038 EXEC = %3 2039 $lex_1_1_end: 2040 e; 2041 EXEC = ~EXEC & %1 2042 $lex_1_else: 2043 f; 2044 EXEC = %1 2045 $lex_1_end: 2046 g; 2047 $lex_end: 2048 2049To create the DWARF location list expression that defines the location 2050description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2051pseudo instruction can be used to annotate the linearized control flow. This can 2052be done by defining an artificial variable for the lane PC. The DWARF location 2053list expression created for it is used as the value of the 2054``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2055 2056A DWARF procedure is defined for each well nested structured control flow region 2057which provides the conceptual lane program location for a lane if it is not 2058active (namely it is divergent). The DWARF operation expression for each region 2059conceptually inherits the value of the immediately enclosing region and modifies 2060it according to the semantics of the region. 2061 2062For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2063the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2064region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2065region since the ``THEN`` region has completed. 2066 2067The lane PC artificial variable is assigned at each region transition. It uses 2068the immediately enclosing region's DWARF procedure to compute the program 2069location for each lane assuming they are divergent, and then modifies the result 2070by inserting the current program location for each lane that the ``EXEC`` mask 2071indicates is active. 2072 2073By having separate DWARF procedures for each region, they can be reused to 2074define the value for any nested region. This reduces the total size of the DWARF 2075operation expressions. 2076 2077The following provides an example using pseudo LLVM MIR. 2078 2079.. code:: 2080 :number-lines: 2081 2082 $lex_start: 2083 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2084 DW_AT_name = "__uint64"; 2085 DW_AT_byte_size = 8; 2086 DW_AT_encoding = DW_ATE_unsigned; 2087 ]; 2088 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2089 DW_AT_name = "__active_lane_pc"; 2090 DW_AT_location = [ 2091 DW_OP_regx PC; 2092 DW_OP_LLVM_extend 64, 64; 2093 DW_OP_regval_type EXEC, %uint_64; 2094 DW_OP_LLVM_select_bit_piece 64, 64; 2095 ]; 2096 ]; 2097 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2098 DW_AT_name = "__divergent_lane_pc"; 2099 DW_AT_location = [ 2100 DW_OP_LLVM_undefined; 2101 DW_OP_LLVM_extend 64, 64; 2102 ]; 2103 ]; 2104 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2105 DW_OP_call_ref %__divergent_lane_pc; 2106 DW_OP_call_ref %__active_lane_pc; 2107 ]; 2108 a; 2109 %1 = EXEC; 2110 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2111 %2 = c1; 2112 $lex_1_start: 2113 EXEC = %1 & %2; 2114 $lex_1_then: 2115 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2116 DW_AT_name = "__divergent_lane_pc_1_then"; 2117 DW_AT_location = DIExpression[ 2118 DW_OP_call_ref %__divergent_lane_pc; 2119 DW_OP_addrx &lex_1_start; 2120 DW_OP_stack_value; 2121 DW_OP_LLVM_extend 64, 64; 2122 DW_OP_call_ref %__lex_1_save_exec; 2123 DW_OP_deref_type 64, %__uint_64; 2124 DW_OP_LLVM_select_bit_piece 64, 64; 2125 ]; 2126 ]; 2127 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2128 DW_OP_call_ref %__divergent_lane_pc_1_then; 2129 DW_OP_call_ref %__active_lane_pc; 2130 ]; 2131 b; 2132 %3 = EXEC; 2133 DBG_VALUE %3, %__lex_1_1_save_exec; 2134 %4 = c2; 2135 $lex_1_1_start: 2136 EXEC = %3 & %4; 2137 $lex_1_1_then: 2138 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2139 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2140 DW_AT_location = DIExpression[ 2141 DW_OP_call_ref %__divergent_lane_pc_1_then; 2142 DW_OP_addrx &lex_1_1_start; 2143 DW_OP_stack_value; 2144 DW_OP_LLVM_extend 64, 64; 2145 DW_OP_call_ref %__lex_1_1_save_exec; 2146 DW_OP_deref_type 64, %__uint_64; 2147 DW_OP_LLVM_select_bit_piece 64, 64; 2148 ]; 2149 ]; 2150 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2151 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2152 DW_OP_call_ref %__active_lane_pc; 2153 ]; 2154 c; 2155 EXEC = ~EXEC & %3; 2156 $lex_1_1_else: 2157 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2158 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2159 DW_AT_location = DIExpression[ 2160 DW_OP_call_ref %__divergent_lane_pc_1_then; 2161 DW_OP_addrx &lex_1_1_end; 2162 DW_OP_stack_value; 2163 DW_OP_LLVM_extend 64, 64; 2164 DW_OP_call_ref %__lex_1_1_save_exec; 2165 DW_OP_deref_type 64, %__uint_64; 2166 DW_OP_LLVM_select_bit_piece 64, 64; 2167 ]; 2168 ]; 2169 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2170 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2171 DW_OP_call_ref %__active_lane_pc; 2172 ]; 2173 d; 2174 EXEC = %3; 2175 $lex_1_1_end: 2176 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2177 DW_OP_call_ref %__divergent_lane_pc; 2178 DW_OP_call_ref %__active_lane_pc; 2179 ]; 2180 e; 2181 EXEC = ~EXEC & %1; 2182 $lex_1_else: 2183 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2184 DW_AT_name = "__divergent_lane_pc_1_else"; 2185 DW_AT_location = DIExpression[ 2186 DW_OP_call_ref %__divergent_lane_pc; 2187 DW_OP_addrx &lex_1_end; 2188 DW_OP_stack_value; 2189 DW_OP_LLVM_extend 64, 64; 2190 DW_OP_call_ref %__lex_1_save_exec; 2191 DW_OP_deref_type 64, %__uint_64; 2192 DW_OP_LLVM_select_bit_piece 64, 64; 2193 ]; 2194 ]; 2195 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2196 DW_OP_call_ref %__divergent_lane_pc_1_else; 2197 DW_OP_call_ref %__active_lane_pc; 2198 ]; 2199 f; 2200 EXEC = %1; 2201 $lex_1_end: 2202 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2203 DW_OP_call_ref %__divergent_lane_pc; 2204 DW_OP_call_ref %__active_lane_pc; 2205 ]; 2206 g; 2207 $lex_end: 2208 2209The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2210that are active, with the current program location. 2211 2212Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2213the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2214instruction, location list entries will be created that describe where the 2215artificial variables are allocated at any given program location. The compiler 2216may allocate them to registers or spill them to memory. 2217 2218The DWARF procedures for each region use the values of the saved execution mask 2219artificial variables to only update the lanes that are active on entry to the 2220region. All other lanes retain the value of the enclosing region where they were 2221last active. If they were not active on entry to the subprogram, then will have 2222the undefined location description. 2223 2224Other structured control flow regions can be handled similarly. For example, 2225loops would set the divergent program location for the region at the end of the 2226loop. Any lanes active will be in the loop, and any lanes not active must have 2227exited the loop. 2228 2229An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2230``IF/THEN/ELSE`` regions. 2231 2232The DWARF procedures can use the active lane artificial variable described in 2233:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2234``EXEC`` mask in order to support whole or quad wavefront mode. 2235 2236.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2237 2238``DW_AT_LLVM_active_lane`` 2239~~~~~~~~~~~~~~~~~~~~~~~~~~ 2240 2241The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2242entry is used to specify the lanes that are conceptually active for a SIMT 2243thread. 2244 2245The execution mask may be modified to implement whole or quad wavefront mode 2246operations. For example, all lanes may need to temporarily be made active to 2247execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2248update it to enable the necessary lanes, perform the operations, and then 2249restore the ``EXEC`` mask from the saved value. While executing the whole 2250wavefront region, the conceptual execution mask is the saved value, not the 2251``EXEC`` value. 2252 2253This is handled by defining an artificial variable for the active lane mask. The 2254active lane mask artificial variable would be the actual ``EXEC`` mask for 2255normal regions, and the saved execution mask for regions where the mask is 2256temporarily updated. The location list expression created for this artificial 2257variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2258attribute. 2259 2260``DW_AT_LLVM_augmentation`` 2261~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2262 2263For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2264debugger information entry has the following value for the augmentation string: 2265 2266:: 2267 2268 [amdgpu:v0.0] 2269 2270The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2271extensions used in the DWARF of the compilation unit. The version number 2272conforms to [SEMVER]_. 2273 2274Call Frame Information 2275---------------------- 2276 2277DWARF Call Frame Information (CFI) describes how a consumer can virtually 2278*unwind* call frames in a running process or core dump. See DWARF Version 5 2279section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2280 2281For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2282 22831. ``augmentation`` string contains the following null-terminated UTF-8 string: 2284 2285 :: 2286 2287 [amd:v0.0] 2288 2289 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2290 extensions used in this CIE or to the FDEs that use it. The version number 2291 conforms to [SEMVER]_. 2292 22932. ``address_size`` for the ``Global`` address space is defined in 2294 :ref:`amdgpu-dwarf-address-space-identifier`. 2295 22963. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2297 22984. ``code_alignment_factor`` is 4 bytes. 2299 2300 .. TODO:: 2301 2302 Add to :ref:`amdgpu-processor-table` table. 2303 23045. ``data_alignment_factor`` is 4 bytes. 2305 2306 .. TODO:: 2307 2308 Add to :ref:`amdgpu-processor-table` table. 2309 23106. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2311 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2312 23137. ``initial_instructions`` Since a subprogram X with fewer registers can be 2314 called from subprogram Y that has more allocated, X will not change any of 2315 the extra registers as it cannot access them. Therefore, the default rule 2316 for all columns is ``same value``. 2317 2318For AMDGPU the register number follows the numbering defined in 2319:ref:`amdgpu-dwarf-register-identifier`. 2320 2321For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2322the return address to get the address of a byte within the call site 2323instructions. See DWARF Version 5 section 6.4.4. 2324 2325Accelerated Access 2326------------------ 2327 2328See DWARF Version 5 section 6.1. 2329 2330Lookup By Name Section Header 2331~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2332 2333See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2334 2335For AMDGPU the lookup by name section header table: 2336 2337``augmentation_string_size`` (uword) 2338 2339 Set to the length of the ``augmentation_string`` value which is always a 2340 multiple of 4. 2341 2342``augmentation_string`` (sequence of UTF-8 characters) 2343 2344 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2345 2346 :: 2347 2348 [amdgpu:v0.0] 2349 2350 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2351 extensions used in the DWARF of this index. The version number conforms to 2352 [SEMVER]_. 2353 2354 .. note:: 2355 2356 This is different to the DWARF Version 5 definition that requires the first 2357 4 characters to be the vendor ID. But this is consistent with the other 2358 augmentation strings and does allow multiple vendor contributions. However, 2359 backwards compatibility may be more desirable. 2360 2361Lookup By Address Section Header 2362~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2363 2364See DWARF Version 5 section 6.1.2. 2365 2366For AMDGPU the lookup by address section header table: 2367 2368``address_size`` (ubyte) 2369 2370 Match the address size for the ``Global`` address space defined in 2371 :ref:`amdgpu-dwarf-address-space-identifier`. 2372 2373``segment_selector_size`` (ubyte) 2374 2375 AMDGPU does not use a segment selector so this is 0. The entries in the 2376 ``.debug_aranges`` do not have a segment selector. 2377 2378Line Number Information 2379----------------------- 2380 2381See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2382 2383AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2384The instruction set must be obtained from the ELF file header ``e_flags`` field 2385in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2386<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2387 2388.. TODO:: 2389 2390 Should the ``isa`` state machine register be used to indicate if the code is 2391 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2392 2393For AMDGPU the line number program header fields have the following values (see 2394DWARF Version 5 section 6.2.4): 2395 2396``address_size`` (ubyte) 2397 Matches the address size for the ``Global`` address space defined in 2398 :ref:`amdgpu-dwarf-address-space-identifier`. 2399 2400``segment_selector_size`` (ubyte) 2401 AMDGPU does not use a segment selector so this is 0. 2402 2403``minimum_instruction_length`` (ubyte) 2404 For GFX9-GFX10 this is 4. 2405 2406``maximum_operations_per_instruction`` (ubyte) 2407 For GFX9-GFX10 this is 1. 2408 2409Source text for online-compiled programs (for example, those compiled by the 2410OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2411See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2412Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2413<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2414 2415The Clang option used to control source embedding in AMDGPU is defined in 2416:ref:`amdgpu-clang-debug-options-table`. 2417 2418 .. table:: AMDGPU Clang Debug Options 2419 :name: amdgpu-clang-debug-options-table 2420 2421 ==================== ================================================== 2422 Debug Flag Description 2423 ==================== ================================================== 2424 -g[no-]embed-source Enable/disable embedding source text in DWARF 2425 debug sections. Useful for environments where 2426 source cannot be written to disk, such as 2427 when performing online compilation. 2428 ==================== ================================================== 2429 2430For example: 2431 2432``-gembed-source`` 2433 Enable the embedded source. 2434 2435``-gno-embed-source`` 2436 Disable the embedded source. 2437 243832-Bit and 64-Bit DWARF Formats 2439------------------------------- 2440 2441See DWARF Version 5 section 7.4 and 2442:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2443 2444For AMDGPU: 2445 2446* For the ``amdgcn`` target architecture only the 64-bit process address space 2447 is supported. 2448 2449* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2450 the 32-bit DWARF format. 2451 2452Unit Headers 2453------------ 2454 2455For AMDGPU the following values apply for each of the unit headers described in 2456DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2457 2458``address_size`` (ubyte) 2459 Matches the address size for the ``Global`` address space defined in 2460 :ref:`amdgpu-dwarf-address-space-identifier`. 2461 2462.. _amdgpu-code-conventions: 2463 2464Code Conventions 2465================ 2466 2467This section provides code conventions used for each supported target triple OS 2468(see :ref:`amdgpu-target-triples`). 2469 2470AMDHSA 2471------ 2472 2473This section provides code conventions used when the target triple OS is 2474``amdhsa`` (see :ref:`amdgpu-target-triples`). 2475 2476.. _amdgpu-amdhsa-code-object-metadata: 2477 2478Code Object Metadata 2479~~~~~~~~~~~~~~~~~~~~ 2480 2481The code object metadata specifies extensible metadata associated with the code 2482objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2483encoding and semantics of this metadata depends on the code object version; see 2484:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2485:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2486:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2487 2488Code object metadata is specified in a note record (see 2489:ref:`amdgpu-note-records`) and is required when the target triple OS is 2490``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2491information necessary to support the HSA compatible runtime kernel queries. For 2492example, the segment sizes needed in a dispatch packet. In addition, a 2493high-level language runtime may require other information to be included. For 2494example, the AMD OpenCL runtime records kernel argument information. 2495 2496.. _amdgpu-amdhsa-code-object-metadata-v2: 2497 2498Code Object V2 Metadata 2499+++++++++++++++++++++++ 2500 2501.. warning:: 2502 Code object V2 is not the default code object version emitted by this version 2503 of LLVM. 2504 2505Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2506(see :ref:`amdgpu-note-records-v2`). 2507 2508The metadata is specified as a YAML formatted string (see [YAML]_ and 2509:doc:`YamlIO`). 2510 2511.. TODO:: 2512 2513 Is the string null terminated? It probably should not if YAML allows it to 2514 contain null characters, otherwise it should be. 2515 2516The metadata is represented as a single YAML document comprised of the mapping 2517defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2518referenced tables. 2519 2520For boolean values, the string values of ``false`` and ``true`` are used for 2521false and true respectively. 2522 2523Additional information can be added to the mappings. To avoid conflicts, any 2524non-AMD key names should be prefixed by "*vendor-name*.". 2525 2526 .. table:: AMDHSA Code Object V2 Metadata Map 2527 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2528 2529 ========== ============== ========= ======================================= 2530 String Key Value Type Required? Description 2531 ========== ============== ========= ======================================= 2532 "Version" sequence of Required - The first integer is the major 2533 2 integers version. Currently 1. 2534 - The second integer is the minor 2535 version. Currently 0. 2536 "Printf" sequence of Each string is encoded information 2537 strings about a printf function call. The 2538 encoded information is organized as 2539 fields separated by colon (':'): 2540 2541 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2542 2543 where: 2544 2545 ``ID`` 2546 A 32-bit integer as a unique id for 2547 each printf function call 2548 2549 ``N`` 2550 A 32-bit integer equal to the number 2551 of arguments of printf function call 2552 minus 1 2553 2554 ``S[i]`` (where i = 0, 1, ... , N-1) 2555 32-bit integers for the size in bytes 2556 of the i-th FormatString argument of 2557 the printf function call 2558 2559 FormatString 2560 The format string passed to the 2561 printf function call. 2562 "Kernels" sequence of Required Sequence of the mappings for each 2563 mapping kernel in the code object. See 2564 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2565 for the definition of the mapping. 2566 ========== ============== ========= ======================================= 2567 2568.. 2569 2570 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2571 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2572 2573 ================= ============== ========= ================================ 2574 String Key Value Type Required? Description 2575 ================= ============== ========= ================================ 2576 "Name" string Required Source name of the kernel. 2577 "SymbolName" string Required Name of the kernel 2578 descriptor ELF symbol. 2579 "Language" string Source language of the kernel. 2580 Values include: 2581 2582 - "OpenCL C" 2583 - "OpenCL C++" 2584 - "HCC" 2585 - "OpenMP" 2586 2587 "LanguageVersion" sequence of - The first integer is the major 2588 2 integers version. 2589 - The second integer is the 2590 minor version. 2591 "Attrs" mapping Mapping of kernel attributes. 2592 See 2593 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2594 for the mapping definition. 2595 "Args" sequence of Sequence of mappings of the 2596 mapping kernel arguments. See 2597 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2598 for the definition of the mapping. 2599 "CodeProps" mapping Mapping of properties related to 2600 the kernel code. See 2601 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2602 for the mapping definition. 2603 ================= ============== ========= ================================ 2604 2605.. 2606 2607 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2608 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2609 2610 =================== ============== ========= ============================== 2611 String Key Value Type Required? Description 2612 =================== ============== ========= ============================== 2613 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2614 3 integers must be >=1 and the dispatch 2615 work-group size X, Y, Z must 2616 correspond to the specified 2617 values. Defaults to 0, 0, 0. 2618 2619 Corresponds to the OpenCL 2620 ``reqd_work_group_size`` 2621 attribute. 2622 "WorkGroupSizeHint" sequence of The dispatch work-group size 2623 3 integers X, Y, Z is likely to be the 2624 specified values. 2625 2626 Corresponds to the OpenCL 2627 ``work_group_size_hint`` 2628 attribute. 2629 "VecTypeHint" string The name of a scalar or vector 2630 type. 2631 2632 Corresponds to the OpenCL 2633 ``vec_type_hint`` attribute. 2634 2635 "RuntimeHandle" string The external symbol name 2636 associated with a kernel. 2637 OpenCL runtime allocates a 2638 global buffer for the symbol 2639 and saves the kernel's address 2640 to it, which is used for 2641 device side enqueueing. Only 2642 available for device side 2643 enqueued kernels. 2644 =================== ============== ========= ============================== 2645 2646.. 2647 2648 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2649 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2650 2651 ================= ============== ========= ================================ 2652 String Key Value Type Required? Description 2653 ================= ============== ========= ================================ 2654 "Name" string Kernel argument name. 2655 "TypeName" string Kernel argument type name. 2656 "Size" integer Required Kernel argument size in bytes. 2657 "Align" integer Required Kernel argument alignment in 2658 bytes. Must be a power of two. 2659 "ValueKind" string Required Kernel argument kind that 2660 specifies how to set up the 2661 corresponding argument. 2662 Values include: 2663 2664 "ByValue" 2665 The argument is copied 2666 directly into the kernarg. 2667 2668 "GlobalBuffer" 2669 A global address space pointer 2670 to the buffer data is passed 2671 in the kernarg. 2672 2673 "DynamicSharedPointer" 2674 A group address space pointer 2675 to dynamically allocated LDS 2676 is passed in the kernarg. 2677 2678 "Sampler" 2679 A global address space 2680 pointer to a S# is passed in 2681 the kernarg. 2682 2683 "Image" 2684 A global address space 2685 pointer to a T# is passed in 2686 the kernarg. 2687 2688 "Pipe" 2689 A global address space pointer 2690 to an OpenCL pipe is passed in 2691 the kernarg. 2692 2693 "Queue" 2694 A global address space pointer 2695 to an OpenCL device enqueue 2696 queue is passed in the 2697 kernarg. 2698 2699 "HiddenGlobalOffsetX" 2700 The OpenCL grid dispatch 2701 global offset for the X 2702 dimension is passed in the 2703 kernarg. 2704 2705 "HiddenGlobalOffsetY" 2706 The OpenCL grid dispatch 2707 global offset for the Y 2708 dimension is passed in the 2709 kernarg. 2710 2711 "HiddenGlobalOffsetZ" 2712 The OpenCL grid dispatch 2713 global offset for the Z 2714 dimension is passed in the 2715 kernarg. 2716 2717 "HiddenNone" 2718 An argument that is not used 2719 by the kernel. Space needs to 2720 be left for it, but it does 2721 not need to be set up. 2722 2723 "HiddenPrintfBuffer" 2724 A global address space pointer 2725 to the runtime printf buffer 2726 is passed in kernarg. 2727 2728 "HiddenHostcallBuffer" 2729 A global address space pointer 2730 to the runtime hostcall buffer 2731 is passed in kernarg. 2732 2733 "HiddenDefaultQueue" 2734 A global address space pointer 2735 to the OpenCL device enqueue 2736 queue that should be used by 2737 the kernel by default is 2738 passed in the kernarg. 2739 2740 "HiddenCompletionAction" 2741 A global address space pointer 2742 to help link enqueued kernels into 2743 the ancestor tree for determining 2744 when the parent kernel has finished. 2745 2746 "HiddenMultiGridSyncArg" 2747 A global address space pointer for 2748 multi-grid synchronization is 2749 passed in the kernarg. 2750 2751 "ValueType" string Unused and deprecated. This should no longer 2752 be emitted, but is accepted for compatibility. 2753 2754 2755 "PointeeAlign" integer Alignment in bytes of pointee 2756 type for pointer type kernel 2757 argument. Must be a power 2758 of 2. Only present if 2759 "ValueKind" is 2760 "DynamicSharedPointer". 2761 "AddrSpaceQual" string Kernel argument address space 2762 qualifier. Only present if 2763 "ValueKind" is "GlobalBuffer" or 2764 "DynamicSharedPointer". Values 2765 are: 2766 2767 - "Private" 2768 - "Global" 2769 - "Constant" 2770 - "Local" 2771 - "Generic" 2772 - "Region" 2773 2774 .. TODO:: 2775 2776 Is GlobalBuffer only Global 2777 or Constant? Is 2778 DynamicSharedPointer always 2779 Local? Can HCC allow Generic? 2780 How can Private or Region 2781 ever happen? 2782 2783 "AccQual" string Kernel argument access 2784 qualifier. Only present if 2785 "ValueKind" is "Image" or 2786 "Pipe". Values 2787 are: 2788 2789 - "ReadOnly" 2790 - "WriteOnly" 2791 - "ReadWrite" 2792 2793 .. TODO:: 2794 2795 Does this apply to 2796 GlobalBuffer? 2797 2798 "ActualAccQual" string The actual memory accesses 2799 performed by the kernel on the 2800 kernel argument. Only present if 2801 "ValueKind" is "GlobalBuffer", 2802 "Image", or "Pipe". This may be 2803 more restrictive than indicated 2804 by "AccQual" to reflect what the 2805 kernel actual does. If not 2806 present then the runtime must 2807 assume what is implied by 2808 "AccQual" and "IsConst". Values 2809 are: 2810 2811 - "ReadOnly" 2812 - "WriteOnly" 2813 - "ReadWrite" 2814 2815 "IsConst" boolean Indicates if the kernel argument 2816 is const qualified. Only present 2817 if "ValueKind" is 2818 "GlobalBuffer". 2819 2820 "IsRestrict" boolean Indicates if the kernel argument 2821 is restrict qualified. Only 2822 present if "ValueKind" is 2823 "GlobalBuffer". 2824 2825 "IsVolatile" boolean Indicates if the kernel argument 2826 is volatile qualified. Only 2827 present if "ValueKind" is 2828 "GlobalBuffer". 2829 2830 "IsPipe" boolean Indicates if the kernel argument 2831 is pipe qualified. Only present 2832 if "ValueKind" is "Pipe". 2833 2834 .. TODO:: 2835 2836 Can GlobalBuffer be pipe 2837 qualified? 2838 2839 ================= ============== ========= ================================ 2840 2841.. 2842 2843 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2844 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2845 2846 ============================ ============== ========= ===================== 2847 String Key Value Type Required? Description 2848 ============================ ============== ========= ===================== 2849 "KernargSegmentSize" integer Required The size in bytes of 2850 the kernarg segment 2851 that holds the values 2852 of the arguments to 2853 the kernel. 2854 "GroupSegmentFixedSize" integer Required The amount of group 2855 segment memory 2856 required by a 2857 work-group in 2858 bytes. This does not 2859 include any 2860 dynamically allocated 2861 group segment memory 2862 that may be added 2863 when the kernel is 2864 dispatched. 2865 "PrivateSegmentFixedSize" integer Required The amount of fixed 2866 private address space 2867 memory required for a 2868 work-item in 2869 bytes. If the kernel 2870 uses a dynamic call 2871 stack then additional 2872 space must be added 2873 to this value for the 2874 call stack. 2875 "KernargSegmentAlign" integer Required The maximum byte 2876 alignment of 2877 arguments in the 2878 kernarg segment. Must 2879 be a power of 2. 2880 "WavefrontSize" integer Required Wavefront size. Must 2881 be a power of 2. 2882 "NumSGPRs" integer Required Number of scalar 2883 registers used by a 2884 wavefront for 2885 GFX6-GFX10. This 2886 includes the special 2887 SGPRs for VCC, Flat 2888 Scratch (GFX7-GFX10) 2889 and XNACK (for 2890 GFX8-GFX10). It does 2891 not include the 16 2892 SGPR added if a trap 2893 handler is 2894 enabled. It is not 2895 rounded up to the 2896 allocation 2897 granularity. 2898 "NumVGPRs" integer Required Number of vector 2899 registers used by 2900 each work-item for 2901 GFX6-GFX10 2902 "MaxFlatWorkGroupSize" integer Required Maximum flat 2903 work-group size 2904 supported by the 2905 kernel in work-items. 2906 Must be >=1 and 2907 consistent with 2908 ReqdWorkGroupSize if 2909 not 0, 0, 0. 2910 "NumSpilledSGPRs" integer Number of stores from 2911 a scalar register to 2912 a register allocator 2913 created spill 2914 location. 2915 "NumSpilledVGPRs" integer Number of stores from 2916 a vector register to 2917 a register allocator 2918 created spill 2919 location. 2920 ============================ ============== ========= ===================== 2921 2922.. _amdgpu-amdhsa-code-object-metadata-v3: 2923 2924Code Object V3 Metadata 2925+++++++++++++++++++++++ 2926 2927Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2928record (see :ref:`amdgpu-note-records-v3-v4`). 2929 2930The metadata is represented as Message Pack formatted binary data (see 2931[MsgPack]_). The top level is a Message Pack map that includes the 2932keys defined in table 2933:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2934tables. 2935 2936Additional information can be added to the maps. To avoid conflicts, 2937any key names should be prefixed by "*vendor-name*." where 2938``vendor-name`` can be the name of the vendor and specific vendor 2939tool that generates the information. The prefix is abbreviated to 2940simply "." when it appears within a map that has been added by the 2941same *vendor-name*. 2942 2943 .. table:: AMDHSA Code Object V3 Metadata Map 2944 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2945 2946 ================= ============== ========= ======================================= 2947 String Key Value Type Required? Description 2948 ================= ============== ========= ======================================= 2949 "amdhsa.version" sequence of Required - The first integer is the major 2950 2 integers version. Currently 1. 2951 - The second integer is the minor 2952 version. Currently 0. 2953 "amdhsa.printf" sequence of Each string is encoded information 2954 strings about a printf function call. The 2955 encoded information is organized as 2956 fields separated by colon (':'): 2957 2958 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2959 2960 where: 2961 2962 ``ID`` 2963 A 32-bit integer as a unique id for 2964 each printf function call 2965 2966 ``N`` 2967 A 32-bit integer equal to the number 2968 of arguments of printf function call 2969 minus 1 2970 2971 ``S[i]`` (where i = 0, 1, ... , N-1) 2972 32-bit integers for the size in bytes 2973 of the i-th FormatString argument of 2974 the printf function call 2975 2976 FormatString 2977 The format string passed to the 2978 printf function call. 2979 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2980 map kernel in the code object. See 2981 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2982 for the definition of the keys included 2983 in that map. 2984 ================= ============== ========= ======================================= 2985 2986.. 2987 2988 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2989 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2990 2991 =================================== ============== ========= ================================ 2992 String Key Value Type Required? Description 2993 =================================== ============== ========= ================================ 2994 ".name" string Required Source name of the kernel. 2995 ".symbol" string Required Name of the kernel 2996 descriptor ELF symbol. 2997 ".language" string Source language of the kernel. 2998 Values include: 2999 3000 - "OpenCL C" 3001 - "OpenCL C++" 3002 - "HCC" 3003 - "HIP" 3004 - "OpenMP" 3005 - "Assembler" 3006 3007 ".language_version" sequence of - The first integer is the major 3008 2 integers version. 3009 - The second integer is the 3010 minor version. 3011 ".args" sequence of Sequence of maps of the 3012 map kernel arguments. See 3013 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3014 for the definition of the keys 3015 included in that map. 3016 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3017 3 integers must be >=1 and the dispatch 3018 work-group size X, Y, Z must 3019 correspond to the specified 3020 values. Defaults to 0, 0, 0. 3021 3022 Corresponds to the OpenCL 3023 ``reqd_work_group_size`` 3024 attribute. 3025 ".workgroup_size_hint" sequence of The dispatch work-group size 3026 3 integers X, Y, Z is likely to be the 3027 specified values. 3028 3029 Corresponds to the OpenCL 3030 ``work_group_size_hint`` 3031 attribute. 3032 ".vec_type_hint" string The name of a scalar or vector 3033 type. 3034 3035 Corresponds to the OpenCL 3036 ``vec_type_hint`` attribute. 3037 3038 ".device_enqueue_symbol" string The external symbol name 3039 associated with a kernel. 3040 OpenCL runtime allocates a 3041 global buffer for the symbol 3042 and saves the kernel's address 3043 to it, which is used for 3044 device side enqueueing. Only 3045 available for device side 3046 enqueued kernels. 3047 ".kernarg_segment_size" integer Required The size in bytes of 3048 the kernarg segment 3049 that holds the values 3050 of the arguments to 3051 the kernel. 3052 ".group_segment_fixed_size" integer Required The amount of group 3053 segment memory 3054 required by a 3055 work-group in 3056 bytes. This does not 3057 include any 3058 dynamically allocated 3059 group segment memory 3060 that may be added 3061 when the kernel is 3062 dispatched. 3063 ".private_segment_fixed_size" integer Required The amount of fixed 3064 private address space 3065 memory required for a 3066 work-item in 3067 bytes. If the kernel 3068 uses a dynamic call 3069 stack then additional 3070 space must be added 3071 to this value for the 3072 call stack. 3073 ".kernarg_segment_align" integer Required The maximum byte 3074 alignment of 3075 arguments in the 3076 kernarg segment. Must 3077 be a power of 2. 3078 ".wavefront_size" integer Required Wavefront size. Must 3079 be a power of 2. 3080 ".sgpr_count" integer Required Number of scalar 3081 registers required by a 3082 wavefront for 3083 GFX6-GFX9. A register 3084 is required if it is 3085 used explicitly, or 3086 if a higher numbered 3087 register is used 3088 explicitly. This 3089 includes the special 3090 SGPRs for VCC, Flat 3091 Scratch (GFX7-GFX9) 3092 and XNACK (for 3093 GFX8-GFX9). It does 3094 not include the 16 3095 SGPR added if a trap 3096 handler is 3097 enabled. It is not 3098 rounded up to the 3099 allocation 3100 granularity. 3101 ".vgpr_count" integer Required Number of vector 3102 registers required by 3103 each work-item for 3104 GFX6-GFX9. A register 3105 is required if it is 3106 used explicitly, or 3107 if a higher numbered 3108 register is used 3109 explicitly. 3110 ".max_flat_workgroup_size" integer Required Maximum flat 3111 work-group size 3112 supported by the 3113 kernel in work-items. 3114 Must be >=1 and 3115 consistent with 3116 ReqdWorkGroupSize if 3117 not 0, 0, 0. 3118 ".sgpr_spill_count" integer Number of stores from 3119 a scalar register to 3120 a register allocator 3121 created spill 3122 location. 3123 ".vgpr_spill_count" integer Number of stores from 3124 a vector register to 3125 a register allocator 3126 created spill 3127 location. 3128 =================================== ============== ========= ================================ 3129 3130.. 3131 3132 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3133 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3134 3135 ====================== ============== ========= ================================ 3136 String Key Value Type Required? Description 3137 ====================== ============== ========= ================================ 3138 ".name" string Kernel argument name. 3139 ".type_name" string Kernel argument type name. 3140 ".size" integer Required Kernel argument size in bytes. 3141 ".offset" integer Required Kernel argument offset in 3142 bytes. The offset must be a 3143 multiple of the alignment 3144 required by the argument. 3145 ".value_kind" string Required Kernel argument kind that 3146 specifies how to set up the 3147 corresponding argument. 3148 Values include: 3149 3150 "by_value" 3151 The argument is copied 3152 directly into the kernarg. 3153 3154 "global_buffer" 3155 A global address space pointer 3156 to the buffer data is passed 3157 in the kernarg. 3158 3159 "dynamic_shared_pointer" 3160 A group address space pointer 3161 to dynamically allocated LDS 3162 is passed in the kernarg. 3163 3164 "sampler" 3165 A global address space 3166 pointer to a S# is passed in 3167 the kernarg. 3168 3169 "image" 3170 A global address space 3171 pointer to a T# is passed in 3172 the kernarg. 3173 3174 "pipe" 3175 A global address space pointer 3176 to an OpenCL pipe is passed in 3177 the kernarg. 3178 3179 "queue" 3180 A global address space pointer 3181 to an OpenCL device enqueue 3182 queue is passed in the 3183 kernarg. 3184 3185 "hidden_global_offset_x" 3186 The OpenCL grid dispatch 3187 global offset for the X 3188 dimension is passed in the 3189 kernarg. 3190 3191 "hidden_global_offset_y" 3192 The OpenCL grid dispatch 3193 global offset for the Y 3194 dimension is passed in the 3195 kernarg. 3196 3197 "hidden_global_offset_z" 3198 The OpenCL grid dispatch 3199 global offset for the Z 3200 dimension is passed in the 3201 kernarg. 3202 3203 "hidden_none" 3204 An argument that is not used 3205 by the kernel. Space needs to 3206 be left for it, but it does 3207 not need to be set up. 3208 3209 "hidden_printf_buffer" 3210 A global address space pointer 3211 to the runtime printf buffer 3212 is passed in kernarg. 3213 3214 "hidden_hostcall_buffer" 3215 A global address space pointer 3216 to the runtime hostcall buffer 3217 is passed in kernarg. 3218 3219 "hidden_default_queue" 3220 A global address space pointer 3221 to the OpenCL device enqueue 3222 queue that should be used by 3223 the kernel by default is 3224 passed in the kernarg. 3225 3226 "hidden_completion_action" 3227 A global address space pointer 3228 to help link enqueued kernels into 3229 the ancestor tree for determining 3230 when the parent kernel has finished. 3231 3232 "hidden_multigrid_sync_arg" 3233 A global address space pointer for 3234 multi-grid synchronization is 3235 passed in the kernarg. 3236 3237 ".value_type" string Unused and deprecated. This should no longer 3238 be emitted, but is accepted for compatibility. 3239 3240 ".pointee_align" integer Alignment in bytes of pointee 3241 type for pointer type kernel 3242 argument. Must be a power 3243 of 2. Only present if 3244 ".value_kind" is 3245 "dynamic_shared_pointer". 3246 ".address_space" string Kernel argument address space 3247 qualifier. Only present if 3248 ".value_kind" is "global_buffer" or 3249 "dynamic_shared_pointer". Values 3250 are: 3251 3252 - "private" 3253 - "global" 3254 - "constant" 3255 - "local" 3256 - "generic" 3257 - "region" 3258 3259 .. TODO:: 3260 3261 Is "global_buffer" only "global" 3262 or "constant"? Is 3263 "dynamic_shared_pointer" always 3264 "local"? Can HCC allow "generic"? 3265 How can "private" or "region" 3266 ever happen? 3267 3268 ".access" string Kernel argument access 3269 qualifier. Only present if 3270 ".value_kind" is "image" or 3271 "pipe". Values 3272 are: 3273 3274 - "read_only" 3275 - "write_only" 3276 - "read_write" 3277 3278 .. TODO:: 3279 3280 Does this apply to 3281 "global_buffer"? 3282 3283 ".actual_access" string The actual memory accesses 3284 performed by the kernel on the 3285 kernel argument. Only present if 3286 ".value_kind" is "global_buffer", 3287 "image", or "pipe". This may be 3288 more restrictive than indicated 3289 by ".access" to reflect what the 3290 kernel actual does. If not 3291 present then the runtime must 3292 assume what is implied by 3293 ".access" and ".is_const" . Values 3294 are: 3295 3296 - "read_only" 3297 - "write_only" 3298 - "read_write" 3299 3300 ".is_const" boolean Indicates if the kernel argument 3301 is const qualified. Only present 3302 if ".value_kind" is 3303 "global_buffer". 3304 3305 ".is_restrict" boolean Indicates if the kernel argument 3306 is restrict qualified. Only 3307 present if ".value_kind" is 3308 "global_buffer". 3309 3310 ".is_volatile" boolean Indicates if the kernel argument 3311 is volatile qualified. Only 3312 present if ".value_kind" is 3313 "global_buffer". 3314 3315 ".is_pipe" boolean Indicates if the kernel argument 3316 is pipe qualified. Only present 3317 if ".value_kind" is "pipe". 3318 3319 .. TODO:: 3320 3321 Can "global_buffer" be pipe 3322 qualified? 3323 3324 ====================== ============== ========= ================================ 3325 3326.. _amdgpu-amdhsa-code-object-metadata-v4: 3327 3328Code Object V4 Metadata 3329+++++++++++++++++++++++ 3330 3331.. warning:: 3332 Code object V4 is not the default code object version emitted by this version 3333 of LLVM. 3334 3335Code object V4 metadata is the same as 3336:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3337defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3338 3339 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3340 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3341 3342 ================= ============== ========= ======================================= 3343 String Key Value Type Required? Description 3344 ================= ============== ========= ======================================= 3345 "amdhsa.version" sequence of Required - The first integer is the major 3346 2 integers version. Currently 1. 3347 - The second integer is the minor 3348 version. Currently 1. 3349 "amdhsa.target" string Required The target name of the code using the syntax: 3350 3351 .. code:: 3352 3353 <target-triple> [ "-" <target-id> ] 3354 3355 A canonical target ID must be 3356 used. See :ref:`amdgpu-target-triples` 3357 and :ref:`amdgpu-target-id`. 3358 ================= ============== ========= ======================================= 3359 3360.. 3361 3362Kernel Dispatch 3363~~~~~~~~~~~~~~~ 3364 3365The HSA architected queuing language (AQL) defines a user space memory interface 3366that can be used to control the dispatch of kernels, in an agent independent 3367way. An agent can have zero or more AQL queues created for it using an HSA 3368compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3369are 64 bytes) can be placed. See the *HSA Platform System Architecture 3370Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3371 3372The packet processor of a kernel agent is responsible for detecting and 3373dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3374packet processor is implemented by the hardware command processor (CP), 3375asynchronous dispatch controller (ADC) and shader processor input controller 3376(SPI). 3377 3378An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3379the kernel mode driver to initialize and register the AQL queue with CP. 3380 3381To dispatch a kernel the following actions are performed. This can occur in the 3382CPU host program, or from an HSA kernel executing on a GPU. 3383 33841. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3385 executed is obtained. 33862. A pointer to the kernel descriptor (see 3387 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3388 It must be for a kernel that is contained in a code object that that was 3389 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3390 queue is associated. 33913. Space is allocated for the kernel arguments using the HSA compatible runtime 3392 allocator for a memory region with the kernarg property for the kernel agent 3393 that will execute the kernel. It must be at least 16-byte aligned. 33944. Kernel argument values are assigned to the kernel argument memory 3395 allocation. The layout is defined in the *HSA Programmer's Language 3396 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3397 kernel argument memory in the same way constant memory is accessed. (Note 3398 that the HSA specification allows an implementation to copy the kernel 3399 argument contents to another location that is accessed by the kernel.) 34005. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3401 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3402 for the packet. The packet must be set up, and the final write must use an 3403 atomic store release to set the packet kind to ensure the packet contents are 3404 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3405 notify the kernel agent that the AQL queue has been updated. These rules, and 3406 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3407 System Architecture Specification* [HSA]_. 34086. A kernel dispatch packet includes information about the actual dispatch, 3409 such as grid and work-group size, together with information from the code 3410 object about the kernel, such as segment sizes. The HSA compatible runtime 3411 queries on the kernel symbol can be used to obtain the code object values 3412 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 34137. CP executes micro-code and is responsible for detecting and setting up the 3414 GPU to execute the wavefronts of a kernel dispatch. 34158. CP ensures that when the a wavefront starts executing the kernel machine 3416 code, the scalar general purpose registers (SGPR) and vector general purpose 3417 registers (VGPR) are set up as required by the machine code. The required 3418 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3419 register state is defined in 3420 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 34219. The prolog of the kernel machine code (see 3422 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3423 before continuing executing the machine code that corresponds to the kernel. 342410. When the kernel dispatch has completed execution, CP signals the completion 3425 signal specified in the kernel dispatch packet if not 0. 3426 3427.. _amdgpu-amdhsa-memory-spaces: 3428 3429Memory Spaces 3430~~~~~~~~~~~~~ 3431 3432The memory space properties are: 3433 3434 .. table:: AMDHSA Memory Spaces 3435 :name: amdgpu-amdhsa-memory-spaces-table 3436 3437 ================= =========== ======== ======= ================== 3438 Memory Space Name HSA Segment Hardware Address NULL Value 3439 Name Name Size 3440 ================= =========== ======== ======= ================== 3441 Private private scratch 32 0x00000000 3442 Local group LDS 32 0xFFFFFFFF 3443 Global global global 64 0x0000000000000000 3444 Constant constant *same as 64 0x0000000000000000 3445 global* 3446 Generic flat flat 64 0x0000000000000000 3447 Region N/A GDS 32 *not implemented 3448 for AMDHSA* 3449 ================= =========== ======== ======= ================== 3450 3451The global and constant memory spaces both use global virtual addresses, which 3452are the same virtual address space used by the CPU. However, some virtual 3453addresses may only be accessible to the CPU, some only accessible by the GPU, 3454and some by both. 3455 3456Using the constant memory space indicates that the data will not change during 3457the execution of the kernel. This allows scalar read instructions to be 3458used. The vector and scalar L1 caches are invalidated of volatile data before 3459each kernel dispatch execution to allow constant memory to change values between 3460kernel dispatches. 3461 3462The local memory space uses the hardware Local Data Store (LDS) which is 3463automatically allocated when the hardware creates work-groups of wavefronts, and 3464freed when all the wavefronts of a work-group have terminated. The data store 3465(DS) instructions can be used to access it. 3466 3467The private memory space uses the hardware scratch memory support. If the kernel 3468uses scratch, then the hardware allocates memory that is accessed using 3469wavefront lane dword (4 byte) interleaving. The mapping used from private 3470address to physical address is: 3471 3472 ``wavefront-scratch-base + 3473 (private-address * wavefront-size * 4) + 3474 (wavefront-lane-id * 4)`` 3475 3476There are different ways that the wavefront scratch base address is determined 3477by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3478memory can be accessed in an interleaved manner using buffer instruction with 3479the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3480instructions, or by flat instructions. If each lane of a wavefront accesses the 3481same private address, the interleaving results in adjacent dwords being accessed 3482and hence requires fewer cache lines to be fetched. Multi-dword access is not 3483supported except by flat and scratch instructions in GFX9-GFX10. 3484 3485The generic address space uses the hardware flat address support available in 3486GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3487local apertures), that are outside the range of addressible global memory, to 3488map from a flat address to a private or local address. 3489 3490FLAT instructions can take a flat address and access global, private (scratch) 3491and group (LDS) memory depending in if the address is within one of the 3492aperture ranges. Flat access to scratch requires hardware aperture setup and 3493setup in the kernel prologue (see 3494:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3495hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3496:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3497 3498To convert between a segment address and a flat address the base address of the 3499apertures address can be used. For GFX7-GFX8 these are available in the 3500:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3501Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3502GFX9-GFX10 the aperture base addresses are directly available as inline constant 3503registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3504address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3505which makes it easier to convert from flat to segment or segment to flat. 3506 3507Image and Samplers 3508~~~~~~~~~~~~~~~~~~ 3509 3510Image and sample handles created by an HSA compatible runtime (see 3511:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3512object respectively. In order to support the HSA ``query_sampler`` operations 3513two extra dwords are used to store the HSA BRIG enumeration values for the 3514queries that are not trivially deducible from the S# representation. 3515 3516HSA Signals 3517~~~~~~~~~~~ 3518 3519HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3520are 64-bit addresses of a structure allocated in memory accessible from both the 3521CPU and GPU. The structure is defined by the runtime and subject to change 3522between releases. For example, see [AMD-ROCm-github]_. 3523 3524.. _amdgpu-amdhsa-hsa-aql-queue: 3525 3526HSA AQL Queue 3527~~~~~~~~~~~~~ 3528 3529The HSA AQL queue structure is defined by an HSA compatible runtime (see 3530:ref:`amdgpu-os`) and subject to change between releases. For example, see 3531[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3532certain language features such as the flat address aperture bases. It also 3533contains fields used by CP such as managing the allocation of scratch memory. 3534 3535.. _amdgpu-amdhsa-kernel-descriptor: 3536 3537Kernel Descriptor 3538~~~~~~~~~~~~~~~~~ 3539 3540A kernel descriptor consists of the information needed by CP to initiate the 3541execution of a kernel, including the entry point address of the machine code 3542that implements the kernel. 3543 3544Code Object V3 Kernel Descriptor 3545++++++++++++++++++++++++++++++++ 3546 3547CP microcode requires the Kernel descriptor to be allocated on 64-byte 3548alignment. 3549 3550The fields used by CP for code objects before V3 also match those specified in 3551:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3552 3553 .. table:: Code Object V3 Kernel Descriptor 3554 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3555 3556 ======= ======= =============================== ============================ 3557 Bits Size Field Name Description 3558 ======= ======= =============================== ============================ 3559 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3560 address space memory 3561 required for a work-group 3562 in bytes. This does not 3563 include any dynamically 3564 allocated local address 3565 space memory that may be 3566 added when the kernel is 3567 dispatched. 3568 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3569 private address space 3570 memory required for a 3571 work-item in bytes. 3572 Additional space may need to 3573 be added to this value if 3574 the call stack has 3575 non-inlined function calls. 3576 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3577 memory pointed to by the 3578 AQL dispatch packet. The 3579 kernarg memory is used to 3580 pass arguments to the 3581 kernel. 3582 3583 * If the kernarg pointer in 3584 the dispatch packet is NULL 3585 then there are no kernel 3586 arguments. 3587 * If the kernarg pointer in 3588 the dispatch packet is 3589 not NULL and this value 3590 is 0 then the kernarg 3591 memory size is 3592 unspecified. 3593 * If the kernarg pointer in 3594 the dispatch packet is 3595 not NULL and this value 3596 is not 0 then the value 3597 specifies the kernarg 3598 memory size in bytes. It 3599 is recommended to provide 3600 a value as it may be used 3601 by CP to optimize making 3602 the kernarg memory 3603 visible to the kernel 3604 code. 3605 3606 127:96 4 bytes Reserved, must be 0. 3607 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3608 negative) from base 3609 address of kernel 3610 descriptor to kernel's 3611 entry point instruction 3612 which must be 256 byte 3613 aligned. 3614 351:272 20 Reserved, must be 0. 3615 bytes 3616 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3617 Reserved, must be 0. 3618 GFX90A 3619 Compute Shader (CS) 3620 program settings used by 3621 CP to set up 3622 ``COMPUTE_PGM_RSRC3`` 3623 configuration 3624 register. See 3625 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3626 GFX10 3627 Compute Shader (CS) 3628 program settings used by 3629 CP to set up 3630 ``COMPUTE_PGM_RSRC3`` 3631 configuration 3632 register. See 3633 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3634 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3635 program settings used by 3636 CP to set up 3637 ``COMPUTE_PGM_RSRC1`` 3638 configuration 3639 register. See 3640 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3641 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3642 program settings used by 3643 CP to set up 3644 ``COMPUTE_PGM_RSRC2`` 3645 configuration 3646 register. See 3647 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3648 458:448 7 bits *See separate bits below.* Enable the setup of the 3649 SGPR user data registers 3650 (see 3651 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3652 3653 The total number of SGPR 3654 user data registers 3655 requested must not exceed 3656 16 and match value in 3657 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3658 Any requests beyond 16 3659 will be ignored. 3660 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3661 _BUFFER 3662 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3663 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3664 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3665 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3666 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT 3667 3668 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3669 _SIZE 3670 457:455 3 bits Reserved, must be 0. 3671 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3672 Reserved, must be 0. 3673 GFX10 3674 - If 0 execute in 3675 wavefront size 64 mode. 3676 - If 1 execute in 3677 native wavefront size 3678 32 mode. 3679 463:459 1 bit Reserved, must be 0. 3680 464 1 bit RESERVED_464 Deprecated, must be 0. 3681 467:465 3 bits Reserved, must be 0. 3682 468 1 bit RESERVED_468 Deprecated, must be 0. 3683 469:471 3 bits Reserved, must be 0. 3684 511:472 5 bytes Reserved, must be 0. 3685 512 **Total size 64 bytes.** 3686 ======= ==================================================================== 3687 3688.. 3689 3690 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3691 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3692 3693 ======= ======= =============================== =========================================================================== 3694 Bits Size Field Name Description 3695 ======= ======= =============================== =========================================================================== 3696 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3697 blocks used by each work-item; 3698 granularity is device 3699 specific: 3700 3701 GFX6-GFX9 3702 - vgprs_used 0..256 3703 - max(0, ceil(vgprs_used / 4) - 1) 3704 GFX90A 3705 - vgprs_used 0..512 3706 - vgprs_used = align(arch_vgprs, 4) 3707 + acc_vgprs 3708 - max(0, ceil(vgprs_used / 8) - 1) 3709 GFX10 (wavefront size 64) 3710 - max_vgpr 1..256 3711 - max(0, ceil(vgprs_used / 4) - 1) 3712 GFX10 (wavefront size 32) 3713 - max_vgpr 1..256 3714 - max(0, ceil(vgprs_used / 8) - 1) 3715 3716 Where vgprs_used is defined 3717 as the highest VGPR number 3718 explicitly referenced plus 3719 one. 3720 3721 Used by CP to set up 3722 ``COMPUTE_PGM_RSRC1.VGPRS``. 3723 3724 The 3725 :ref:`amdgpu-assembler` 3726 calculates this 3727 automatically for the 3728 selected processor from 3729 values provided to the 3730 `.amdhsa_kernel` directive 3731 by the 3732 `.amdhsa_next_free_vgpr` 3733 nested directive (see 3734 :ref:`amdhsa-kernel-directives-table`). 3735 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3736 blocks used by a wavefront; 3737 granularity is device 3738 specific: 3739 3740 GFX6-GFX8 3741 - sgprs_used 0..112 3742 - max(0, ceil(sgprs_used / 8) - 1) 3743 GFX9 3744 - sgprs_used 0..112 3745 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3746 GFX10 3747 Reserved, must be 0. 3748 (128 SGPRs always 3749 allocated.) 3750 3751 Where sgprs_used is 3752 defined as the highest 3753 SGPR number explicitly 3754 referenced plus one, plus 3755 a target specific number 3756 of additional special 3757 SGPRs for VCC, 3758 FLAT_SCRATCH (GFX7+) and 3759 XNACK_MASK (GFX8+), and 3760 any additional 3761 target specific 3762 limitations. It does not 3763 include the 16 SGPRs added 3764 if a trap handler is 3765 enabled. 3766 3767 The target specific 3768 limitations and special 3769 SGPR layout are defined in 3770 the hardware 3771 documentation, which can 3772 be found in the 3773 :ref:`amdgpu-processors` 3774 table. 3775 3776 Used by CP to set up 3777 ``COMPUTE_PGM_RSRC1.SGPRS``. 3778 3779 The 3780 :ref:`amdgpu-assembler` 3781 calculates this 3782 automatically for the 3783 selected processor from 3784 values provided to the 3785 `.amdhsa_kernel` directive 3786 by the 3787 `.amdhsa_next_free_sgpr` 3788 and `.amdhsa_reserve_*` 3789 nested directives (see 3790 :ref:`amdhsa-kernel-directives-table`). 3791 11:10 2 bits PRIORITY Must be 0. 3792 3793 Start executing wavefront 3794 at the specified priority. 3795 3796 CP is responsible for 3797 filling in 3798 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3799 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3800 with specified rounding 3801 mode for single (32 3802 bit) floating point 3803 precision floating point 3804 operations. 3805 3806 Floating point rounding 3807 mode values are defined in 3808 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3809 3810 Used by CP to set up 3811 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3812 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3813 with specified rounding 3814 denorm mode for half/double (16 3815 and 64-bit) floating point 3816 precision floating point 3817 operations. 3818 3819 Floating point rounding 3820 mode values are defined in 3821 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3822 3823 Used by CP to set up 3824 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3825 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3826 with specified denorm mode 3827 for single (32 3828 bit) floating point 3829 precision floating point 3830 operations. 3831 3832 Floating point denorm mode 3833 values are defined in 3834 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3835 3836 Used by CP to set up 3837 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3838 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3839 with specified denorm mode 3840 for half/double (16 3841 and 64-bit) floating point 3842 precision floating point 3843 operations. 3844 3845 Floating point denorm mode 3846 values are defined in 3847 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3848 3849 Used by CP to set up 3850 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3851 20 1 bit PRIV Must be 0. 3852 3853 Start executing wavefront 3854 in privilege trap handler 3855 mode. 3856 3857 CP is responsible for 3858 filling in 3859 ``COMPUTE_PGM_RSRC1.PRIV``. 3860 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3861 with DX10 clamp mode 3862 enabled. Used by the vector 3863 ALU to force DX10 style 3864 treatment of NaN's (when 3865 set, clamp NaN to zero, 3866 otherwise pass NaN 3867 through). 3868 3869 Used by CP to set up 3870 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3871 22 1 bit DEBUG_MODE Must be 0. 3872 3873 Start executing wavefront 3874 in single step mode. 3875 3876 CP is responsible for 3877 filling in 3878 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3879 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3880 with IEEE mode 3881 enabled. Floating point 3882 opcodes that support 3883 exception flag gathering 3884 will quiet and propagate 3885 signaling-NaN inputs per 3886 IEEE 754-2008. Min_dx10 and 3887 max_dx10 become IEEE 3888 754-2008 compliant due to 3889 signaling-NaN propagation 3890 and quieting. 3891 3892 Used by CP to set up 3893 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3894 24 1 bit BULKY Must be 0. 3895 3896 Only one work-group allowed 3897 to execute on a compute 3898 unit. 3899 3900 CP is responsible for 3901 filling in 3902 ``COMPUTE_PGM_RSRC1.BULKY``. 3903 25 1 bit CDBG_USER Must be 0. 3904 3905 Flag that can be used to 3906 control debugging code. 3907 3908 CP is responsible for 3909 filling in 3910 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3911 26 1 bit FP16_OVFL GFX6-GFX8 3912 Reserved, must be 0. 3913 GFX9-GFX10 3914 Wavefront starts execution 3915 with specified fp16 overflow 3916 mode. 3917 3918 - If 0, fp16 overflow generates 3919 +/-INF values. 3920 - If 1, fp16 overflow that is the 3921 result of an +/-INF input value 3922 or divide by 0 produces a +/-INF, 3923 otherwise clamps computed 3924 overflow to +/-MAX_FP16 as 3925 appropriate. 3926 3927 Used by CP to set up 3928 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3929 28:27 2 bits Reserved, must be 0. 3930 29 1 bit WGP_MODE GFX6-GFX9 3931 Reserved, must be 0. 3932 GFX10 3933 - If 0 execute work-groups in 3934 CU wavefront execution mode. 3935 - If 1 execute work-groups on 3936 in WGP wavefront execution mode. 3937 3938 See :ref:`amdgpu-amdhsa-memory-model`. 3939 3940 Used by CP to set up 3941 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3942 30 1 bit MEM_ORDERED GFX6-GFX9 3943 Reserved, must be 0. 3944 GFX10 3945 Controls the behavior of the 3946 s_waitcnt's vmcnt and vscnt 3947 counters. 3948 3949 - If 0 vmcnt reports completion 3950 of load and atomic with return 3951 out of order with sample 3952 instructions, and the vscnt 3953 reports the completion of 3954 store and atomic without 3955 return in order. 3956 - If 1 vmcnt reports completion 3957 of load, atomic with return 3958 and sample instructions in 3959 order, and the vscnt reports 3960 the completion of store and 3961 atomic without return in order. 3962 3963 Used by CP to set up 3964 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3965 31 1 bit FWD_PROGRESS GFX6-GFX9 3966 Reserved, must be 0. 3967 GFX10 3968 - If 0 execute SIMD wavefronts 3969 using oldest first policy. 3970 - If 1 execute SIMD wavefronts to 3971 ensure wavefronts will make some 3972 forward progress. 3973 3974 Used by CP to set up 3975 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3976 32 **Total size 4 bytes** 3977 ======= =================================================================================================================== 3978 3979.. 3980 3981 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3982 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3983 3984 ======= ======= =============================== =========================================================================== 3985 Bits Size Field Name Description 3986 ======= ======= =============================== =========================================================================== 3987 0 1 bit ENABLE_PRIVATE_SEGMENT Enable the setup of the 3988 private segment. 3989 3990 In addition, enable the 3991 setup of the SGPR 3992 wavefront scratch offset 3993 system register (see 3994 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3995 3996 Used by CP to set up 3997 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3998 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3999 user data registers 4000 requested. This number must 4001 match the number of user 4002 data registers enabled. 4003 4004 Used by CP to set up 4005 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4006 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4007 4008 This bit represents 4009 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4010 which is set by the CP if 4011 the runtime has installed a 4012 trap handler. 4013 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4014 system SGPR register for 4015 the work-group id in the X 4016 dimension (see 4017 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4018 4019 Used by CP to set up 4020 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4021 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4022 system SGPR register for 4023 the work-group id in the Y 4024 dimension (see 4025 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4026 4027 Used by CP to set up 4028 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4029 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4030 system SGPR register for 4031 the work-group id in the Z 4032 dimension (see 4033 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4034 4035 Used by CP to set up 4036 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4037 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4038 system SGPR register for 4039 work-group information (see 4040 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4041 4042 Used by CP to set up 4043 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4044 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4045 VGPR system registers used 4046 for the work-item ID. 4047 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4048 defines the values. 4049 4050 Used by CP to set up 4051 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4052 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4053 4054 Wavefront starts execution 4055 with address watch 4056 exceptions enabled which 4057 are generated when L1 has 4058 witnessed a thread access 4059 an *address of 4060 interest*. 4061 4062 CP is responsible for 4063 filling in the address 4064 watch bit in 4065 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4066 according to what the 4067 runtime requests. 4068 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4069 4070 Wavefront starts execution 4071 with memory violation 4072 exceptions exceptions 4073 enabled which are generated 4074 when a memory violation has 4075 occurred for this wavefront from 4076 L1 or LDS 4077 (write-to-read-only-memory, 4078 mis-aligned atomic, LDS 4079 address out of range, 4080 illegal address, etc.). 4081 4082 CP sets the memory 4083 violation bit in 4084 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4085 according to what the 4086 runtime requests. 4087 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4088 4089 CP uses the rounded value 4090 from the dispatch packet, 4091 not this value, as the 4092 dispatch may contain 4093 dynamically allocated group 4094 segment memory. CP writes 4095 directly to 4096 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4097 4098 Amount of group segment 4099 (LDS) to allocate for each 4100 work-group. Granularity is 4101 device specific: 4102 4103 GFX6 4104 roundup(lds-size / (64 * 4)) 4105 GFX7-GFX10 4106 roundup(lds-size / (128 * 4)) 4107 4108 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4109 _INVALID_OPERATION with specified exceptions 4110 enabled. 4111 4112 Used by CP to set up 4113 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4114 (set from bits 0..6). 4115 4116 IEEE 754 FP Invalid 4117 Operation 4118 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4119 _SOURCE input operands is a 4120 denormal number 4121 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4122 _DIVISION_BY_ZERO Zero 4123 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4124 _OVERFLOW 4125 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4126 _UNDERFLOW 4127 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4128 _INEXACT 4129 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4130 _ZERO (rcp_iflag_f32 instruction 4131 only) 4132 31 1 bit Reserved, must be 0. 4133 32 **Total size 4 bytes.** 4134 ======= =================================================================================================================== 4135 4136.. 4137 4138 .. table:: compute_pgm_rsrc3 for GFX90A 4139 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4140 4141 ======= ======= =============================== =========================================================================== 4142 Bits Size Field Name Description 4143 ======= ======= =============================== =========================================================================== 4144 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4145 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4146 63 - accum-offset = 256. 4147 6:15 10 Reserved, must be 0. 4148 bits 4149 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4150 launched in the same CU. 4151 - If 1 the waves of a work-group can be 4152 launched in different CUs. The waves 4153 cannot use S_BARRIER or LDS. 4154 17:31 15 Reserved, must be 0. 4155 bits 4156 32 **Total size 4 bytes.** 4157 ======= =================================================================================================================== 4158 4159.. 4160 4161 .. table:: compute_pgm_rsrc3 for GFX10 4162 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4163 4164 ======= ======= =============================== =========================================================================== 4165 Bits Size Field Name Description 4166 ======= ======= =============================== =========================================================================== 4167 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4168 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4169 31:4 28 Reserved, must be 0. 4170 bits 4171 32 **Total size 4 bytes.** 4172 ======= =================================================================================================================== 4173 4174.. 4175 4176 .. table:: Floating Point Rounding Mode Enumeration Values 4177 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4178 4179 ====================================== ===== ============================== 4180 Enumeration Name Value Description 4181 ====================================== ===== ============================== 4182 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4183 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4184 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4185 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4186 ====================================== ===== ============================== 4187 4188.. 4189 4190 .. table:: Floating Point Denorm Mode Enumeration Values 4191 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4192 4193 ====================================== ===== ============================== 4194 Enumeration Name Value Description 4195 ====================================== ===== ============================== 4196 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4197 Denorms 4198 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4199 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4200 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4201 ====================================== ===== ============================== 4202 4203.. 4204 4205 .. table:: System VGPR Work-Item ID Enumeration Values 4206 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4207 4208 ======================================== ===== ============================ 4209 Enumeration Name Value Description 4210 ======================================== ===== ============================ 4211 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4212 ID. 4213 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4214 dimensions ID. 4215 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4216 dimensions ID. 4217 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4218 ======================================== ===== ============================ 4219 4220.. _amdgpu-amdhsa-initial-kernel-execution-state: 4221 4222Initial Kernel Execution State 4223~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4224 4225This section defines the register state that will be set up by the packet 4226processor prior to the start of execution of every wavefront. This is limited by 4227the constraints of the hardware controllers of CP/ADC/SPI. 4228 4229The order of the SGPR registers is defined, but the compiler can specify which 4230ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4231fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4232for enabled registers are dense starting at SGPR0: the first enabled register is 4233SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4234an SGPR number. 4235 4236The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4237all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4238using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4239actually initialized. These are then immediately followed by the System SGPRs 4240that are set up by ADC/SPI and can have different values for each wavefront of 4241the grid dispatch. 4242 4243SGPR register initial state is defined in 4244:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4245 4246 .. table:: SGPR Register Set Up Order 4247 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4248 4249 ========== ========================== ====== ============================== 4250 SGPR Order Name Number Description 4251 (kernel descriptor enable of 4252 field) SGPRs 4253 ========== ========================== ====== ============================== 4254 First Private Segment Buffer 4 See 4255 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4256 _segment_buffer) 4257 then Dispatch Ptr 2 64-bit address of AQL dispatch 4258 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4259 actually executing. 4260 then Queue Ptr 2 64-bit address of amd_queue_t 4261 (enable_sgpr_queue_ptr) object for AQL queue on which 4262 the dispatch packet was 4263 queued. 4264 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4265 (enable_sgpr_kernarg segment. This is directly 4266 _segment_ptr) copied from the 4267 kernarg_address in the kernel 4268 dispatch packet. 4269 4270 Having CP load it once avoids 4271 loading it at the beginning of 4272 every wavefront. 4273 then Dispatch Id 2 64-bit Dispatch ID of the 4274 (enable_sgpr_dispatch_id) dispatch packet being 4275 executed. 4276 then Flat Scratch Init 2 See 4277 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4278 _init) 4279 then Private Segment Size 1 The 32-bit byte size of a 4280 (enable_sgpr_private single work-item's memory 4281 _segment_size) allocation. This is the 4282 value from the kernel 4283 dispatch packet Private 4284 Segment Byte Size rounded up 4285 by CP to a multiple of 4286 DWORD. 4287 4288 Having CP load it once avoids 4289 loading it at the beginning of 4290 every wavefront. 4291 4292 This is not used for 4293 GFX7-GFX8 since it is the same 4294 value as the second SGPR of 4295 Flat Scratch Init. However, it 4296 may be needed for GFX9-GFX10 which 4297 changes the meaning of the 4298 Flat Scratch Init value. 4299 then Work-Group Id X 1 32-bit work-group id in X 4300 (enable_sgpr_workgroup_id dimension of grid for 4301 _X) wavefront. 4302 then Work-Group Id Y 1 32-bit work-group id in Y 4303 (enable_sgpr_workgroup_id dimension of grid for 4304 _Y) wavefront. 4305 then Work-Group Id Z 1 32-bit work-group id in Z 4306 (enable_sgpr_workgroup_id dimension of grid for 4307 _Z) wavefront. 4308 then Work-Group Info 1 {first_wavefront, 14'b0000, 4309 (enable_sgpr_workgroup ordered_append_term[10:0], 4310 _info) threadgroup_size_in_wavefronts[5:0]} 4311 then Scratch Wavefront Offset 1 See 4312 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4313 _segment_wavefront_offset) and 4314 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4315 ========== ========================== ====== ============================== 4316 4317The order of the VGPR registers is defined, but the compiler can specify which 4318ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4319fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4320for enabled registers are dense starting at VGPR0: the first enabled register is 4321VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4322VGPR number. 4323 4324There are different methods used for the VGPR initial state: 4325 4326* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4327 specifies otherwise, a separate VGPR register is used per work-item ID. The 4328 VGPR register initial state for this method is defined in 4329 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4330* If *Target Properties* column of :ref:`amdgpu-processor-table` 4331 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4332 for all work-item IDs. The register layout for this method is defined in 4333 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4334 4335 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4336 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4337 4338 ========== ========================== ====== ============================== 4339 VGPR Order Name Number Description 4340 (kernel descriptor enable of 4341 field) VGPRs 4342 ========== ========================== ====== ============================== 4343 First Work-Item Id X 1 32-bit work-item id in X 4344 (Always initialized) dimension of work-group for 4345 wavefront lane. 4346 then Work-Item Id Y 1 32-bit work-item id in Y 4347 (enable_vgpr_workitem_id dimension of work-group for 4348 > 0) wavefront lane. 4349 then Work-Item Id Z 1 32-bit work-item id in Z 4350 (enable_vgpr_workitem_id dimension of work-group for 4351 > 1) wavefront lane. 4352 ========== ========================== ====== ============================== 4353 4354.. 4355 4356 .. table:: Register Layout for Packed Work-Item ID Method 4357 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4358 4359 ======= ======= ================ ========================================= 4360 Bits Size Field Name Description 4361 ======= ======= ================ ========================================= 4362 0:9 10 bits Work-Item Id X Work-item id in X 4363 dimension of work-group for 4364 wavefront lane. 4365 4366 Always initialized. 4367 4368 10:19 10 bits Work-Item Id Y Work-item id in Y 4369 dimension of work-group for 4370 wavefront lane. 4371 4372 Initialized if enable_vgpr_workitem_id > 4373 0, otherwise set to 0. 4374 20:29 10 bits Work-Item Id Z Work-item id in Z 4375 dimension of work-group for 4376 wavefront lane. 4377 4378 Initialized if enable_vgpr_workitem_id > 4379 1, otherwise set to 0. 4380 30:31 2 bits Reserved, set to 0. 4381 ======= ======= ================ ========================================= 4382 4383The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4384 43851. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4386 registers. 43872. Work-group Id registers X, Y, Z are set by ADC which supports any 4388 combination including none. 43893. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4390 its value cannot be included with the flat scratch init value which is per 4391 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 43924. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4393 or (X, Y, Z). 43945. Flat Scratch register pair initialization is described in 4395 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4396 4397The global segment can be accessed either using buffer instructions (GFX6 which 4398has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4399instructions (GFX9-GFX10). 4400 4401If buffer operations are used, then the compiler can generate a V# with the 4402following properties: 4403 4404* base address of 0 4405* no swizzle 4406* ATC: 1 if IOMMU present (such as APU) 4407* ptr64: 1 4408* MTYPE set to support memory coherence that matches the runtime (such as CC for 4409 APU and NC for dGPU). 4410 4411.. _amdgpu-amdhsa-kernel-prolog: 4412 4413Kernel Prolog 4414~~~~~~~~~~~~~ 4415 4416The compiler performs initialization in the kernel prologue depending on the 4417target and information about things like stack usage in the kernel and called 4418functions. Some of this initialization requires the compiler to request certain 4419User and System SGPRs be present in the 4420:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4421:ref:`amdgpu-amdhsa-kernel-descriptor`. 4422 4423.. _amdgpu-amdhsa-kernel-prolog-cfi: 4424 4425CFI 4426+++ 4427 44281. The CFI return address is undefined. 4429 44302. The CFI CFA is defined using an expression which evaluates to a location 4431 description that comprises one memory location description for the 4432 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4433 4434.. _amdgpu-amdhsa-kernel-prolog-m0: 4435 4436M0 4437++ 4438 4439GFX6-GFX8 4440 The M0 register must be initialized with a value at least the total LDS size 4441 if the kernel may access LDS via DS or flat operations. Total LDS size is 4442 available in dispatch packet. For M0, it is also possible to use maximum 4443 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4444 GFX7-GFX8). 4445GFX9-GFX10 4446 The M0 register is not used for range checking LDS accesses and so does not 4447 need to be initialized in the prolog. 4448 4449.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4450 4451Stack Pointer 4452+++++++++++++ 4453 4454If the kernel has function calls it must set up the ABI stack pointer described 4455in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4456SGPR32 to the unswizzled scratch offset of the address past the last local 4457allocation. 4458 4459.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4460 4461Frame Pointer 4462+++++++++++++ 4463 4464If the kernel needs a frame pointer for the reasons defined in 4465``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4466kernel prolog. If a frame pointer is not required then all uses of the frame 4467pointer are replaced with immediate ``0`` offsets. 4468 4469.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4470 4471Flat Scratch 4472++++++++++++ 4473 4474There are different methods used for initializing flat scratch: 4475 4476* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4477 specifies *Does not support generic address space*: 4478 4479 Flat scratch is not supported and there is no flat scratch register pair. 4480 4481* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4482 specifies *Offset flat scratch*: 4483 4484 If the kernel or any function it calls may use flat operations to access 4485 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4486 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4487 Scratch Wavefront Offset SGPR registers (see 4488 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4489 4490 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4491 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4492 being managed by SPI for the queue executing the kernel dispatch. This is 4493 the same value used in the Scratch Segment Buffer V# base address. 4494 4495 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4496 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4497 4498 The prolog must add the value of Scratch Wavefront Offset to get the 4499 wavefront's byte scratch backing memory offset from 4500 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4501 4502 The Scratch Wavefront Offset must also be used as an offset with Private 4503 segment address when using the Scratch Segment Buffer. 4504 4505 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4506 shifted by 8 before moving into FLAT_SCRATCH_HI. 4507 4508 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4509 SGPRn is the highest numbered SGPR allocated to the wavefront). 4510 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4511 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4512 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4513 aperture. 4514 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4515 work-items scratch memory usage. 4516 4517 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4518 checks that the value in the kernel dispatch packet Private Segment Byte 4519 Size is not larger and requests the runtime to increase the queue's scratch 4520 size if necessary. 4521 4522 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4523 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4524 loading it at the beginning of every wavefront. 4525 4526 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4527 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4528 in flat memory instructions. 4529 4530* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4531 specifies *Absolute flat scratch*: 4532 4533 If the kernel or any function it calls may use flat operations to access 4534 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4535 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4536 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4537 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4538 4539 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4540 memory being managed by SPI for the queue executing the kernel dispatch. 4541 4542 CP obtains this from the runtime. 4543 4544 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4545 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4546 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4547 memory instructions. 4548 4549 The Scratch Wavefront Offset must also be used as an offset with Private 4550 segment address when using the Scratch Segment Buffer (see 4551 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4552 4553.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4554 4555Private Segment Buffer 4556++++++++++++++++++++++ 4557 4558Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4559that are used as a V# to access scratch. CP uses the value provided by the 4560runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4561access the private memory space using a segment address. See 4562:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4563 4564The scratch V# is a four-aligned SGPR and always selected for the kernel as 4565follows: 4566 4567 - If it is known during instruction selection that there is stack usage, 4568 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4569 optimizations are disabled (``-O0``), if stack objects already exist (for 4570 locals, etc.), or if there are any function calls. 4571 4572 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4573 are reserved for the tentative scratch V#. These will be used if it is 4574 determined that spilling is needed. 4575 4576 - If no use is made of the tentative scratch V#, then it is unreserved, 4577 and the register count is determined ignoring it. 4578 - If use is made of the tentative scratch V#, then its register numbers 4579 are shifted to the first four-aligned SGPR index after the highest one 4580 allocated by the register allocator, and all uses are updated. The 4581 register count includes them in the shifted location. 4582 - In either case, if the processor has the SGPR allocation bug, the 4583 tentative allocation is not shifted or unreserved in order to ensure 4584 the register count is higher to workaround the bug. 4585 4586 .. note:: 4587 4588 This approach of using a tentative scratch V# and shifting the register 4589 numbers if used avoids having to perform register allocation a second 4590 time if the tentative V# is eliminated. This is more efficient and 4591 avoids the problem that the second register allocation may perform 4592 spilling which will fail as there is no longer a scratch V#. 4593 4594When the kernel prolog code is being emitted it is known whether the scratch V# 4595described above is actually used. If it is, the prolog code must set it up by 4596copying the Private Segment Buffer to the scratch V# registers and then adding 4597the Private Segment Wavefront Offset to the queue base address in the V#. The 4598result is a V# with a base address pointing to the beginning of the wavefront 4599scratch backing memory. 4600 4601The Private Segment Buffer is always requested, but the Private Segment 4602Wavefront Offset is only requested if it is used (see 4603:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4604 4605.. _amdgpu-amdhsa-memory-model: 4606 4607Memory Model 4608~~~~~~~~~~~~ 4609 4610This section describes the mapping of the LLVM memory model onto AMDGPU machine 4611code (see :ref:`memmodel`). 4612 4613The AMDGPU backend supports the memory synchronization scopes specified in 4614:ref:`amdgpu-memory-scopes`. 4615 4616The code sequences used to implement the memory model specify the order of 4617instructions that a single thread must execute. The ``s_waitcnt`` and cache 4618management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4619to other memory instructions executed by the same thread. This allows them to be 4620moved earlier or later which can allow them to be combined with other instances 4621of the same instruction, or hoisted/sunk out of loops to improve performance. 4622Only the instructions related to the memory model are given; additional 4623``s_waitcnt`` instructions are required to ensure registers are defined before 4624being used. These may be able to be combined with the memory model ``s_waitcnt`` 4625instructions as described above. 4626 4627The AMDGPU backend supports the following memory models: 4628 4629 HSA Memory Model [HSA]_ 4630 The HSA memory model uses a single happens-before relation for all address 4631 spaces (see :ref:`amdgpu-address-spaces`). 4632 OpenCL Memory Model [OpenCL]_ 4633 The OpenCL memory model which has separate happens-before relations for the 4634 global and local address spaces. Only a fence specifying both global and 4635 local address space, and seq_cst instructions join the relationships. Since 4636 the LLVM ``memfence`` instruction does not allow an address space to be 4637 specified the OpenCL fence has to conservatively assume both local and 4638 global address space was specified. However, optimizations can often be 4639 done to eliminate the additional ``s_waitcnt`` instructions when there are 4640 no intervening memory instructions which access the corresponding address 4641 space. The code sequences in the table indicate what can be omitted for the 4642 OpenCL memory. The target triple environment is used to determine if the 4643 source language is OpenCL (see :ref:`amdgpu-opencl`). 4644 4645``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4646operations. 4647 4648``buffer/global/flat_load/store/atomic`` instructions to global memory are 4649termed vector memory operations. 4650 4651Private address space uses ``buffer_load/store`` using the scratch V# 4652(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4653is accessing the memory, atomic memory orderings are not meaningful, and all 4654accesses are treated as non-atomic. 4655 4656Constant address space uses ``buffer/global_load`` instructions (or equivalent 4657scalar memory instructions). Since the constant address space contents do not 4658change during the execution of a kernel dispatch it is not legal to perform 4659stores, and atomic memory orderings are not meaningful, and all accesses are 4660treated as non-atomic. 4661 4662A memory synchronization scope wider than work-group is not meaningful for the 4663group (LDS) address space and is treated as work-group. 4664 4665The memory model does not support the region address space which is treated as 4666non-atomic. 4667 4668Acquire memory ordering is not meaningful on store atomic instructions and is 4669treated as non-atomic. 4670 4671Release memory ordering is not meaningful on load atomic instructions and is 4672treated a non-atomic. 4673 4674Acquire-release memory ordering is not meaningful on load or store atomic 4675instructions and is treated as acquire and release respectively. 4676 4677The memory order also adds the single thread optimization constraints defined in 4678table 4679:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4680 4681 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4682 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4683 4684 ============ ============================================================== 4685 LLVM Memory Optimization Constraints 4686 Ordering 4687 ============ ============================================================== 4688 unordered *none* 4689 monotonic *none* 4690 acquire - If a load atomic/atomicrmw then no following load/load 4691 atomic/store/store atomic/atomicrmw/fence instruction can be 4692 moved before the acquire. 4693 - If a fence then same as load atomic, plus no preceding 4694 associated fence-paired-atomic can be moved after the fence. 4695 release - If a store atomic/atomicrmw then no preceding load/load 4696 atomic/store/store atomic/atomicrmw/fence instruction can be 4697 moved after the release. 4698 - If a fence then same as store atomic, plus no following 4699 associated fence-paired-atomic can be moved before the 4700 fence. 4701 acq_rel Same constraints as both acquire and release. 4702 seq_cst - If a load atomic then same constraints as acquire, plus no 4703 preceding sequentially consistent load atomic/store 4704 atomic/atomicrmw/fence instruction can be moved after the 4705 seq_cst. 4706 - If a store atomic then the same constraints as release, plus 4707 no following sequentially consistent load atomic/store 4708 atomic/atomicrmw/fence instruction can be moved before the 4709 seq_cst. 4710 - If an atomicrmw/fence then same constraints as acq_rel. 4711 ============ ============================================================== 4712 4713The code sequences used to implement the memory model are defined in the 4714following sections: 4715 4716* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4717* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4718* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4719 4720.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4721 4722Memory Model GFX6-GFX9 4723++++++++++++++++++++++ 4724 4725For GFX6-GFX9: 4726 4727* Each agent has multiple shader arrays (SA). 4728* Each SA has multiple compute units (CU). 4729* Each CU has multiple SIMDs that execute wavefronts. 4730* The wavefronts for a single work-group are executed in the same CU but may be 4731 executed by different SIMDs. 4732* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4733 executing on it. 4734* All LDS operations of a CU are performed as wavefront wide operations in a 4735 global order and involve no caching. Completion is reported to a wavefront in 4736 execution order. 4737* The LDS memory has multiple request queues shared by the SIMDs of a 4738 CU. Therefore, the LDS operations performed by different wavefronts of a 4739 work-group can be reordered relative to each other, which can result in 4740 reordering the visibility of vector memory operations with respect to LDS 4741 operations of other wavefronts in the same work-group. A ``s_waitcnt 4742 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4743 vector memory operations between wavefronts of a work-group, but not between 4744 operations performed by the same wavefront. 4745* The vector memory operations are performed as wavefront wide operations and 4746 completion is reported to a wavefront in execution order. The exception is 4747 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4748 vector memory order if they access LDS memory, and out of LDS operation order 4749 if they access global memory. 4750* The vector memory operations access a single vector L1 cache shared by all 4751 SIMDs a CU. Therefore, no special action is required for coherence between the 4752 lanes of a single wavefront, or for coherence between wavefronts in the same 4753 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4754 wavefronts executing in different work-groups as they may be executing on 4755 different CUs. 4756* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4757 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4758 scalar operations are used in a restricted way so do not impact the memory 4759 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4760* The vector and scalar memory operations use an L2 cache shared by all CUs on 4761 the same agent. 4762* The L2 cache has independent channels to service disjoint ranges of virtual 4763 addresses. 4764* Each CU has a separate request queue per channel. Therefore, the vector and 4765 scalar memory operations performed by wavefronts executing in different 4766 work-groups (which may be executing on different CUs) of an agent can be 4767 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4768 ensure synchronization between vector memory operations of different CUs. It 4769 ensures a previous vector memory operation has completed before executing a 4770 subsequent vector memory or LDS operation and so can be used to meet the 4771 requirements of acquire and release. 4772* The L2 cache can be kept coherent with other agents on some targets, or ranges 4773 of virtual addresses can be set up to bypass it to ensure system coherence. 4774 4775Scalar memory operations are only used to access memory that is proven to not 4776change during the execution of the kernel dispatch. This includes constant 4777address space and global address space for program scope ``const`` variables. 4778Therefore, the kernel machine code does not have to maintain the scalar cache to 4779ensure it is coherent with the vector caches. The scalar and vector caches are 4780invalidated between kernel dispatches by CP since constant address space data 4781may change between kernel dispatch executions. See 4782:ref:`amdgpu-amdhsa-memory-spaces`. 4783 4784The one exception is if scalar writes are used to spill SGPR registers. In this 4785case the AMDGPU backend ensures the memory location used to spill is never 4786accessed by vector memory operations at the same time. If scalar writes are used 4787then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4788return since the locations may be used for vector memory instructions by a 4789future wavefront that uses the same scratch area, or a function call that 4790creates a frame at the same address, respectively. There is no need for a 4791``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4792 4793For kernarg backing memory: 4794 4795* CP invalidates the L1 cache at the start of each kernel dispatch. 4796* On dGPU the kernarg backing memory is allocated in host memory accessed as 4797 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4798 causes it to be treated as non-volatile and so is not invalidated by 4799 ``*_vol``. 4800* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4801 and so the L2 cache will be coherent with the CPU and other agents. 4802 4803Scratch backing memory (which is used for the private address space) is accessed 4804with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4805only accessed by a single thread, and is always write-before-read, there is 4806never a need to invalidate these entries from the L1 cache. Hence all cache 4807invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4808 4809The code sequences used to implement the memory model for GFX6-GFX9 are defined 4810in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4811 4812 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4813 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4814 4815 ============ ============ ============== ========== ================================ 4816 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4817 Ordering Sync Scope Address GFX6-GFX9 4818 Space 4819 ============ ============ ============== ========== ================================ 4820 **Non-Atomic** 4821 ------------------------------------------------------------------------------------ 4822 load *none* *none* - global - !volatile & !nontemporal 4823 - generic 4824 - private 1. buffer/global/flat_load 4825 - constant 4826 - !volatile & nontemporal 4827 4828 1. buffer/global/flat_load 4829 glc=1 slc=1 4830 4831 - volatile 4832 4833 1. buffer/global/flat_load 4834 glc=1 4835 2. s_waitcnt vmcnt(0) 4836 4837 - Must happen before 4838 any following volatile 4839 global/generic 4840 load/store. 4841 - Ensures that 4842 volatile 4843 operations to 4844 different 4845 addresses will not 4846 be reordered by 4847 hardware. 4848 4849 load *none* *none* - local 1. ds_load 4850 store *none* *none* - global - !volatile & !nontemporal 4851 - generic 4852 - private 1. buffer/global/flat_store 4853 - constant 4854 - !volatile & nontemporal 4855 4856 1. buffer/global/flat_store 4857 glc=1 slc=1 4858 4859 - volatile 4860 4861 1. buffer/global/flat_store 4862 2. s_waitcnt vmcnt(0) 4863 4864 - Must happen before 4865 any following volatile 4866 global/generic 4867 load/store. 4868 - Ensures that 4869 volatile 4870 operations to 4871 different 4872 addresses will not 4873 be reordered by 4874 hardware. 4875 4876 store *none* *none* - local 1. ds_store 4877 **Unordered Atomic** 4878 ------------------------------------------------------------------------------------ 4879 load atomic unordered *any* *any* *Same as non-atomic*. 4880 store atomic unordered *any* *any* *Same as non-atomic*. 4881 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4882 **Monotonic Atomic** 4883 ------------------------------------------------------------------------------------ 4884 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4885 - wavefront - local 4886 - workgroup - generic 4887 load atomic monotonic - agent - global 1. buffer/global/flat_load 4888 - system - generic glc=1 4889 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4890 - wavefront - generic 4891 - workgroup 4892 - agent 4893 - system 4894 store atomic monotonic - singlethread - local 1. ds_store 4895 - wavefront 4896 - workgroup 4897 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4898 - wavefront - generic 4899 - workgroup 4900 - agent 4901 - system 4902 atomicrmw monotonic - singlethread - local 1. ds_atomic 4903 - wavefront 4904 - workgroup 4905 **Acquire Atomic** 4906 ------------------------------------------------------------------------------------ 4907 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4908 - wavefront - local 4909 - generic 4910 load atomic acquire - workgroup - global 1. buffer/global_load 4911 load atomic acquire - workgroup - local 1. ds/flat_load 4912 - generic 2. s_waitcnt lgkmcnt(0) 4913 4914 - If OpenCL, omit. 4915 - Must happen before 4916 any following 4917 global/generic 4918 load/load 4919 atomic/store/store 4920 atomic/atomicrmw. 4921 - Ensures any 4922 following global 4923 data read is no 4924 older than a local load 4925 atomic value being 4926 acquired. 4927 4928 load atomic acquire - agent - global 1. buffer/global_load 4929 - system glc=1 4930 2. s_waitcnt vmcnt(0) 4931 4932 - Must happen before 4933 following 4934 buffer_wbinvl1_vol. 4935 - Ensures the load 4936 has completed 4937 before invalidating 4938 the cache. 4939 4940 3. buffer_wbinvl1_vol 4941 4942 - Must happen before 4943 any following 4944 global/generic 4945 load/load 4946 atomic/atomicrmw. 4947 - Ensures that 4948 following 4949 loads will not see 4950 stale global data. 4951 4952 load atomic acquire - agent - generic 1. flat_load glc=1 4953 - system 2. s_waitcnt vmcnt(0) & 4954 lgkmcnt(0) 4955 4956 - If OpenCL omit 4957 lgkmcnt(0). 4958 - Must happen before 4959 following 4960 buffer_wbinvl1_vol. 4961 - Ensures the flat_load 4962 has completed 4963 before invalidating 4964 the cache. 4965 4966 3. buffer_wbinvl1_vol 4967 4968 - Must happen before 4969 any following 4970 global/generic 4971 load/load 4972 atomic/atomicrmw. 4973 - Ensures that 4974 following loads 4975 will not see stale 4976 global data. 4977 4978 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 4979 - wavefront - local 4980 - generic 4981 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 4982 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 4983 - generic 2. s_waitcnt lgkmcnt(0) 4984 4985 - If OpenCL, omit. 4986 - Must happen before 4987 any following 4988 global/generic 4989 load/load 4990 atomic/store/store 4991 atomic/atomicrmw. 4992 - Ensures any 4993 following global 4994 data read is no 4995 older than a local 4996 atomicrmw value 4997 being acquired. 4998 4999 atomicrmw acquire - agent - global 1. buffer/global_atomic 5000 - system 2. s_waitcnt vmcnt(0) 5001 5002 - Must happen before 5003 following 5004 buffer_wbinvl1_vol. 5005 - Ensures the 5006 atomicrmw has 5007 completed before 5008 invalidating the 5009 cache. 5010 5011 3. buffer_wbinvl1_vol 5012 5013 - Must happen before 5014 any following 5015 global/generic 5016 load/load 5017 atomic/atomicrmw. 5018 - Ensures that 5019 following loads 5020 will not see stale 5021 global data. 5022 5023 atomicrmw acquire - agent - generic 1. flat_atomic 5024 - system 2. s_waitcnt vmcnt(0) & 5025 lgkmcnt(0) 5026 5027 - If OpenCL, omit 5028 lgkmcnt(0). 5029 - Must happen before 5030 following 5031 buffer_wbinvl1_vol. 5032 - Ensures the 5033 atomicrmw has 5034 completed before 5035 invalidating the 5036 cache. 5037 5038 3. buffer_wbinvl1_vol 5039 5040 - Must happen before 5041 any following 5042 global/generic 5043 load/load 5044 atomic/atomicrmw. 5045 - Ensures that 5046 following loads 5047 will not see stale 5048 global data. 5049 5050 fence acquire - singlethread *none* *none* 5051 - wavefront 5052 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5053 5054 - If OpenCL and 5055 address space is 5056 not generic, omit. 5057 - However, since LLVM 5058 currently has no 5059 address space on 5060 the fence need to 5061 conservatively 5062 always generate. If 5063 fence had an 5064 address space then 5065 set to address 5066 space of OpenCL 5067 fence flag, or to 5068 generic if both 5069 local and global 5070 flags are 5071 specified. 5072 - Must happen after 5073 any preceding 5074 local/generic load 5075 atomic/atomicrmw 5076 with an equal or 5077 wider sync scope 5078 and memory ordering 5079 stronger than 5080 unordered (this is 5081 termed the 5082 fence-paired-atomic). 5083 - Must happen before 5084 any following 5085 global/generic 5086 load/load 5087 atomic/store/store 5088 atomic/atomicrmw. 5089 - Ensures any 5090 following global 5091 data read is no 5092 older than the 5093 value read by the 5094 fence-paired-atomic. 5095 5096 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5097 - system vmcnt(0) 5098 5099 - If OpenCL and 5100 address space is 5101 not generic, omit 5102 lgkmcnt(0). 5103 - However, since LLVM 5104 currently has no 5105 address space on 5106 the fence need to 5107 conservatively 5108 always generate 5109 (see comment for 5110 previous fence). 5111 - Could be split into 5112 separate s_waitcnt 5113 vmcnt(0) and 5114 s_waitcnt 5115 lgkmcnt(0) to allow 5116 them to be 5117 independently moved 5118 according to the 5119 following rules. 5120 - s_waitcnt vmcnt(0) 5121 must happen after 5122 any preceding 5123 global/generic load 5124 atomic/atomicrmw 5125 with an equal or 5126 wider sync scope 5127 and memory ordering 5128 stronger than 5129 unordered (this is 5130 termed the 5131 fence-paired-atomic). 5132 - s_waitcnt lgkmcnt(0) 5133 must happen after 5134 any preceding 5135 local/generic load 5136 atomic/atomicrmw 5137 with an equal or 5138 wider sync scope 5139 and memory ordering 5140 stronger than 5141 unordered (this is 5142 termed the 5143 fence-paired-atomic). 5144 - Must happen before 5145 the following 5146 buffer_wbinvl1_vol. 5147 - Ensures that the 5148 fence-paired atomic 5149 has completed 5150 before invalidating 5151 the 5152 cache. Therefore 5153 any following 5154 locations read must 5155 be no older than 5156 the value read by 5157 the 5158 fence-paired-atomic. 5159 5160 2. buffer_wbinvl1_vol 5161 5162 - Must happen before any 5163 following global/generic 5164 load/load 5165 atomic/store/store 5166 atomic/atomicrmw. 5167 - Ensures that 5168 following loads 5169 will not see stale 5170 global data. 5171 5172 **Release Atomic** 5173 ------------------------------------------------------------------------------------ 5174 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5175 - wavefront - local 5176 - generic 5177 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5178 - generic 5179 - If OpenCL, omit. 5180 - Must happen after 5181 any preceding 5182 local/generic 5183 load/store/load 5184 atomic/store 5185 atomic/atomicrmw. 5186 - Must happen before 5187 the following 5188 store. 5189 - Ensures that all 5190 memory operations 5191 to local have 5192 completed before 5193 performing the 5194 store that is being 5195 released. 5196 5197 2. buffer/global/flat_store 5198 store atomic release - workgroup - local 1. ds_store 5199 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5200 - system - generic vmcnt(0) 5201 5202 - If OpenCL and 5203 address space is 5204 not generic, omit 5205 lgkmcnt(0). 5206 - Could be split into 5207 separate s_waitcnt 5208 vmcnt(0) and 5209 s_waitcnt 5210 lgkmcnt(0) to allow 5211 them to be 5212 independently moved 5213 according to the 5214 following rules. 5215 - s_waitcnt vmcnt(0) 5216 must happen after 5217 any preceding 5218 global/generic 5219 load/store/load 5220 atomic/store 5221 atomic/atomicrmw. 5222 - s_waitcnt lgkmcnt(0) 5223 must happen after 5224 any preceding 5225 local/generic 5226 load/store/load 5227 atomic/store 5228 atomic/atomicrmw. 5229 - Must happen before 5230 the following 5231 store. 5232 - Ensures that all 5233 memory operations 5234 to memory have 5235 completed before 5236 performing the 5237 store that is being 5238 released. 5239 5240 2. buffer/global/flat_store 5241 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5242 - wavefront - local 5243 - generic 5244 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5245 - generic 5246 - If OpenCL, omit. 5247 - Must happen after 5248 any preceding 5249 local/generic 5250 load/store/load 5251 atomic/store 5252 atomic/atomicrmw. 5253 - Must happen before 5254 the following 5255 atomicrmw. 5256 - Ensures that all 5257 memory operations 5258 to local have 5259 completed before 5260 performing the 5261 atomicrmw that is 5262 being released. 5263 5264 2. buffer/global/flat_atomic 5265 atomicrmw release - workgroup - local 1. ds_atomic 5266 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5267 - system - generic vmcnt(0) 5268 5269 - If OpenCL, omit 5270 lgkmcnt(0). 5271 - Could be split into 5272 separate s_waitcnt 5273 vmcnt(0) and 5274 s_waitcnt 5275 lgkmcnt(0) to allow 5276 them to be 5277 independently moved 5278 according to the 5279 following rules. 5280 - s_waitcnt vmcnt(0) 5281 must happen after 5282 any preceding 5283 global/generic 5284 load/store/load 5285 atomic/store 5286 atomic/atomicrmw. 5287 - s_waitcnt lgkmcnt(0) 5288 must happen after 5289 any preceding 5290 local/generic 5291 load/store/load 5292 atomic/store 5293 atomic/atomicrmw. 5294 - Must happen before 5295 the following 5296 atomicrmw. 5297 - Ensures that all 5298 memory operations 5299 to global and local 5300 have completed 5301 before performing 5302 the atomicrmw that 5303 is being released. 5304 5305 2. buffer/global/flat_atomic 5306 fence release - singlethread *none* *none* 5307 - wavefront 5308 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5309 5310 - If OpenCL and 5311 address space is 5312 not generic, omit. 5313 - However, since LLVM 5314 currently has no 5315 address space on 5316 the fence need to 5317 conservatively 5318 always generate. If 5319 fence had an 5320 address space then 5321 set to address 5322 space of OpenCL 5323 fence flag, or to 5324 generic if both 5325 local and global 5326 flags are 5327 specified. 5328 - Must happen after 5329 any preceding 5330 local/generic 5331 load/load 5332 atomic/store/store 5333 atomic/atomicrmw. 5334 - Must happen before 5335 any following store 5336 atomic/atomicrmw 5337 with an equal or 5338 wider sync scope 5339 and memory ordering 5340 stronger than 5341 unordered (this is 5342 termed the 5343 fence-paired-atomic). 5344 - Ensures that all 5345 memory operations 5346 to local have 5347 completed before 5348 performing the 5349 following 5350 fence-paired-atomic. 5351 5352 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5353 - system vmcnt(0) 5354 5355 - If OpenCL and 5356 address space is 5357 not generic, omit 5358 lgkmcnt(0). 5359 - If OpenCL and 5360 address space is 5361 local, omit 5362 vmcnt(0). 5363 - However, since LLVM 5364 currently has no 5365 address space on 5366 the fence need to 5367 conservatively 5368 always generate. If 5369 fence had an 5370 address space then 5371 set to address 5372 space of OpenCL 5373 fence flag, or to 5374 generic if both 5375 local and global 5376 flags are 5377 specified. 5378 - Could be split into 5379 separate s_waitcnt 5380 vmcnt(0) and 5381 s_waitcnt 5382 lgkmcnt(0) to allow 5383 them to be 5384 independently moved 5385 according to the 5386 following rules. 5387 - s_waitcnt vmcnt(0) 5388 must happen after 5389 any preceding 5390 global/generic 5391 load/store/load 5392 atomic/store 5393 atomic/atomicrmw. 5394 - s_waitcnt lgkmcnt(0) 5395 must happen after 5396 any preceding 5397 local/generic 5398 load/store/load 5399 atomic/store 5400 atomic/atomicrmw. 5401 - Must happen before 5402 any following store 5403 atomic/atomicrmw 5404 with an equal or 5405 wider sync scope 5406 and memory ordering 5407 stronger than 5408 unordered (this is 5409 termed the 5410 fence-paired-atomic). 5411 - Ensures that all 5412 memory operations 5413 have 5414 completed before 5415 performing the 5416 following 5417 fence-paired-atomic. 5418 5419 **Acquire-Release Atomic** 5420 ------------------------------------------------------------------------------------ 5421 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5422 - wavefront - local 5423 - generic 5424 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5425 5426 - If OpenCL, omit. 5427 - Must happen after 5428 any preceding 5429 local/generic 5430 load/store/load 5431 atomic/store 5432 atomic/atomicrmw. 5433 - Must happen before 5434 the following 5435 atomicrmw. 5436 - Ensures that all 5437 memory operations 5438 to local have 5439 completed before 5440 performing the 5441 atomicrmw that is 5442 being released. 5443 5444 2. buffer/global_atomic 5445 5446 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5447 2. s_waitcnt lgkmcnt(0) 5448 5449 - If OpenCL, omit. 5450 - Must happen before 5451 any following 5452 global/generic 5453 load/load 5454 atomic/store/store 5455 atomic/atomicrmw. 5456 - Ensures any 5457 following global 5458 data read is no 5459 older than the local load 5460 atomic value being 5461 acquired. 5462 5463 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5464 5465 - If OpenCL, omit. 5466 - Must happen after 5467 any preceding 5468 local/generic 5469 load/store/load 5470 atomic/store 5471 atomic/atomicrmw. 5472 - Must happen before 5473 the following 5474 atomicrmw. 5475 - Ensures that all 5476 memory operations 5477 to local have 5478 completed before 5479 performing the 5480 atomicrmw that is 5481 being released. 5482 5483 2. flat_atomic 5484 3. s_waitcnt lgkmcnt(0) 5485 5486 - If OpenCL, omit. 5487 - Must happen before 5488 any following 5489 global/generic 5490 load/load 5491 atomic/store/store 5492 atomic/atomicrmw. 5493 - Ensures any 5494 following global 5495 data read is no 5496 older than a local load 5497 atomic value being 5498 acquired. 5499 5500 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5501 - system vmcnt(0) 5502 5503 - If OpenCL, omit 5504 lgkmcnt(0). 5505 - Could be split into 5506 separate s_waitcnt 5507 vmcnt(0) and 5508 s_waitcnt 5509 lgkmcnt(0) to allow 5510 them to be 5511 independently moved 5512 according to the 5513 following rules. 5514 - s_waitcnt vmcnt(0) 5515 must happen after 5516 any preceding 5517 global/generic 5518 load/store/load 5519 atomic/store 5520 atomic/atomicrmw. 5521 - s_waitcnt lgkmcnt(0) 5522 must happen after 5523 any preceding 5524 local/generic 5525 load/store/load 5526 atomic/store 5527 atomic/atomicrmw. 5528 - Must happen before 5529 the following 5530 atomicrmw. 5531 - Ensures that all 5532 memory operations 5533 to global have 5534 completed before 5535 performing the 5536 atomicrmw that is 5537 being released. 5538 5539 2. buffer/global_atomic 5540 3. s_waitcnt vmcnt(0) 5541 5542 - Must happen before 5543 following 5544 buffer_wbinvl1_vol. 5545 - Ensures the 5546 atomicrmw has 5547 completed before 5548 invalidating the 5549 cache. 5550 5551 4. buffer_wbinvl1_vol 5552 5553 - Must happen before 5554 any following 5555 global/generic 5556 load/load 5557 atomic/atomicrmw. 5558 - Ensures that 5559 following loads 5560 will not see stale 5561 global data. 5562 5563 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5564 - system vmcnt(0) 5565 5566 - If OpenCL, omit 5567 lgkmcnt(0). 5568 - Could be split into 5569 separate s_waitcnt 5570 vmcnt(0) and 5571 s_waitcnt 5572 lgkmcnt(0) to allow 5573 them to be 5574 independently moved 5575 according to the 5576 following rules. 5577 - s_waitcnt vmcnt(0) 5578 must happen after 5579 any preceding 5580 global/generic 5581 load/store/load 5582 atomic/store 5583 atomic/atomicrmw. 5584 - s_waitcnt lgkmcnt(0) 5585 must happen after 5586 any preceding 5587 local/generic 5588 load/store/load 5589 atomic/store 5590 atomic/atomicrmw. 5591 - Must happen before 5592 the following 5593 atomicrmw. 5594 - Ensures that all 5595 memory operations 5596 to global have 5597 completed before 5598 performing the 5599 atomicrmw that is 5600 being released. 5601 5602 2. flat_atomic 5603 3. s_waitcnt vmcnt(0) & 5604 lgkmcnt(0) 5605 5606 - If OpenCL, omit 5607 lgkmcnt(0). 5608 - Must happen before 5609 following 5610 buffer_wbinvl1_vol. 5611 - Ensures the 5612 atomicrmw has 5613 completed before 5614 invalidating the 5615 cache. 5616 5617 4. buffer_wbinvl1_vol 5618 5619 - Must happen before 5620 any following 5621 global/generic 5622 load/load 5623 atomic/atomicrmw. 5624 - Ensures that 5625 following loads 5626 will not see stale 5627 global data. 5628 5629 fence acq_rel - singlethread *none* *none* 5630 - wavefront 5631 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5632 5633 - If OpenCL and 5634 address space is 5635 not generic, omit. 5636 - However, 5637 since LLVM 5638 currently has no 5639 address space on 5640 the fence need to 5641 conservatively 5642 always generate 5643 (see comment for 5644 previous fence). 5645 - Must happen after 5646 any preceding 5647 local/generic 5648 load/load 5649 atomic/store/store 5650 atomic/atomicrmw. 5651 - Must happen before 5652 any following 5653 global/generic 5654 load/load 5655 atomic/store/store 5656 atomic/atomicrmw. 5657 - Ensures that all 5658 memory operations 5659 to local have 5660 completed before 5661 performing any 5662 following global 5663 memory operations. 5664 - Ensures that the 5665 preceding 5666 local/generic load 5667 atomic/atomicrmw 5668 with an equal or 5669 wider sync scope 5670 and memory ordering 5671 stronger than 5672 unordered (this is 5673 termed the 5674 acquire-fence-paired-atomic) 5675 has completed 5676 before following 5677 global memory 5678 operations. This 5679 satisfies the 5680 requirements of 5681 acquire. 5682 - Ensures that all 5683 previous memory 5684 operations have 5685 completed before a 5686 following 5687 local/generic store 5688 atomic/atomicrmw 5689 with an equal or 5690 wider sync scope 5691 and memory ordering 5692 stronger than 5693 unordered (this is 5694 termed the 5695 release-fence-paired-atomic). 5696 This satisfies the 5697 requirements of 5698 release. 5699 5700 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5701 - system vmcnt(0) 5702 5703 - If OpenCL and 5704 address space is 5705 not generic, omit 5706 lgkmcnt(0). 5707 - However, since LLVM 5708 currently has no 5709 address space on 5710 the fence need to 5711 conservatively 5712 always generate 5713 (see comment for 5714 previous fence). 5715 - Could be split into 5716 separate s_waitcnt 5717 vmcnt(0) and 5718 s_waitcnt 5719 lgkmcnt(0) to allow 5720 them to be 5721 independently moved 5722 according to the 5723 following rules. 5724 - s_waitcnt vmcnt(0) 5725 must happen after 5726 any preceding 5727 global/generic 5728 load/store/load 5729 atomic/store 5730 atomic/atomicrmw. 5731 - s_waitcnt lgkmcnt(0) 5732 must happen after 5733 any preceding 5734 local/generic 5735 load/store/load 5736 atomic/store 5737 atomic/atomicrmw. 5738 - Must happen before 5739 the following 5740 buffer_wbinvl1_vol. 5741 - Ensures that the 5742 preceding 5743 global/local/generic 5744 load 5745 atomic/atomicrmw 5746 with an equal or 5747 wider sync scope 5748 and memory ordering 5749 stronger than 5750 unordered (this is 5751 termed the 5752 acquire-fence-paired-atomic) 5753 has completed 5754 before invalidating 5755 the cache. This 5756 satisfies the 5757 requirements of 5758 acquire. 5759 - Ensures that all 5760 previous memory 5761 operations have 5762 completed before a 5763 following 5764 global/local/generic 5765 store 5766 atomic/atomicrmw 5767 with an equal or 5768 wider sync scope 5769 and memory ordering 5770 stronger than 5771 unordered (this is 5772 termed the 5773 release-fence-paired-atomic). 5774 This satisfies the 5775 requirements of 5776 release. 5777 5778 2. buffer_wbinvl1_vol 5779 5780 - Must happen before 5781 any following 5782 global/generic 5783 load/load 5784 atomic/store/store 5785 atomic/atomicrmw. 5786 - Ensures that 5787 following loads 5788 will not see stale 5789 global data. This 5790 satisfies the 5791 requirements of 5792 acquire. 5793 5794 **Sequential Consistent Atomic** 5795 ------------------------------------------------------------------------------------ 5796 load atomic seq_cst - singlethread - global *Same as corresponding 5797 - wavefront - local load atomic acquire, 5798 - generic except must generated 5799 all instructions even 5800 for OpenCL.* 5801 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5802 - generic 5803 5804 - Must 5805 happen after 5806 preceding 5807 local/generic load 5808 atomic/store 5809 atomic/atomicrmw 5810 with memory 5811 ordering of seq_cst 5812 and with equal or 5813 wider sync scope. 5814 (Note that seq_cst 5815 fences have their 5816 own s_waitcnt 5817 lgkmcnt(0) and so do 5818 not need to be 5819 considered.) 5820 - Ensures any 5821 preceding 5822 sequential 5823 consistent local 5824 memory instructions 5825 have completed 5826 before executing 5827 this sequentially 5828 consistent 5829 instruction. This 5830 prevents reordering 5831 a seq_cst store 5832 followed by a 5833 seq_cst load. (Note 5834 that seq_cst is 5835 stronger than 5836 acquire/release as 5837 the reordering of 5838 load acquire 5839 followed by a store 5840 release is 5841 prevented by the 5842 s_waitcnt of 5843 the release, but 5844 there is nothing 5845 preventing a store 5846 release followed by 5847 load acquire from 5848 completing out of 5849 order. The s_waitcnt 5850 could be placed after 5851 seq_store or before 5852 the seq_load. We 5853 choose the load to 5854 make the s_waitcnt be 5855 as late as possible 5856 so that the store 5857 may have already 5858 completed.) 5859 5860 2. *Following 5861 instructions same as 5862 corresponding load 5863 atomic acquire, 5864 except must generated 5865 all instructions even 5866 for OpenCL.* 5867 load atomic seq_cst - workgroup - local *Same as corresponding 5868 load atomic acquire, 5869 except must generated 5870 all instructions even 5871 for OpenCL.* 5872 5873 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5874 - system - generic vmcnt(0) 5875 5876 - Could be split into 5877 separate s_waitcnt 5878 vmcnt(0) 5879 and s_waitcnt 5880 lgkmcnt(0) to allow 5881 them to be 5882 independently moved 5883 according to the 5884 following rules. 5885 - s_waitcnt lgkmcnt(0) 5886 must happen after 5887 preceding 5888 global/generic load 5889 atomic/store 5890 atomic/atomicrmw 5891 with memory 5892 ordering of seq_cst 5893 and with equal or 5894 wider sync scope. 5895 (Note that seq_cst 5896 fences have their 5897 own s_waitcnt 5898 lgkmcnt(0) and so do 5899 not need to be 5900 considered.) 5901 - s_waitcnt vmcnt(0) 5902 must happen after 5903 preceding 5904 global/generic load 5905 atomic/store 5906 atomic/atomicrmw 5907 with memory 5908 ordering of seq_cst 5909 and with equal or 5910 wider sync scope. 5911 (Note that seq_cst 5912 fences have their 5913 own s_waitcnt 5914 vmcnt(0) and so do 5915 not need to be 5916 considered.) 5917 - Ensures any 5918 preceding 5919 sequential 5920 consistent global 5921 memory instructions 5922 have completed 5923 before executing 5924 this sequentially 5925 consistent 5926 instruction. This 5927 prevents reordering 5928 a seq_cst store 5929 followed by a 5930 seq_cst load. (Note 5931 that seq_cst is 5932 stronger than 5933 acquire/release as 5934 the reordering of 5935 load acquire 5936 followed by a store 5937 release is 5938 prevented by the 5939 s_waitcnt of 5940 the release, but 5941 there is nothing 5942 preventing a store 5943 release followed by 5944 load acquire from 5945 completing out of 5946 order. The s_waitcnt 5947 could be placed after 5948 seq_store or before 5949 the seq_load. We 5950 choose the load to 5951 make the s_waitcnt be 5952 as late as possible 5953 so that the store 5954 may have already 5955 completed.) 5956 5957 2. *Following 5958 instructions same as 5959 corresponding load 5960 atomic acquire, 5961 except must generated 5962 all instructions even 5963 for OpenCL.* 5964 store atomic seq_cst - singlethread - global *Same as corresponding 5965 - wavefront - local store atomic release, 5966 - workgroup - generic except must generated 5967 - agent all instructions even 5968 - system for OpenCL.* 5969 atomicrmw seq_cst - singlethread - global *Same as corresponding 5970 - wavefront - local atomicrmw acq_rel, 5971 - workgroup - generic except must generated 5972 - agent all instructions even 5973 - system for OpenCL.* 5974 fence seq_cst - singlethread *none* *Same as corresponding 5975 - wavefront fence acq_rel, 5976 - workgroup except must generated 5977 - agent all instructions even 5978 - system for OpenCL.* 5979 ============ ============ ============== ========== ================================ 5980 5981.. _amdgpu-amdhsa-memory-model-gfx90a: 5982 5983Memory Model GFX90A 5984+++++++++++++++++++ 5985 5986For GFX90A: 5987 5988* Each agent has multiple shader arrays (SA). 5989* Each SA has multiple compute units (CU). 5990* Each CU has multiple SIMDs that execute wavefronts. 5991* The wavefronts for a single work-group are executed in the same CU but may be 5992 executed by different SIMDs. The exception is when in tgsplit execution mode 5993 when the wavefronts may be executed by different SIMDs in different CUs. 5994* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5995 executing on it. The exception is when in tgsplit execution mode when no LDS 5996 is allocated as wavefronts of the same work-group can be in different CUs. 5997* All LDS operations of a CU are performed as wavefront wide operations in a 5998 global order and involve no caching. Completion is reported to a wavefront in 5999 execution order. 6000* The LDS memory has multiple request queues shared by the SIMDs of a 6001 CU. Therefore, the LDS operations performed by different wavefronts of a 6002 work-group can be reordered relative to each other, which can result in 6003 reordering the visibility of vector memory operations with respect to LDS 6004 operations of other wavefronts in the same work-group. A ``s_waitcnt 6005 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6006 vector memory operations between wavefronts of a work-group, but not between 6007 operations performed by the same wavefront. 6008* The vector memory operations are performed as wavefront wide operations and 6009 completion is reported to a wavefront in execution order. The exception is 6010 that ``flat_load/store/atomic`` instructions can report out of vector memory 6011 order if they access LDS memory, and out of LDS operation order if they access 6012 global memory. 6013* The vector memory operations access a single vector L1 cache shared by all 6014 SIMDs a CU. Therefore: 6015 6016 * No special action is required for coherence between the lanes of a single 6017 wavefront. 6018 6019 * No special action is required for coherence between wavefronts in the same 6020 work-group since they execute on the same CU. The exception is when in 6021 tgsplit execution mode as wavefronts of the same work-group can be in 6022 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6023 the following item. 6024 6025 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6026 executing in different work-groups as they may be executing on different 6027 CUs. 6028 6029* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6030 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6031 scalar operations are used in a restricted way so do not impact the memory 6032 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6033* The vector and scalar memory operations use an L2 cache shared by all CUs on 6034 the same agent. 6035 6036 * The L2 cache has independent channels to service disjoint ranges of virtual 6037 addresses. 6038 * Each CU has a separate request queue per channel. Therefore, the vector and 6039 scalar memory operations performed by wavefronts executing in different 6040 work-groups (which may be executing on different CUs), or the same 6041 work-group if executing in tgsplit mode, of an agent can be reordered 6042 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6043 synchronization between vector memory operations of different CUs. It 6044 ensures a previous vector memory operation has completed before executing a 6045 subsequent vector memory or LDS operation and so can be used to meet the 6046 requirements of acquire and release. 6047 * The L2 cache of one agent can be kept coherent with other agents by using 6048 the MTYPE CC (cache-coherent) with the PTE C-bit for memory local to the L2, 6049 and MTYPE UC (uncached) with the PTE C-bit set for memory not local to the 6050 L2. 6051 6052 * Any local memory cache lines will be automatically invalidated by writes 6053 from CUs associated with other L2 caches, or writes from the CPU, due to 6054 the cache probe caused by coherent requests. Coherent requests are caused 6055 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6056 XGMI, and by PCIe requests that are configured to be coherent requests. 6057 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6058 Subsequent access from the GPU will automatically invalidate or writeback 6059 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6060 * Since all work-groups on the same agent share the same L2, no L2 6061 invalidation or writeback is required for coherence. 6062 * Since local memory reads and writes of work-groups in different agents 6063 access memory using MTYPE CC, no L2 invalidate or writeback is required 6064 for coherence. MTYPE CC causes write through to DRAM and local reads to be 6065 invalidated by remote writes with with the PTE C-bit. 6066 * Since remote memory reads and writes of work-groups in different agents 6067 access memory using MTYPE UC, no L2 invalidate or writeback is required 6068 for coherence. MTYPE UC causes direct accesses to DRAM. 6069 6070 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6071 MTYPE UC (uncached) which bypasses the L2. 6072 6073Scalar memory operations are only used to access memory that is proven to not 6074change during the execution of the kernel dispatch. This includes constant 6075address space and global address space for program scope ``const`` variables. 6076Therefore, the kernel machine code does not have to maintain the scalar cache to 6077ensure it is coherent with the vector caches. The scalar and vector caches are 6078invalidated between kernel dispatches by CP since constant address space data 6079may change between kernel dispatch executions. See 6080:ref:`amdgpu-amdhsa-memory-spaces`. 6081 6082The one exception is if scalar writes are used to spill SGPR registers. In this 6083case the AMDGPU backend ensures the memory location used to spill is never 6084accessed by vector memory operations at the same time. If scalar writes are used 6085then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6086return since the locations may be used for vector memory instructions by a 6087future wavefront that uses the same scratch area, or a function call that 6088creates a frame at the same address, respectively. There is no need for a 6089``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6090 6091For kernarg backing memory: 6092 6093* CP invalidates the L1 cache at the start of each kernel dispatch. 6094* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6095 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6096 cache. This also causes it to be treated as non-volatile and so is not 6097 invalidated by ``*_vol``. 6098* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6099 so the L2 cache will be coherent with the CPU and other agents. 6100 6101Scratch backing memory (which is used for the private address space) is accessed 6102with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6103only accessed by a single thread, and is always write-before-read, there is 6104never a need to invalidate these entries from the L1 cache. Hence all cache 6105invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6106 6107The code sequences used to implement the memory model for GFX90A are defined 6108in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6109 6110 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6111 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6112 6113 ============ ============ ============== ========== ================================ 6114 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6115 Ordering Sync Scope Address GFX90A 6116 Space 6117 ============ ============ ============== ========== ================================ 6118 **Non-Atomic** 6119 ------------------------------------------------------------------------------------ 6120 load *none* *none* - global - !volatile & !nontemporal 6121 - generic 6122 - private 1. buffer/global/flat_load 6123 - constant 6124 - !volatile & nontemporal 6125 6126 1. buffer/global/flat_load 6127 glc=1 slc=1 6128 6129 - volatile 6130 6131 1. buffer/global/flat_load 6132 glc=1 6133 2. s_waitcnt vmcnt(0) 6134 6135 - Must happen before 6136 any following volatile 6137 global/generic 6138 load/store. 6139 - Ensures that 6140 volatile 6141 operations to 6142 different 6143 addresses will not 6144 be reordered by 6145 hardware. 6146 6147 load *none* *none* - local 1. ds_load 6148 store *none* *none* - global - !volatile & !nontemporal 6149 - generic 6150 - private 1. buffer/global/flat_store 6151 - constant 6152 - !volatile & nontemporal 6153 6154 1. buffer/global/flat_store 6155 glc=1 slc=1 6156 6157 - volatile 6158 6159 1. buffer/global/flat_store 6160 2. s_waitcnt vmcnt(0) 6161 6162 - Must happen before 6163 any following volatile 6164 global/generic 6165 load/store. 6166 - Ensures that 6167 volatile 6168 operations to 6169 different 6170 addresses will not 6171 be reordered by 6172 hardware. 6173 6174 store *none* *none* - local 1. ds_store 6175 **Unordered Atomic** 6176 ------------------------------------------------------------------------------------ 6177 load atomic unordered *any* *any* *Same as non-atomic*. 6178 store atomic unordered *any* *any* *Same as non-atomic*. 6179 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6180 **Monotonic Atomic** 6181 ------------------------------------------------------------------------------------ 6182 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6183 - wavefront - generic 6184 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6185 - generic glc=1 6186 6187 - If not TgSplit execution 6188 mode, omit glc=1. 6189 6190 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6191 - wavefront local address space cannot 6192 - workgroup be used.* 6193 6194 1. ds_load 6195 load atomic monotonic - agent - global 1. buffer/global/flat_load 6196 - generic glc=1 6197 load atomic monotonic - system - global 1. buffer/global/flat_load 6198 - generic glc=1 6199 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6200 - wavefront - generic 6201 - workgroup 6202 - agent 6203 store atomic monotonic - system - global 1. buffer/global/flat_store 6204 - generic 6205 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6206 - wavefront local address space cannot 6207 - workgroup be used.* 6208 6209 1. ds_store 6210 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6211 - wavefront - generic 6212 - workgroup 6213 - agent 6214 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6215 - generic 6216 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6217 - wavefront local address space cannot 6218 - workgroup be used.* 6219 6220 1. ds_atomic 6221 **Acquire Atomic** 6222 ------------------------------------------------------------------------------------ 6223 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6224 - wavefront - local 6225 - generic 6226 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6227 6228 - If not TgSplit execution 6229 mode, omit glc=1. 6230 6231 2. s_waitcnt vmcnt(0) 6232 6233 - If not TgSplit execution 6234 mode, omit. 6235 - Must happen before the 6236 following buffer_wbinvl1_vol. 6237 6238 3. buffer_wbinvl1_vol 6239 6240 - If not TgSplit execution 6241 mode, omit. 6242 - Must happen before 6243 any following 6244 global/generic 6245 load/load 6246 atomic/store/store 6247 atomic/atomicrmw. 6248 - Ensures that 6249 following 6250 loads will not see 6251 stale data. 6252 6253 load atomic acquire - workgroup - local *If TgSplit execution mode, 6254 local address space cannot 6255 be used.* 6256 6257 1. ds_load 6258 2. s_waitcnt lgkmcnt(0) 6259 6260 - If OpenCL, omit. 6261 - Must happen before 6262 any following 6263 global/generic 6264 load/load 6265 atomic/store/store 6266 atomic/atomicrmw. 6267 - Ensures any 6268 following global 6269 data read is no 6270 older than the local load 6271 atomic value being 6272 acquired. 6273 6274 load atomic acquire - workgroup - generic 1. flat_load glc=1 6275 6276 - If not TgSplit execution 6277 mode, omit glc=1. 6278 6279 2. s_waitcnt lgkm/vmcnt(0) 6280 6281 - Use lgkmcnt(0) if not 6282 TgSplit execution mode 6283 and vmcnt(0) if TgSplit 6284 execution mode. 6285 - If OpenCL, omit lgkmcnt(0). 6286 - Must happen before 6287 the following 6288 buffer_wbinvl1_vol and any 6289 following global/generic 6290 load/load 6291 atomic/store/store 6292 atomic/atomicrmw. 6293 - Ensures any 6294 following global 6295 data read is no 6296 older than a local load 6297 atomic value being 6298 acquired. 6299 6300 3. buffer_wbinvl1_vol 6301 6302 - If not TgSplit execution 6303 mode, omit. 6304 - Ensures that 6305 following 6306 loads will not see 6307 stale data. 6308 6309 load atomic acquire - agent - global 1. buffer/global_load 6310 glc=1 6311 2. s_waitcnt vmcnt(0) 6312 6313 - Must happen before 6314 following 6315 buffer_wbinvl1_vol. 6316 - Ensures the load 6317 has completed 6318 before invalidating 6319 the cache. 6320 6321 3. buffer_wbinvl1_vol 6322 6323 - Must happen before 6324 any following 6325 global/generic 6326 load/load 6327 atomic/atomicrmw. 6328 - Ensures that 6329 following 6330 loads will not see 6331 stale global data. 6332 6333 load atomic acquire - system - global 1. buffer/global/flat_load 6334 glc=1 6335 2. s_waitcnt vmcnt(0) 6336 6337 - Must happen before 6338 following 6339 buffer_wbinvl1_vol. 6340 - Ensures the load 6341 has completed 6342 before invalidating 6343 the cache. 6344 6345 3. buffer_wbinvl1_vol 6346 6347 - Must happen before 6348 any following 6349 global/generic 6350 load/load 6351 atomic/atomicrmw. 6352 - Ensures that 6353 following 6354 loads will not see 6355 stale L1 global data. 6356 MTYPE RW and CC memory will 6357 never be stale in L2 due to 6358 the memory probes. 6359 6360 load atomic acquire - agent - generic 1. flat_load glc=1 6361 2. s_waitcnt vmcnt(0) & 6362 lgkmcnt(0) 6363 6364 - If TgSplit execution mode, 6365 omit lgkmcnt(0). 6366 - If OpenCL omit 6367 lgkmcnt(0). 6368 - Must happen before 6369 following 6370 buffer_wbinvl1_vol. 6371 - Ensures the flat_load 6372 has completed 6373 before invalidating 6374 the cache. 6375 6376 3. buffer_wbinvl1_vol 6377 6378 - Must happen before 6379 any following 6380 global/generic 6381 load/load 6382 atomic/atomicrmw. 6383 - Ensures that 6384 following loads 6385 will not see stale 6386 global data. 6387 6388 load atomic acquire - system - generic 1. flat_load glc=1 6389 2. s_waitcnt vmcnt(0) & 6390 lgkmcnt(0) 6391 6392 - If TgSplit execution mode, 6393 omit lgkmcnt(0). 6394 - If OpenCL omit 6395 lgkmcnt(0). 6396 - Must happen before 6397 following 6398 buffer_wbinvl1_vol. 6399 - Ensures the flat_load 6400 has completed 6401 before invalidating 6402 the caches. 6403 6404 3. buffer_wbinvl1_vol 6405 6406 - Must happen before 6407 any following 6408 global/generic 6409 load/load 6410 atomic/atomicrmw. 6411 - Ensures that 6412 following 6413 L1 loads will not see 6414 stale global data. 6415 MTYPE RW and CC memory will 6416 never be stale in L2 due to 6417 the memory probes. 6418 6419 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6420 - wavefront - generic 6421 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6422 - wavefront local address space cannot 6423 be used.* 6424 6425 1. ds_atomic 6426 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6427 2. s_waitcnt vmcnt(0) 6428 6429 - If not TgSplit execution 6430 mode, omit. 6431 - Must happen before the 6432 following buffer_wbinvl1_vol. 6433 - Ensures the atomicrmw 6434 has completed 6435 before invalidating 6436 the cache. 6437 6438 3. buffer_wbinvl1_vol 6439 6440 - If not TgSplit execution 6441 mode, omit. 6442 - Must happen before 6443 any following 6444 global/generic 6445 load/load 6446 atomic/atomicrmw. 6447 - Ensures that 6448 following loads 6449 will not see stale 6450 global data. 6451 6452 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6453 local address space cannot 6454 be used.* 6455 6456 1. ds_atomic 6457 2. s_waitcnt lgkmcnt(0) 6458 6459 - If OpenCL, omit. 6460 - Must happen before 6461 any following 6462 global/generic 6463 load/load 6464 atomic/store/store 6465 atomic/atomicrmw. 6466 - Ensures any 6467 following global 6468 data read is no 6469 older than the local 6470 atomicrmw value 6471 being acquired. 6472 6473 atomicrmw acquire - workgroup - generic 1. flat_atomic 6474 2. s_waitcnt lgkm/vmcnt(0) 6475 6476 - Use lgkmcnt(0) if not 6477 TgSplit execution mode 6478 and vmcnt(0) if TgSplit 6479 execution mode. 6480 - If OpenCL, omit lgkmcnt(0). 6481 - Must happen before 6482 the following 6483 buffer_wbinvl1_vol and 6484 any following 6485 global/generic 6486 load/load 6487 atomic/store/store 6488 atomic/atomicrmw. 6489 - Ensures any 6490 following global 6491 data read is no 6492 older than a local 6493 atomicrmw value 6494 being acquired. 6495 6496 3. buffer_wbinvl1_vol 6497 6498 - If not TgSplit execution 6499 mode, omit. 6500 - Ensures that 6501 following 6502 loads will not see 6503 stale data. 6504 6505 atomicrmw acquire - agent - global 1. buffer/global_atomic 6506 2. s_waitcnt vmcnt(0) 6507 6508 - Must happen before 6509 following 6510 buffer_wbinvl1_vol. 6511 - Ensures the 6512 atomicrmw has 6513 completed before 6514 invalidating the 6515 cache. 6516 6517 3. buffer_wbinvl1_vol 6518 6519 - Must happen before 6520 any following 6521 global/generic 6522 load/load 6523 atomic/atomicrmw. 6524 - Ensures that 6525 following loads 6526 will not see stale 6527 global data. 6528 6529 atomicrmw acquire - system - global 1. buffer/global_atomic 6530 2. s_waitcnt vmcnt(0) 6531 6532 - Must happen before 6533 following 6534 buffer_wbinvl1_vol. 6535 - Ensures the 6536 atomicrmw has 6537 completed before 6538 invalidating the 6539 caches. 6540 6541 3. buffer_wbinvl1_vol 6542 6543 - Must happen before 6544 any following 6545 global/generic 6546 load/load 6547 atomic/atomicrmw. 6548 - Ensures that 6549 following 6550 loads will not see 6551 stale L1 global data. 6552 MTYPE RW and CC L2 memory 6553 never be stale in L2 due to 6554 the memory probes. 6555 6556 atomicrmw acquire - agent - generic 1. flat_atomic 6557 2. s_waitcnt vmcnt(0) & 6558 lgkmcnt(0) 6559 6560 - If TgSplit execution mode, 6561 omit lgkmcnt(0). 6562 - If OpenCL, omit 6563 lgkmcnt(0). 6564 - Must happen before 6565 following 6566 buffer_wbinvl1_vol. 6567 - Ensures the 6568 atomicrmw has 6569 completed before 6570 invalidating the 6571 cache. 6572 6573 3. buffer_wbinvl1_vol 6574 6575 - Must happen before 6576 any following 6577 global/generic 6578 load/load 6579 atomic/atomicrmw. 6580 - Ensures that 6581 following loads 6582 will not see stale 6583 global data. 6584 6585 atomicrmw acquire - system - generic 1. flat_atomic 6586 2. s_waitcnt vmcnt(0) & 6587 lgkmcnt(0) 6588 6589 - If TgSplit execution mode, 6590 omit lgkmcnt(0). 6591 - If OpenCL, omit 6592 lgkmcnt(0). 6593 - Must happen before 6594 following 6595 buffer_wbinvl1_vol. 6596 - Ensures the 6597 atomicrmw has 6598 completed before 6599 invalidating the 6600 caches. 6601 6602 3. buffer_wbinvl1_vol 6603 6604 - Must happen before 6605 any following 6606 global/generic 6607 load/load 6608 atomic/atomicrmw. 6609 - Ensures that 6610 following 6611 loads will not see 6612 stale L1 global data. 6613 MTYPE RW and CC memory will 6614 never be stale in L2 due to 6615 the memory probes. 6616 6617 fence acquire - singlethread *none* *none* 6618 - wavefront 6619 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6620 6621 - Use lgkmcnt(0) if not 6622 TgSplit execution mode 6623 and vmcnt(0) if TgSplit 6624 execution mode. 6625 - If OpenCL and 6626 address space is 6627 not generic, omit 6628 lgkmcnt(0). 6629 - If OpenCL and 6630 address space is 6631 local, omit 6632 vmcnt(0). 6633 - However, since LLVM 6634 currently has no 6635 address space on 6636 the fence need to 6637 conservatively 6638 always generate. If 6639 fence had an 6640 address space then 6641 set to address 6642 space of OpenCL 6643 fence flag, or to 6644 generic if both 6645 local and global 6646 flags are 6647 specified. 6648 - s_waitcnt vmcnt(0) 6649 must happen after 6650 any preceding 6651 global/generic load 6652 atomic/ 6653 atomicrmw 6654 with an equal or 6655 wider sync scope 6656 and memory ordering 6657 stronger than 6658 unordered (this is 6659 termed the 6660 fence-paired-atomic). 6661 - s_waitcnt lgkmcnt(0) 6662 must happen after 6663 any preceding 6664 local/generic load 6665 atomic/atomicrmw 6666 with an equal or 6667 wider sync scope 6668 and memory ordering 6669 stronger than 6670 unordered (this is 6671 termed the 6672 fence-paired-atomic). 6673 - Must happen before 6674 the following 6675 buffer_wbinvl1_vol and 6676 any following 6677 global/generic 6678 load/load 6679 atomic/store/store 6680 atomic/atomicrmw. 6681 - Ensures any 6682 following global 6683 data read is no 6684 older than the 6685 value read by the 6686 fence-paired-atomic. 6687 6688 3. buffer_wbinvl1_vol 6689 6690 - If not TgSplit execution 6691 mode, omit. 6692 - Ensures that 6693 following 6694 loads will not see 6695 stale data. 6696 6697 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6698 vmcnt(0) 6699 6700 - If TgSplit execution mode, 6701 omit lgkmcnt(0). 6702 - If OpenCL and 6703 address space is 6704 not generic, omit 6705 lgkmcnt(0). 6706 - However, since LLVM 6707 currently has no 6708 address space on 6709 the fence need to 6710 conservatively 6711 always generate 6712 (see comment for 6713 previous fence). 6714 - Could be split into 6715 separate s_waitcnt 6716 vmcnt(0) and 6717 s_waitcnt 6718 lgkmcnt(0) to allow 6719 them to be 6720 independently moved 6721 according to the 6722 following rules. 6723 - s_waitcnt vmcnt(0) 6724 must happen after 6725 any preceding 6726 global/generic load 6727 atomic/atomicrmw 6728 with an equal or 6729 wider sync scope 6730 and memory ordering 6731 stronger than 6732 unordered (this is 6733 termed the 6734 fence-paired-atomic). 6735 - s_waitcnt lgkmcnt(0) 6736 must happen after 6737 any preceding 6738 local/generic load 6739 atomic/atomicrmw 6740 with an equal or 6741 wider sync scope 6742 and memory ordering 6743 stronger than 6744 unordered (this is 6745 termed the 6746 fence-paired-atomic). 6747 - Must happen before 6748 the following 6749 buffer_wbinvl1_vol. 6750 - Ensures that the 6751 fence-paired atomic 6752 has completed 6753 before invalidating 6754 the 6755 cache. Therefore 6756 any following 6757 locations read must 6758 be no older than 6759 the value read by 6760 the 6761 fence-paired-atomic. 6762 6763 2. buffer_wbinvl1_vol 6764 6765 - Must happen before any 6766 following global/generic 6767 load/load 6768 atomic/store/store 6769 atomic/atomicrmw. 6770 - Ensures that 6771 following loads 6772 will not see stale 6773 global data. 6774 6775 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6776 vmcnt(0) 6777 6778 - If TgSplit execution mode, 6779 omit lgkmcnt(0). 6780 - If OpenCL and 6781 address space is 6782 not generic, omit 6783 lgkmcnt(0). 6784 - However, since LLVM 6785 currently has no 6786 address space on 6787 the fence need to 6788 conservatively 6789 always generate 6790 (see comment for 6791 previous fence). 6792 - Could be split into 6793 separate s_waitcnt 6794 vmcnt(0) and 6795 s_waitcnt 6796 lgkmcnt(0) to allow 6797 them to be 6798 independently moved 6799 according to the 6800 following rules. 6801 - s_waitcnt vmcnt(0) 6802 must happen after 6803 any preceding 6804 global/generic load 6805 atomic/atomicrmw 6806 with an equal or 6807 wider sync scope 6808 and memory ordering 6809 stronger than 6810 unordered (this is 6811 termed the 6812 fence-paired-atomic). 6813 - s_waitcnt lgkmcnt(0) 6814 must happen after 6815 any preceding 6816 local/generic load 6817 atomic/atomicrmw 6818 with an equal or 6819 wider sync scope 6820 and memory ordering 6821 stronger than 6822 unordered (this is 6823 termed the 6824 fence-paired-atomic). 6825 - Must happen before 6826 the following 6827 buffer_wbinvl1_vol. 6828 - Ensures that the 6829 fence-paired atomic 6830 has completed 6831 before invalidating 6832 the 6833 cache. Therefore 6834 any following 6835 locations read must 6836 be no older than 6837 the value read by 6838 the 6839 fence-paired-atomic. 6840 6841 2. buffer_wbinvl1_vol 6842 6843 - Must happen before any 6844 following global/generic 6845 load/load 6846 atomic/store/store 6847 atomic/atomicrmw. 6848 - Ensures that 6849 following 6850 loads will not see 6851 stale L1 global data. 6852 MTYPE RW and CC memory will 6853 never be stale in L2 due to 6854 the memory probes. 6855 **Release Atomic** 6856 ------------------------------------------------------------------------------------ 6857 store atomic release - singlethread - global 1. buffer/global/flat_store 6858 - wavefront - generic 6859 store atomic release - singlethread - local *If TgSplit execution mode, 6860 - wavefront local address space cannot 6861 be used.* 6862 6863 1. ds_store 6864 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6865 - generic 6866 - Use lgkmcnt(0) if not 6867 TgSplit execution mode 6868 and vmcnt(0) if TgSplit 6869 execution mode. 6870 - If OpenCL, omit lgkmcnt(0). 6871 - s_waitcnt vmcnt(0) 6872 must happen after 6873 any preceding 6874 global/generic load/store/ 6875 load atomic/store atomic/ 6876 atomicrmw. 6877 - s_waitcnt lgkmcnt(0) 6878 must happen after 6879 any preceding 6880 local/generic 6881 load/store/load 6882 atomic/store 6883 atomic/atomicrmw. 6884 - Must happen before 6885 the following 6886 store. 6887 - Ensures that all 6888 memory operations 6889 have 6890 completed before 6891 performing the 6892 store that is being 6893 released. 6894 6895 2. buffer/global/flat_store 6896 store atomic release - workgroup - local *If TgSplit execution mode, 6897 local address space cannot 6898 be used.* 6899 6900 1. ds_store 6901 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6902 - generic vmcnt(0) 6903 6904 - If TgSplit execution mode, 6905 omit lgkmcnt(0). 6906 - If OpenCL and 6907 address space is 6908 not generic, omit 6909 lgkmcnt(0). 6910 - Could be split into 6911 separate s_waitcnt 6912 vmcnt(0) and 6913 s_waitcnt 6914 lgkmcnt(0) to allow 6915 them to be 6916 independently moved 6917 according to the 6918 following rules. 6919 - s_waitcnt vmcnt(0) 6920 must happen after 6921 any preceding 6922 global/generic 6923 load/store/load 6924 atomic/store 6925 atomic/atomicrmw. 6926 - s_waitcnt lgkmcnt(0) 6927 must happen after 6928 any preceding 6929 local/generic 6930 load/store/load 6931 atomic/store 6932 atomic/atomicrmw. 6933 - Must happen before 6934 the following 6935 store. 6936 - Ensures that all 6937 memory operations 6938 to memory have 6939 completed before 6940 performing the 6941 store that is being 6942 released. 6943 6944 2. buffer/global/flat_store 6945 store atomic release - system - global 1. s_waitcnt lgkmcnt(0) & 6946 - generic vmcnt(0) 6947 6948 - If TgSplit execution mode, 6949 omit lgkmcnt(0). 6950 - If OpenCL and 6951 address space is 6952 not generic, omit 6953 lgkmcnt(0). 6954 - Could be split into 6955 separate s_waitcnt 6956 vmcnt(0) and 6957 s_waitcnt 6958 lgkmcnt(0) to allow 6959 them to be 6960 independently moved 6961 according to the 6962 following rules. 6963 - s_waitcnt vmcnt(0) 6964 must happen after any 6965 preceding 6966 global/generic 6967 load/store/load 6968 atomic/store 6969 atomic/atomicrmw. 6970 - s_waitcnt lgkmcnt(0) 6971 must happen after any 6972 preceding 6973 local/generic 6974 load/store/load 6975 atomic/store 6976 atomic/atomicrmw. 6977 - Must happen before 6978 the following 6979 store. 6980 - Ensures that all 6981 memory operations 6982 to memory and the L2 6983 writeback have 6984 completed before 6985 performing the 6986 store that is being 6987 released. 6988 6989 2. buffer/global/flat_store 6990 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 6991 - wavefront - generic 6992 atomicrmw release - singlethread - local *If TgSplit execution mode, 6993 - wavefront local address space cannot 6994 be used.* 6995 6996 1. ds_atomic 6997 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6998 - generic 6999 - Use lgkmcnt(0) if not 7000 TgSplit execution mode 7001 and vmcnt(0) if TgSplit 7002 execution mode. 7003 - If OpenCL, omit 7004 lgkmcnt(0). 7005 - s_waitcnt vmcnt(0) 7006 must happen after 7007 any preceding 7008 global/generic load/store/ 7009 load atomic/store atomic/ 7010 atomicrmw. 7011 - s_waitcnt lgkmcnt(0) 7012 must happen after 7013 any preceding 7014 local/generic 7015 load/store/load 7016 atomic/store 7017 atomic/atomicrmw. 7018 - Must happen before 7019 the following 7020 atomicrmw. 7021 - Ensures that all 7022 memory operations 7023 have 7024 completed before 7025 performing the 7026 atomicrmw that is 7027 being released. 7028 7029 2. buffer/global/flat_atomic 7030 atomicrmw release - workgroup - local *If TgSplit execution mode, 7031 local address space cannot 7032 be used.* 7033 7034 1. ds_atomic 7035 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7036 - generic vmcnt(0) 7037 7038 - If TgSplit execution mode, 7039 omit lgkmcnt(0). 7040 - If OpenCL, omit 7041 lgkmcnt(0). 7042 - Could be split into 7043 separate s_waitcnt 7044 vmcnt(0) and 7045 s_waitcnt 7046 lgkmcnt(0) to allow 7047 them to be 7048 independently moved 7049 according to the 7050 following rules. 7051 - s_waitcnt vmcnt(0) 7052 must happen after 7053 any preceding 7054 global/generic 7055 load/store/load 7056 atomic/store 7057 atomic/atomicrmw. 7058 - s_waitcnt lgkmcnt(0) 7059 must happen after 7060 any preceding 7061 local/generic 7062 load/store/load 7063 atomic/store 7064 atomic/atomicrmw. 7065 - Must happen before 7066 the following 7067 atomicrmw. 7068 - Ensures that all 7069 memory operations 7070 to global and local 7071 have completed 7072 before performing 7073 the atomicrmw that 7074 is being released. 7075 7076 2. buffer/global/flat_atomic 7077 atomicrmw release - system - global 1. s_waitcnt lgkmcnt(0) & 7078 - generic vmcnt(0) 7079 7080 - If TgSplit execution mode, 7081 omit lgkmcnt(0). 7082 - If OpenCL, omit 7083 lgkmcnt(0). 7084 - Could be split into 7085 separate s_waitcnt 7086 vmcnt(0) and 7087 s_waitcnt 7088 lgkmcnt(0) to allow 7089 them to be 7090 independently moved 7091 according to the 7092 following rules. 7093 - s_waitcnt vmcnt(0) 7094 must happen after 7095 any preceding 7096 global/generic 7097 load/store/load 7098 atomic/store 7099 atomic/atomicrmw. 7100 - s_waitcnt lgkmcnt(0) 7101 must happen after 7102 any preceding 7103 local/generic 7104 load/store/load 7105 atomic/store 7106 atomic/atomicrmw. 7107 - Must happen before 7108 the following 7109 atomicrmw. 7110 - Ensures that all 7111 memory operations 7112 to memory and the L2 7113 writeback have 7114 completed before 7115 performing the 7116 store that is being 7117 released. 7118 7119 2. buffer/global/flat_atomic 7120 fence release - singlethread *none* *none* 7121 - wavefront 7122 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7123 7124 - Use lgkmcnt(0) if not 7125 TgSplit execution mode 7126 and vmcnt(0) if TgSplit 7127 execution mode. 7128 - If OpenCL and 7129 address space is 7130 not generic, omit 7131 lgkmcnt(0). 7132 - If OpenCL and 7133 address space is 7134 local, omit 7135 vmcnt(0). 7136 - However, since LLVM 7137 currently has no 7138 address space on 7139 the fence need to 7140 conservatively 7141 always generate. If 7142 fence had an 7143 address space then 7144 set to address 7145 space of OpenCL 7146 fence flag, or to 7147 generic if both 7148 local and global 7149 flags are 7150 specified. 7151 - s_waitcnt vmcnt(0) 7152 must happen after 7153 any preceding 7154 global/generic 7155 load/store/ 7156 load atomic/store atomic/ 7157 atomicrmw. 7158 - s_waitcnt lgkmcnt(0) 7159 must happen after 7160 any preceding 7161 local/generic 7162 load/load 7163 atomic/store/store 7164 atomic/atomicrmw. 7165 - Must happen before 7166 any following store 7167 atomic/atomicrmw 7168 with an equal or 7169 wider sync scope 7170 and memory ordering 7171 stronger than 7172 unordered (this is 7173 termed the 7174 fence-paired-atomic). 7175 - Ensures that all 7176 memory operations 7177 have 7178 completed before 7179 performing the 7180 following 7181 fence-paired-atomic. 7182 7183 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7184 vmcnt(0) 7185 7186 - If TgSplit execution mode, 7187 omit lgkmcnt(0). 7188 - If OpenCL and 7189 address space is 7190 not generic, omit 7191 lgkmcnt(0). 7192 - If OpenCL and 7193 address space is 7194 local, omit 7195 vmcnt(0). 7196 - However, since LLVM 7197 currently has no 7198 address space on 7199 the fence need to 7200 conservatively 7201 always generate. If 7202 fence had an 7203 address space then 7204 set to address 7205 space of OpenCL 7206 fence flag, or to 7207 generic if both 7208 local and global 7209 flags are 7210 specified. 7211 - Could be split into 7212 separate s_waitcnt 7213 vmcnt(0) and 7214 s_waitcnt 7215 lgkmcnt(0) to allow 7216 them to be 7217 independently moved 7218 according to the 7219 following rules. 7220 - s_waitcnt vmcnt(0) 7221 must happen after 7222 any preceding 7223 global/generic 7224 load/store/load 7225 atomic/store 7226 atomic/atomicrmw. 7227 - s_waitcnt lgkmcnt(0) 7228 must happen after 7229 any preceding 7230 local/generic 7231 load/store/load 7232 atomic/store 7233 atomic/atomicrmw. 7234 - Must happen before 7235 any following store 7236 atomic/atomicrmw 7237 with an equal or 7238 wider sync scope 7239 and memory ordering 7240 stronger than 7241 unordered (this is 7242 termed the 7243 fence-paired-atomic). 7244 - Ensures that all 7245 memory operations 7246 have 7247 completed before 7248 performing the 7249 following 7250 fence-paired-atomic. 7251 7252 fence release - system *none* 1. s_waitcnt lgkmcnt(0) & 7253 vmcnt(0) 7254 7255 - If TgSplit execution mode, 7256 omit lgkmcnt(0). 7257 - If OpenCL and 7258 address space is 7259 not generic, omit 7260 lgkmcnt(0). 7261 - If OpenCL and 7262 address space is 7263 local, omit 7264 vmcnt(0). 7265 - However, since LLVM 7266 currently has no 7267 address space on 7268 the fence need to 7269 conservatively 7270 always generate. If 7271 fence had an 7272 address space then 7273 set to address 7274 space of OpenCL 7275 fence flag, or to 7276 generic if both 7277 local and global 7278 flags are 7279 specified. 7280 - Could be split into 7281 separate s_waitcnt 7282 vmcnt(0) and 7283 s_waitcnt 7284 lgkmcnt(0) to allow 7285 them to be 7286 independently moved 7287 according to the 7288 following rules. 7289 - s_waitcnt vmcnt(0) 7290 must happen after 7291 any preceding 7292 global/generic 7293 load/store/load 7294 atomic/store 7295 atomic/atomicrmw. 7296 - s_waitcnt lgkmcnt(0) 7297 must happen after 7298 any preceding 7299 local/generic 7300 load/store/load 7301 atomic/store 7302 atomic/atomicrmw. 7303 - Must happen before 7304 any following store 7305 atomic/atomicrmw 7306 with an equal or 7307 wider sync scope 7308 and memory ordering 7309 stronger than 7310 unordered (this is 7311 termed the 7312 fence-paired-atomic). 7313 - Ensures that all 7314 memory operations 7315 have 7316 completed before 7317 performing the 7318 following 7319 fence-paired-atomic. 7320 7321 **Acquire-Release Atomic** 7322 ------------------------------------------------------------------------------------ 7323 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7324 - wavefront - generic 7325 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7326 - wavefront local address space cannot 7327 be used.* 7328 7329 1. ds_atomic 7330 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7331 7332 - Use lgkmcnt(0) if not 7333 TgSplit execution mode 7334 and vmcnt(0) if TgSplit 7335 execution mode. 7336 - If OpenCL, omit 7337 lgkmcnt(0). 7338 - Must happen after 7339 any preceding 7340 local/generic 7341 load/store/load 7342 atomic/store 7343 atomic/atomicrmw. 7344 - s_waitcnt vmcnt(0) 7345 must happen after 7346 any preceding 7347 global/generic load/store/ 7348 load atomic/store atomic/ 7349 atomicrmw. 7350 - s_waitcnt lgkmcnt(0) 7351 must happen after 7352 any preceding 7353 local/generic 7354 load/store/load 7355 atomic/store 7356 atomic/atomicrmw. 7357 - Must happen before 7358 the following 7359 atomicrmw. 7360 - Ensures that all 7361 memory operations 7362 have 7363 completed before 7364 performing the 7365 atomicrmw that is 7366 being released. 7367 7368 2. buffer/global_atomic 7369 3. s_waitcnt vmcnt(0) 7370 7371 - If not TgSplit execution 7372 mode, omit. 7373 - Must happen before 7374 the following 7375 buffer_wbinvl1_vol. 7376 - Ensures any 7377 following global 7378 data read is no 7379 older than the 7380 atomicrmw value 7381 being acquired. 7382 7383 4. buffer_wbinvl1_vol 7384 7385 - If not TgSplit execution 7386 mode, omit. 7387 - Ensures that 7388 following 7389 loads will not see 7390 stale data. 7391 7392 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7393 local address space cannot 7394 be used.* 7395 7396 1. ds_atomic 7397 2. s_waitcnt lgkmcnt(0) 7398 7399 - If OpenCL, omit. 7400 - Must happen before 7401 any following 7402 global/generic 7403 load/load 7404 atomic/store/store 7405 atomic/atomicrmw. 7406 - Ensures any 7407 following global 7408 data read is no 7409 older than the local load 7410 atomic value being 7411 acquired. 7412 7413 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7414 7415 - Use lgkmcnt(0) if not 7416 TgSplit execution mode 7417 and vmcnt(0) if TgSplit 7418 execution mode. 7419 - If OpenCL, omit 7420 lgkmcnt(0). 7421 - s_waitcnt vmcnt(0) 7422 must happen after 7423 any preceding 7424 global/generic load/store/ 7425 load atomic/store atomic/ 7426 atomicrmw. 7427 - s_waitcnt lgkmcnt(0) 7428 must happen after 7429 any preceding 7430 local/generic 7431 load/store/load 7432 atomic/store 7433 atomic/atomicrmw. 7434 - Must happen before 7435 the following 7436 atomicrmw. 7437 - Ensures that all 7438 memory operations 7439 have 7440 completed before 7441 performing the 7442 atomicrmw that is 7443 being released. 7444 7445 2. flat_atomic 7446 3. s_waitcnt lgkmcnt(0) & 7447 vmcnt(0) 7448 7449 - If not TgSplit execution 7450 mode, omit vmcnt(0). 7451 - If OpenCL, omit 7452 lgkmcnt(0). 7453 - Must happen before 7454 the following 7455 buffer_wbinvl1_vol and 7456 any following 7457 global/generic 7458 load/load 7459 atomic/store/store 7460 atomic/atomicrmw. 7461 - Ensures any 7462 following global 7463 data read is no 7464 older than a local load 7465 atomic value being 7466 acquired. 7467 7468 3. buffer_wbinvl1_vol 7469 7470 - If not TgSplit execution 7471 mode, omit. 7472 - Ensures that 7473 following 7474 loads will not see 7475 stale data. 7476 7477 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7478 vmcnt(0) 7479 7480 - If TgSplit execution mode, 7481 omit lgkmcnt(0). 7482 - If OpenCL, omit 7483 lgkmcnt(0). 7484 - Could be split into 7485 separate s_waitcnt 7486 vmcnt(0) and 7487 s_waitcnt 7488 lgkmcnt(0) to allow 7489 them to be 7490 independently moved 7491 according to the 7492 following rules. 7493 - s_waitcnt vmcnt(0) 7494 must happen after 7495 any preceding 7496 global/generic 7497 load/store/load 7498 atomic/store 7499 atomic/atomicrmw. 7500 - s_waitcnt lgkmcnt(0) 7501 must happen after 7502 any preceding 7503 local/generic 7504 load/store/load 7505 atomic/store 7506 atomic/atomicrmw. 7507 - Must happen before 7508 the following 7509 atomicrmw. 7510 - Ensures that all 7511 memory operations 7512 to global have 7513 completed before 7514 performing the 7515 atomicrmw that is 7516 being released. 7517 7518 2. buffer/global_atomic 7519 3. s_waitcnt vmcnt(0) 7520 7521 - Must happen before 7522 following 7523 buffer_wbinvl1_vol. 7524 - Ensures the 7525 atomicrmw has 7526 completed before 7527 invalidating the 7528 cache. 7529 7530 4. buffer_wbinvl1_vol 7531 7532 - Must happen before 7533 any following 7534 global/generic 7535 load/load 7536 atomic/atomicrmw. 7537 - Ensures that 7538 following loads 7539 will not see stale 7540 global data. 7541 7542 atomicrmw acq_rel - system - global 1. s_waitcnt lgkmcnt(0) & 7543 vmcnt(0) 7544 7545 - If TgSplit execution mode, 7546 omit lgkmcnt(0). 7547 - If OpenCL, omit 7548 lgkmcnt(0). 7549 - Could be split into 7550 separate s_waitcnt 7551 vmcnt(0) and 7552 s_waitcnt 7553 lgkmcnt(0) to allow 7554 them to be 7555 independently moved 7556 according to the 7557 following rules. 7558 - s_waitcnt vmcnt(0) 7559 must happen after 7560 any preceding 7561 global/generic 7562 load/store/load 7563 atomic/store 7564 atomic/atomicrmw. 7565 - s_waitcnt lgkmcnt(0) 7566 must happen after 7567 any preceding 7568 local/generic 7569 load/store/load 7570 atomic/store 7571 atomic/atomicrmw. 7572 - Must happen before 7573 the following 7574 atomicrmw. 7575 - Ensures that all 7576 memory operations 7577 to global and L2 writeback 7578 have completed before 7579 performing the 7580 atomicrmw that is 7581 being released. 7582 7583 2. buffer/global_atomic 7584 3. s_waitcnt vmcnt(0) 7585 7586 - Must happen before 7587 following 7588 buffer_wbinvl1_vol. 7589 - Ensures the 7590 atomicrmw has 7591 completed before 7592 invalidating the 7593 caches. 7594 7595 4. buffer_wbinvl1_vol 7596 7597 - Must happen before 7598 any following 7599 global/generic 7600 load/load 7601 atomic/atomicrmw. 7602 - Ensures that 7603 following 7604 loads will not see 7605 stale L1 global data. 7606 MTYPE RW and CC memory will 7607 never be stale in L2 due to 7608 the memory probes. 7609 7610 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7611 vmcnt(0) 7612 7613 - If TgSplit execution mode, 7614 omit lgkmcnt(0). 7615 - If OpenCL, omit 7616 lgkmcnt(0). 7617 - Could be split into 7618 separate s_waitcnt 7619 vmcnt(0) and 7620 s_waitcnt 7621 lgkmcnt(0) to allow 7622 them to be 7623 independently moved 7624 according to the 7625 following rules. 7626 - s_waitcnt vmcnt(0) 7627 must happen after 7628 any preceding 7629 global/generic 7630 load/store/load 7631 atomic/store 7632 atomic/atomicrmw. 7633 - s_waitcnt lgkmcnt(0) 7634 must happen after 7635 any preceding 7636 local/generic 7637 load/store/load 7638 atomic/store 7639 atomic/atomicrmw. 7640 - Must happen before 7641 the following 7642 atomicrmw. 7643 - Ensures that all 7644 memory operations 7645 to global have 7646 completed before 7647 performing the 7648 atomicrmw that is 7649 being released. 7650 7651 2. flat_atomic 7652 3. s_waitcnt vmcnt(0) & 7653 lgkmcnt(0) 7654 7655 - If TgSplit execution mode, 7656 omit lgkmcnt(0). 7657 - If OpenCL, omit 7658 lgkmcnt(0). 7659 - Must happen before 7660 following 7661 buffer_wbinvl1_vol. 7662 - Ensures the 7663 atomicrmw has 7664 completed before 7665 invalidating the 7666 cache. 7667 7668 4. buffer_wbinvl1_vol 7669 7670 - Must happen before 7671 any following 7672 global/generic 7673 load/load 7674 atomic/atomicrmw. 7675 - Ensures that 7676 following loads 7677 will not see stale 7678 global data. 7679 7680 atomicrmw acq_rel - system - generic 1. s_waitcnt lgkmcnt(0) & 7681 vmcnt(0) 7682 7683 - If TgSplit execution mode, 7684 omit lgkmcnt(0). 7685 - If OpenCL, omit 7686 lgkmcnt(0). 7687 - Could be split into 7688 separate s_waitcnt 7689 vmcnt(0) and 7690 s_waitcnt 7691 lgkmcnt(0) to allow 7692 them to be 7693 independently moved 7694 according to the 7695 following rules. 7696 - s_waitcnt vmcnt(0) 7697 must happen after 7698 any preceding 7699 global/generic 7700 load/store/load 7701 atomic/store 7702 atomic/atomicrmw. 7703 - s_waitcnt lgkmcnt(0) 7704 must happen after 7705 any preceding 7706 local/generic 7707 load/store/load 7708 atomic/store 7709 atomic/atomicrmw. 7710 - Must happen before 7711 the following 7712 atomicrmw. 7713 - Ensures that all 7714 memory operations 7715 to global and L2 writeback 7716 have completed before 7717 performing the 7718 atomicrmw that is 7719 being released. 7720 7721 2. flat_atomic 7722 3. s_waitcnt vmcnt(0) & 7723 lgkmcnt(0) 7724 7725 - If TgSplit execution mode, 7726 omit lgkmcnt(0). 7727 - If OpenCL, omit 7728 lgkmcnt(0). 7729 - Must happen before 7730 following 7731 buffer_wbinvl1_vol. 7732 - Ensures the 7733 atomicrmw has 7734 completed before 7735 invalidating the 7736 caches. 7737 7738 4. buffer_wbinvl1_vol 7739 7740 - Must happen before 7741 any following 7742 global/generic 7743 load/load 7744 atomic/atomicrmw. 7745 - Ensures that 7746 following 7747 loads will not see 7748 stale L1 global data. 7749 MTYPE RW and CC memory will 7750 never be stale in L2 due to 7751 the memory probes. 7752 7753 fence acq_rel - singlethread *none* *none* 7754 - wavefront 7755 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7756 7757 - Use lgkmcnt(0) if not 7758 TgSplit execution mode 7759 and vmcnt(0) if TgSplit 7760 execution mode. 7761 - If OpenCL and 7762 address space is 7763 not generic, omit 7764 lgkmcnt(0). 7765 - If OpenCL and 7766 address space is 7767 local, omit 7768 vmcnt(0). 7769 - However, 7770 since LLVM 7771 currently has no 7772 address space on 7773 the fence need to 7774 conservatively 7775 always generate 7776 (see comment for 7777 previous fence). 7778 - s_waitcnt vmcnt(0) 7779 must happen after 7780 any preceding 7781 global/generic 7782 load/store/ 7783 load atomic/store atomic/ 7784 atomicrmw. 7785 - s_waitcnt lgkmcnt(0) 7786 must happen after 7787 any preceding 7788 local/generic 7789 load/load 7790 atomic/store/store 7791 atomic/atomicrmw. 7792 - Must happen before 7793 any following 7794 global/generic 7795 load/load 7796 atomic/store/store 7797 atomic/atomicrmw. 7798 - Ensures that all 7799 memory operations 7800 have 7801 completed before 7802 performing any 7803 following global 7804 memory operations. 7805 - Ensures that the 7806 preceding 7807 local/generic load 7808 atomic/atomicrmw 7809 with an equal or 7810 wider sync scope 7811 and memory ordering 7812 stronger than 7813 unordered (this is 7814 termed the 7815 acquire-fence-paired-atomic) 7816 has completed 7817 before following 7818 global memory 7819 operations. This 7820 satisfies the 7821 requirements of 7822 acquire. 7823 - Ensures that all 7824 previous memory 7825 operations have 7826 completed before a 7827 following 7828 local/generic store 7829 atomic/atomicrmw 7830 with an equal or 7831 wider sync scope 7832 and memory ordering 7833 stronger than 7834 unordered (this is 7835 termed the 7836 release-fence-paired-atomic). 7837 This satisfies the 7838 requirements of 7839 release. 7840 - Must happen before 7841 the following 7842 buffer_wbinvl1_vol. 7843 - Ensures that the 7844 acquire-fence-paired 7845 atomic has completed 7846 before invalidating 7847 the 7848 cache. Therefore 7849 any following 7850 locations read must 7851 be no older than 7852 the value read by 7853 the 7854 acquire-fence-paired-atomic. 7855 7856 3. buffer_wbinvl1_vol 7857 7858 - If not TgSplit execution 7859 mode, omit. 7860 - Ensures that 7861 following 7862 loads will not see 7863 stale data. 7864 7865 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7866 vmcnt(0) 7867 7868 - If TgSplit execution mode, 7869 omit lgkmcnt(0). 7870 - If OpenCL and 7871 address space is 7872 not generic, omit 7873 lgkmcnt(0). 7874 - However, since LLVM 7875 currently has no 7876 address space on 7877 the fence need to 7878 conservatively 7879 always generate 7880 (see comment for 7881 previous fence). 7882 - Could be split into 7883 separate s_waitcnt 7884 vmcnt(0) and 7885 s_waitcnt 7886 lgkmcnt(0) to allow 7887 them to be 7888 independently moved 7889 according to the 7890 following rules. 7891 - s_waitcnt vmcnt(0) 7892 must happen after 7893 any preceding 7894 global/generic 7895 load/store/load 7896 atomic/store 7897 atomic/atomicrmw. 7898 - s_waitcnt lgkmcnt(0) 7899 must happen after 7900 any preceding 7901 local/generic 7902 load/store/load 7903 atomic/store 7904 atomic/atomicrmw. 7905 - Must happen before 7906 the following 7907 buffer_wbinvl1_vol. 7908 - Ensures that the 7909 preceding 7910 global/local/generic 7911 load 7912 atomic/atomicrmw 7913 with an equal or 7914 wider sync scope 7915 and memory ordering 7916 stronger than 7917 unordered (this is 7918 termed the 7919 acquire-fence-paired-atomic) 7920 has completed 7921 before invalidating 7922 the cache. This 7923 satisfies the 7924 requirements of 7925 acquire. 7926 - Ensures that all 7927 previous memory 7928 operations have 7929 completed before a 7930 following 7931 global/local/generic 7932 store 7933 atomic/atomicrmw 7934 with an equal or 7935 wider sync scope 7936 and memory ordering 7937 stronger than 7938 unordered (this is 7939 termed the 7940 release-fence-paired-atomic). 7941 This satisfies the 7942 requirements of 7943 release. 7944 7945 2. buffer_wbinvl1_vol 7946 7947 - Must happen before 7948 any following 7949 global/generic 7950 load/load 7951 atomic/store/store 7952 atomic/atomicrmw. 7953 - Ensures that 7954 following loads 7955 will not see stale 7956 global data. This 7957 satisfies the 7958 requirements of 7959 acquire. 7960 7961 fence acq_rel - system *none* 1. s_waitcnt lgkmcnt(0) & 7962 vmcnt(0) 7963 7964 - If TgSplit execution mode, 7965 omit lgkmcnt(0). 7966 - If OpenCL and 7967 address space is 7968 not generic, omit 7969 lgkmcnt(0). 7970 - However, since LLVM 7971 currently has no 7972 address space on 7973 the fence need to 7974 conservatively 7975 always generate 7976 (see comment for 7977 previous fence). 7978 - Could be split into 7979 separate s_waitcnt 7980 vmcnt(0) and 7981 s_waitcnt 7982 lgkmcnt(0) to allow 7983 them to be 7984 independently moved 7985 according to the 7986 following rules. 7987 - s_waitcnt vmcnt(0) 7988 must happen after 7989 any preceding 7990 global/generic 7991 load/store/load 7992 atomic/store 7993 atomic/atomicrmw. 7994 - s_waitcnt lgkmcnt(0) 7995 must happen after 7996 any preceding 7997 local/generic 7998 load/store/load 7999 atomic/store 8000 atomic/atomicrmw. 8001 - Must happen before 8002 the following 8003 buffer_wbinvl1_vol. 8004 - Ensures that the 8005 preceding 8006 global/local/generic 8007 load 8008 atomic/atomicrmw 8009 with an equal or 8010 wider sync scope 8011 and memory ordering 8012 stronger than 8013 unordered (this is 8014 termed the 8015 acquire-fence-paired-atomic) 8016 has completed 8017 before invalidating 8018 the cache. This 8019 satisfies the 8020 requirements of 8021 acquire. 8022 - Ensures that all 8023 previous memory 8024 operations have 8025 completed before a 8026 following 8027 global/local/generic 8028 store 8029 atomic/atomicrmw 8030 with an equal or 8031 wider sync scope 8032 and memory ordering 8033 stronger than 8034 unordered (this is 8035 termed the 8036 release-fence-paired-atomic). 8037 This satisfies the 8038 requirements of 8039 release. 8040 8041 2. buffer_wbinvl1_vol 8042 8043 - Must happen before 8044 any following 8045 global/generic 8046 load/load 8047 atomic/store/store 8048 atomic/atomicrmw. 8049 - Ensures that 8050 following 8051 loads will not see 8052 stale L1 global data. 8053 MTYPE RW and CC memory will 8054 never be stale in L2 due to 8055 the memory probes. 8056 8057 **Sequential Consistent Atomic** 8058 ------------------------------------------------------------------------------------ 8059 load atomic seq_cst - singlethread - global *Same as corresponding 8060 - wavefront - local load atomic acquire, 8061 - generic except must generated 8062 all instructions even 8063 for OpenCL.* 8064 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8065 - generic 8066 - Use lgkmcnt(0) if not 8067 TgSplit execution mode 8068 and vmcnt(0) if TgSplit 8069 execution mode. 8070 - s_waitcnt lgkmcnt(0) must 8071 happen after 8072 preceding 8073 local/generic load 8074 atomic/store 8075 atomic/atomicrmw 8076 with memory 8077 ordering of seq_cst 8078 and with equal or 8079 wider sync scope. 8080 (Note that seq_cst 8081 fences have their 8082 own s_waitcnt 8083 lgkmcnt(0) and so do 8084 not need to be 8085 considered.) 8086 - s_waitcnt vmcnt(0) 8087 must happen after 8088 preceding 8089 global/generic load 8090 atomic/store 8091 atomic/atomicrmw 8092 with memory 8093 ordering of seq_cst 8094 and with equal or 8095 wider sync scope. 8096 (Note that seq_cst 8097 fences have their 8098 own s_waitcnt 8099 vmcnt(0) and so do 8100 not need to be 8101 considered.) 8102 - Ensures any 8103 preceding 8104 sequential 8105 consistent global/local 8106 memory instructions 8107 have completed 8108 before executing 8109 this sequentially 8110 consistent 8111 instruction. This 8112 prevents reordering 8113 a seq_cst store 8114 followed by a 8115 seq_cst load. (Note 8116 that seq_cst is 8117 stronger than 8118 acquire/release as 8119 the reordering of 8120 load acquire 8121 followed by a store 8122 release is 8123 prevented by the 8124 s_waitcnt of 8125 the release, but 8126 there is nothing 8127 preventing a store 8128 release followed by 8129 load acquire from 8130 completing out of 8131 order. The s_waitcnt 8132 could be placed after 8133 seq_store or before 8134 the seq_load. We 8135 choose the load to 8136 make the s_waitcnt be 8137 as late as possible 8138 so that the store 8139 may have already 8140 completed.) 8141 8142 2. *Following 8143 instructions same as 8144 corresponding load 8145 atomic acquire, 8146 except must generated 8147 all instructions even 8148 for OpenCL.* 8149 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8150 local address space cannot 8151 be used.* 8152 8153 *Same as corresponding 8154 load atomic acquire, 8155 except must generated 8156 all instructions even 8157 for OpenCL.* 8158 8159 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8160 - system - generic vmcnt(0) 8161 8162 - If TgSplit execution mode, 8163 omit lgkmcnt(0). 8164 - Could be split into 8165 separate s_waitcnt 8166 vmcnt(0) 8167 and s_waitcnt 8168 lgkmcnt(0) to allow 8169 them to be 8170 independently moved 8171 according to the 8172 following rules. 8173 - s_waitcnt lgkmcnt(0) 8174 must happen after 8175 preceding 8176 global/generic load 8177 atomic/store 8178 atomic/atomicrmw 8179 with memory 8180 ordering of seq_cst 8181 and with equal or 8182 wider sync scope. 8183 (Note that seq_cst 8184 fences have their 8185 own s_waitcnt 8186 lgkmcnt(0) and so do 8187 not need to be 8188 considered.) 8189 - s_waitcnt vmcnt(0) 8190 must happen after 8191 preceding 8192 global/generic load 8193 atomic/store 8194 atomic/atomicrmw 8195 with memory 8196 ordering of seq_cst 8197 and with equal or 8198 wider sync scope. 8199 (Note that seq_cst 8200 fences have their 8201 own s_waitcnt 8202 vmcnt(0) and so do 8203 not need to be 8204 considered.) 8205 - Ensures any 8206 preceding 8207 sequential 8208 consistent global 8209 memory instructions 8210 have completed 8211 before executing 8212 this sequentially 8213 consistent 8214 instruction. This 8215 prevents reordering 8216 a seq_cst store 8217 followed by a 8218 seq_cst load. (Note 8219 that seq_cst is 8220 stronger than 8221 acquire/release as 8222 the reordering of 8223 load acquire 8224 followed by a store 8225 release is 8226 prevented by the 8227 s_waitcnt of 8228 the release, but 8229 there is nothing 8230 preventing a store 8231 release followed by 8232 load acquire from 8233 completing out of 8234 order. The s_waitcnt 8235 could be placed after 8236 seq_store or before 8237 the seq_load. We 8238 choose the load to 8239 make the s_waitcnt be 8240 as late as possible 8241 so that the store 8242 may have already 8243 completed.) 8244 8245 2. *Following 8246 instructions same as 8247 corresponding load 8248 atomic acquire, 8249 except must generated 8250 all instructions even 8251 for OpenCL.* 8252 store atomic seq_cst - singlethread - global *Same as corresponding 8253 - wavefront - local store atomic release, 8254 - workgroup - generic except must generated 8255 - agent all instructions even 8256 - system for OpenCL.* 8257 atomicrmw seq_cst - singlethread - global *Same as corresponding 8258 - wavefront - local atomicrmw acq_rel, 8259 - workgroup - generic except must generated 8260 - agent all instructions even 8261 - system for OpenCL.* 8262 fence seq_cst - singlethread *none* *Same as corresponding 8263 - wavefront fence acq_rel, 8264 - workgroup except must generated 8265 - agent all instructions even 8266 - system for OpenCL.* 8267 ============ ============ ============== ========== ================================ 8268 8269.. _amdgpu-amdhsa-memory-model-gfx10: 8270 8271Memory Model GFX10 8272++++++++++++++++++ 8273 8274For GFX10: 8275 8276* Each agent has multiple shader arrays (SA). 8277* Each SA has multiple work-group processors (WGP). 8278* Each WGP has multiple compute units (CU). 8279* Each CU has multiple SIMDs that execute wavefronts. 8280* The wavefronts for a single work-group are executed in the same 8281 WGP. In CU wavefront execution mode the wavefronts may be executed by 8282 different SIMDs in the same CU. In WGP wavefront execution mode the 8283 wavefronts may be executed by different SIMDs in different CUs in the same 8284 WGP. 8285* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8286 executing on it. 8287* All LDS operations of a WGP are performed as wavefront wide operations in a 8288 global order and involve no caching. Completion is reported to a wavefront in 8289 execution order. 8290* The LDS memory has multiple request queues shared by the SIMDs of a 8291 WGP. Therefore, the LDS operations performed by different wavefronts of a 8292 work-group can be reordered relative to each other, which can result in 8293 reordering the visibility of vector memory operations with respect to LDS 8294 operations of other wavefronts in the same work-group. A ``s_waitcnt 8295 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8296 vector memory operations between wavefronts of a work-group, but not between 8297 operations performed by the same wavefront. 8298* The vector memory operations are performed as wavefront wide operations. 8299 Completion of load/store/sample operations are reported to a wavefront in 8300 execution order of other load/store/sample operations performed by that 8301 wavefront. 8302* The vector memory operations access a vector L0 cache. There is a single L0 8303 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8304 special action is required for coherence between the lanes of a single 8305 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8306 wavefronts executing in the same work-group as they may be executing on SIMDs 8307 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8308 required for coherence between wavefronts executing in different work-groups 8309 as they may be executing on different WGPs. 8310* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8311 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8312 operations are used in a restricted way so do not impact the memory model. See 8313 :ref:`amdgpu-amdhsa-memory-spaces`. 8314* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8315 the same SA. Therefore, no special action is required for coherence between 8316 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8317 required for coherence between wavefronts executing in different work-groups 8318 as they may be executing on different SAs that access different L1s. 8319* The L1 caches have independent quadrants to service disjoint ranges of virtual 8320 addresses. 8321* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8322 vector and scalar memory operations performed by different wavefronts, whether 8323 executing in the same or different work-groups (which may be executing on 8324 different CUs accessing different L0s), can be reordered relative to each 8325 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8326 synchronization between vector memory operations of different wavefronts. It 8327 ensures a previous vector memory operation has completed before executing a 8328 subsequent vector memory or LDS operation and so can be used to meet the 8329 requirements of acquire, release and sequential consistency. 8330* The L1 caches use an L2 cache shared by all SAs on the same agent. 8331* The L2 cache has independent channels to service disjoint ranges of virtual 8332 addresses. 8333* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8334 quadrant has a separate request queue per L2 channel. Therefore, the vector 8335 and scalar memory operations performed by wavefronts executing in different 8336 work-groups (which may be executing on different SAs) of an agent can be 8337 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8338 required to ensure synchronization between vector memory operations of 8339 different SAs. It ensures a previous vector memory operation has completed 8340 before executing a subsequent vector memory and so can be used to meet the 8341 requirements of acquire, release and sequential consistency. 8342* The L2 cache can be kept coherent with other agents on some targets, or ranges 8343 of virtual addresses can be set up to bypass it to ensure system coherence. 8344 8345Scalar memory operations are only used to access memory that is proven to not 8346change during the execution of the kernel dispatch. This includes constant 8347address space and global address space for program scope ``const`` variables. 8348Therefore, the kernel machine code does not have to maintain the scalar cache to 8349ensure it is coherent with the vector caches. The scalar and vector caches are 8350invalidated between kernel dispatches by CP since constant address space data 8351may change between kernel dispatch executions. See 8352:ref:`amdgpu-amdhsa-memory-spaces`. 8353 8354The one exception is if scalar writes are used to spill SGPR registers. In this 8355case the AMDGPU backend ensures the memory location used to spill is never 8356accessed by vector memory operations at the same time. If scalar writes are used 8357then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8358return since the locations may be used for vector memory instructions by a 8359future wavefront that uses the same scratch area, or a function call that 8360creates a frame at the same address, respectively. There is no need for a 8361``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8362 8363For kernarg backing memory: 8364 8365* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8366* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8367 needing to invalidate the L2 cache. 8368* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8369 so the L2 cache will be coherent with the CPU and other agents. 8370 8371Scratch backing memory (which is used for the private address space) is accessed 8372with MTYPE NC (non-coherent). Since the private address space is only accessed 8373by a single thread, and is always write-before-read, there is never a need to 8374invalidate these entries from the L0 or L1 caches. 8375 8376Wavefronts are executed in native mode with in-order reporting of loads and 8377sample instructions. In this mode vmcnt reports completion of load, atomic with 8378return and sample instructions in order, and the vscnt reports the completion of 8379store and atomic without return in order. See ``MEM_ORDERED`` field in 8380:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8381 8382Wavefronts can be executed in WGP or CU wavefront execution mode: 8383 8384* In WGP wavefront execution mode the wavefronts of a work-group are executed 8385 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8386 CU L0 caches is required for work-group synchronization. Also accesses to L1 8387 at work-group scope need to be explicitly ordered as the accesses from 8388 different CUs are not ordered. 8389* In CU wavefront execution mode the wavefronts of a work-group are executed on 8390 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8391 the work-group access the same L0 which in turn ensures L1 accesses are 8392 ordered and so do not require explicit management of the caches for 8393 work-group synchronization. 8394 8395See ``WGP_MODE`` field in 8396:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8397:ref:`amdgpu-target-features`. 8398 8399The code sequences used to implement the memory model for GFX10 are defined in 8400table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8401 8402 .. table:: AMDHSA Memory Model Code Sequences GFX10 8403 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8404 8405 ============ ============ ============== ========== ================================ 8406 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8407 Ordering Sync Scope Address GFX10 8408 Space 8409 ============ ============ ============== ========== ================================ 8410 **Non-Atomic** 8411 ------------------------------------------------------------------------------------ 8412 load *none* *none* - global - !volatile & !nontemporal 8413 - generic 8414 - private 1. buffer/global/flat_load 8415 - constant 8416 - !volatile & nontemporal 8417 8418 1. buffer/global/flat_load 8419 slc=1 8420 8421 - volatile 8422 8423 1. buffer/global/flat_load 8424 glc=1 dlc=1 8425 2. s_waitcnt vmcnt(0) 8426 8427 - Must happen before 8428 any following volatile 8429 global/generic 8430 load/store. 8431 - Ensures that 8432 volatile 8433 operations to 8434 different 8435 addresses will not 8436 be reordered by 8437 hardware. 8438 8439 load *none* *none* - local 1. ds_load 8440 store *none* *none* - global - !volatile & !nontemporal 8441 - generic 8442 - private 1. buffer/global/flat_store 8443 - constant 8444 - !volatile & nontemporal 8445 8446 1. buffer/global/flat_store 8447 slc=1 8448 8449 - volatile 8450 8451 1. buffer/global/flat_store 8452 2. s_waitcnt vscnt(0) 8453 8454 - Must happen before 8455 any following volatile 8456 global/generic 8457 load/store. 8458 - Ensures that 8459 volatile 8460 operations to 8461 different 8462 addresses will not 8463 be reordered by 8464 hardware. 8465 8466 store *none* *none* - local 1. ds_store 8467 **Unordered Atomic** 8468 ------------------------------------------------------------------------------------ 8469 load atomic unordered *any* *any* *Same as non-atomic*. 8470 store atomic unordered *any* *any* *Same as non-atomic*. 8471 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8472 **Monotonic Atomic** 8473 ------------------------------------------------------------------------------------ 8474 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8475 - wavefront - generic 8476 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8477 - generic glc=1 8478 8479 - If CU wavefront execution 8480 mode, omit glc=1. 8481 8482 load atomic monotonic - singlethread - local 1. ds_load 8483 - wavefront 8484 - workgroup 8485 load atomic monotonic - agent - global 1. buffer/global/flat_load 8486 - system - generic glc=1 dlc=1 8487 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8488 - wavefront - generic 8489 - workgroup 8490 - agent 8491 - system 8492 store atomic monotonic - singlethread - local 1. ds_store 8493 - wavefront 8494 - workgroup 8495 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8496 - wavefront - generic 8497 - workgroup 8498 - agent 8499 - system 8500 atomicrmw monotonic - singlethread - local 1. ds_atomic 8501 - wavefront 8502 - workgroup 8503 **Acquire Atomic** 8504 ------------------------------------------------------------------------------------ 8505 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8506 - wavefront - local 8507 - generic 8508 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8509 8510 - If CU wavefront execution 8511 mode, omit glc=1. 8512 8513 2. s_waitcnt vmcnt(0) 8514 8515 - If CU wavefront execution 8516 mode, omit. 8517 - Must happen before 8518 the following buffer_gl0_inv 8519 and before any following 8520 global/generic 8521 load/load 8522 atomic/store/store 8523 atomic/atomicrmw. 8524 8525 3. buffer_gl0_inv 8526 8527 - If CU wavefront execution 8528 mode, omit. 8529 - Ensures that 8530 following 8531 loads will not see 8532 stale data. 8533 8534 load atomic acquire - workgroup - local 1. ds_load 8535 2. s_waitcnt lgkmcnt(0) 8536 8537 - If OpenCL, omit. 8538 - Must happen before 8539 the following buffer_gl0_inv 8540 and before any following 8541 global/generic load/load 8542 atomic/store/store 8543 atomic/atomicrmw. 8544 - Ensures any 8545 following global 8546 data read is no 8547 older than the local load 8548 atomic value being 8549 acquired. 8550 8551 3. buffer_gl0_inv 8552 8553 - If CU wavefront execution 8554 mode, omit. 8555 - If OpenCL, omit. 8556 - Ensures that 8557 following 8558 loads will not see 8559 stale data. 8560 8561 load atomic acquire - workgroup - generic 1. flat_load glc=1 8562 8563 - If CU wavefront execution 8564 mode, omit glc=1. 8565 8566 2. s_waitcnt lgkmcnt(0) & 8567 vmcnt(0) 8568 8569 - If CU wavefront execution 8570 mode, omit vmcnt(0). 8571 - If OpenCL, omit 8572 lgkmcnt(0). 8573 - Must happen before 8574 the following 8575 buffer_gl0_inv and any 8576 following global/generic 8577 load/load 8578 atomic/store/store 8579 atomic/atomicrmw. 8580 - Ensures any 8581 following global 8582 data read is no 8583 older than a local load 8584 atomic value being 8585 acquired. 8586 8587 3. buffer_gl0_inv 8588 8589 - If CU wavefront execution 8590 mode, omit. 8591 - Ensures that 8592 following 8593 loads will not see 8594 stale data. 8595 8596 load atomic acquire - agent - global 1. buffer/global_load 8597 - system glc=1 dlc=1 8598 2. s_waitcnt vmcnt(0) 8599 8600 - Must happen before 8601 following 8602 buffer_gl*_inv. 8603 - Ensures the load 8604 has completed 8605 before invalidating 8606 the caches. 8607 8608 3. buffer_gl0_inv; 8609 buffer_gl1_inv 8610 8611 - Must happen before 8612 any following 8613 global/generic 8614 load/load 8615 atomic/atomicrmw. 8616 - Ensures that 8617 following 8618 loads will not see 8619 stale global data. 8620 8621 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8622 - system 2. s_waitcnt vmcnt(0) & 8623 lgkmcnt(0) 8624 8625 - If OpenCL omit 8626 lgkmcnt(0). 8627 - Must happen before 8628 following 8629 buffer_gl*_invl. 8630 - Ensures the flat_load 8631 has completed 8632 before invalidating 8633 the caches. 8634 8635 3. buffer_gl0_inv; 8636 buffer_gl1_inv 8637 8638 - Must happen before 8639 any following 8640 global/generic 8641 load/load 8642 atomic/atomicrmw. 8643 - Ensures that 8644 following loads 8645 will not see stale 8646 global data. 8647 8648 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8649 - wavefront - local 8650 - generic 8651 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8652 2. s_waitcnt vm/vscnt(0) 8653 8654 - If CU wavefront execution 8655 mode, omit. 8656 - Use vmcnt(0) if atomic with 8657 return and vscnt(0) if 8658 atomic with no-return. 8659 - Must happen before 8660 the following buffer_gl0_inv 8661 and before any following 8662 global/generic 8663 load/load 8664 atomic/store/store 8665 atomic/atomicrmw. 8666 8667 3. buffer_gl0_inv 8668 8669 - If CU wavefront execution 8670 mode, omit. 8671 - Ensures that 8672 following 8673 loads will not see 8674 stale data. 8675 8676 atomicrmw acquire - workgroup - local 1. ds_atomic 8677 2. s_waitcnt lgkmcnt(0) 8678 8679 - If OpenCL, omit. 8680 - Must happen before 8681 the following 8682 buffer_gl0_inv. 8683 - Ensures any 8684 following global 8685 data read is no 8686 older than the local 8687 atomicrmw value 8688 being acquired. 8689 8690 3. buffer_gl0_inv 8691 8692 - If OpenCL omit. 8693 - Ensures that 8694 following 8695 loads will not see 8696 stale data. 8697 8698 atomicrmw acquire - workgroup - generic 1. flat_atomic 8699 2. s_waitcnt lgkmcnt(0) & 8700 vm/vscnt(0) 8701 8702 - If CU wavefront execution 8703 mode, omit vm/vscnt(0). 8704 - If OpenCL, omit lgkmcnt(0). 8705 - Use vmcnt(0) if atomic with 8706 return and vscnt(0) if 8707 atomic with no-return. 8708 - Must happen before 8709 the following 8710 buffer_gl0_inv. 8711 - Ensures any 8712 following global 8713 data read is no 8714 older than a local 8715 atomicrmw value 8716 being acquired. 8717 8718 3. buffer_gl0_inv 8719 8720 - If CU wavefront execution 8721 mode, omit. 8722 - Ensures that 8723 following 8724 loads will not see 8725 stale data. 8726 8727 atomicrmw acquire - agent - global 1. buffer/global_atomic 8728 - system 2. s_waitcnt vm/vscnt(0) 8729 8730 - Use vmcnt(0) if atomic with 8731 return and vscnt(0) if 8732 atomic with no-return. 8733 - Must happen before 8734 following 8735 buffer_gl*_inv. 8736 - Ensures the 8737 atomicrmw has 8738 completed before 8739 invalidating the 8740 caches. 8741 8742 3. buffer_gl0_inv; 8743 buffer_gl1_inv 8744 8745 - Must happen before 8746 any following 8747 global/generic 8748 load/load 8749 atomic/atomicrmw. 8750 - Ensures that 8751 following loads 8752 will not see stale 8753 global data. 8754 8755 atomicrmw acquire - agent - generic 1. flat_atomic 8756 - system 2. s_waitcnt vm/vscnt(0) & 8757 lgkmcnt(0) 8758 8759 - If OpenCL, omit 8760 lgkmcnt(0). 8761 - Use vmcnt(0) if atomic with 8762 return and vscnt(0) if 8763 atomic with no-return. 8764 - Must happen before 8765 following 8766 buffer_gl*_inv. 8767 - Ensures the 8768 atomicrmw has 8769 completed before 8770 invalidating the 8771 caches. 8772 8773 3. buffer_gl0_inv; 8774 buffer_gl1_inv 8775 8776 - Must happen before 8777 any following 8778 global/generic 8779 load/load 8780 atomic/atomicrmw. 8781 - Ensures that 8782 following loads 8783 will not see stale 8784 global data. 8785 8786 fence acquire - singlethread *none* *none* 8787 - wavefront 8788 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 8789 vmcnt(0) & vscnt(0) 8790 8791 - If CU wavefront execution 8792 mode, omit vmcnt(0) and 8793 vscnt(0). 8794 - If OpenCL and 8795 address space is 8796 not generic, omit 8797 lgkmcnt(0). 8798 - If OpenCL and 8799 address space is 8800 local, omit 8801 vmcnt(0) and vscnt(0). 8802 - However, since LLVM 8803 currently has no 8804 address space on 8805 the fence need to 8806 conservatively 8807 always generate. If 8808 fence had an 8809 address space then 8810 set to address 8811 space of OpenCL 8812 fence flag, or to 8813 generic if both 8814 local and global 8815 flags are 8816 specified. 8817 - Could be split into 8818 separate s_waitcnt 8819 vmcnt(0), s_waitcnt 8820 vscnt(0) and s_waitcnt 8821 lgkmcnt(0) to allow 8822 them to be 8823 independently moved 8824 according to the 8825 following rules. 8826 - s_waitcnt vmcnt(0) 8827 must happen after 8828 any preceding 8829 global/generic load 8830 atomic/ 8831 atomicrmw-with-return-value 8832 with an equal or 8833 wider sync scope 8834 and memory ordering 8835 stronger than 8836 unordered (this is 8837 termed the 8838 fence-paired-atomic). 8839 - s_waitcnt vscnt(0) 8840 must happen after 8841 any preceding 8842 global/generic 8843 atomicrmw-no-return-value 8844 with an equal or 8845 wider sync scope 8846 and memory ordering 8847 stronger than 8848 unordered (this is 8849 termed the 8850 fence-paired-atomic). 8851 - s_waitcnt lgkmcnt(0) 8852 must happen after 8853 any preceding 8854 local/generic load 8855 atomic/atomicrmw 8856 with an equal or 8857 wider sync scope 8858 and memory ordering 8859 stronger than 8860 unordered (this is 8861 termed the 8862 fence-paired-atomic). 8863 - Must happen before 8864 the following 8865 buffer_gl0_inv. 8866 - Ensures that the 8867 fence-paired atomic 8868 has completed 8869 before invalidating 8870 the 8871 cache. Therefore 8872 any following 8873 locations read must 8874 be no older than 8875 the value read by 8876 the 8877 fence-paired-atomic. 8878 8879 3. buffer_gl0_inv 8880 8881 - If CU wavefront execution 8882 mode, omit. 8883 - Ensures that 8884 following 8885 loads will not see 8886 stale data. 8887 8888 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 8889 - system vmcnt(0) & vscnt(0) 8890 8891 - If OpenCL and 8892 address space is 8893 not generic, omit 8894 lgkmcnt(0). 8895 - If OpenCL and 8896 address space is 8897 local, omit 8898 vmcnt(0) and vscnt(0). 8899 - However, since LLVM 8900 currently has no 8901 address space on 8902 the fence need to 8903 conservatively 8904 always generate 8905 (see comment for 8906 previous fence). 8907 - Could be split into 8908 separate s_waitcnt 8909 vmcnt(0), s_waitcnt 8910 vscnt(0) and s_waitcnt 8911 lgkmcnt(0) to allow 8912 them to be 8913 independently moved 8914 according to the 8915 following rules. 8916 - s_waitcnt vmcnt(0) 8917 must happen after 8918 any preceding 8919 global/generic load 8920 atomic/ 8921 atomicrmw-with-return-value 8922 with an equal or 8923 wider sync scope 8924 and memory ordering 8925 stronger than 8926 unordered (this is 8927 termed the 8928 fence-paired-atomic). 8929 - s_waitcnt vscnt(0) 8930 must happen after 8931 any preceding 8932 global/generic 8933 atomicrmw-no-return-value 8934 with an equal or 8935 wider sync scope 8936 and memory ordering 8937 stronger than 8938 unordered (this is 8939 termed the 8940 fence-paired-atomic). 8941 - s_waitcnt lgkmcnt(0) 8942 must happen after 8943 any preceding 8944 local/generic load 8945 atomic/atomicrmw 8946 with an equal or 8947 wider sync scope 8948 and memory ordering 8949 stronger than 8950 unordered (this is 8951 termed the 8952 fence-paired-atomic). 8953 - Must happen before 8954 the following 8955 buffer_gl*_inv. 8956 - Ensures that the 8957 fence-paired atomic 8958 has completed 8959 before invalidating 8960 the 8961 caches. Therefore 8962 any following 8963 locations read must 8964 be no older than 8965 the value read by 8966 the 8967 fence-paired-atomic. 8968 8969 2. buffer_gl0_inv; 8970 buffer_gl1_inv 8971 8972 - Must happen before any 8973 following global/generic 8974 load/load 8975 atomic/store/store 8976 atomic/atomicrmw. 8977 - Ensures that 8978 following loads 8979 will not see stale 8980 global data. 8981 8982 **Release Atomic** 8983 ------------------------------------------------------------------------------------ 8984 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 8985 - wavefront - local 8986 - generic 8987 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 8988 - generic vmcnt(0) & vscnt(0) 8989 8990 - If CU wavefront execution 8991 mode, omit vmcnt(0) and 8992 vscnt(0). 8993 - If OpenCL, omit 8994 lgkmcnt(0). 8995 - Could be split into 8996 separate s_waitcnt 8997 vmcnt(0), s_waitcnt 8998 vscnt(0) and s_waitcnt 8999 lgkmcnt(0) to allow 9000 them to be 9001 independently moved 9002 according to the 9003 following rules. 9004 - s_waitcnt vmcnt(0) 9005 must happen after 9006 any preceding 9007 global/generic load/load 9008 atomic/ 9009 atomicrmw-with-return-value. 9010 - s_waitcnt vscnt(0) 9011 must happen after 9012 any preceding 9013 global/generic 9014 store/store 9015 atomic/ 9016 atomicrmw-no-return-value. 9017 - s_waitcnt lgkmcnt(0) 9018 must happen after 9019 any preceding 9020 local/generic 9021 load/store/load 9022 atomic/store 9023 atomic/atomicrmw. 9024 - Must happen before 9025 the following 9026 store. 9027 - Ensures that all 9028 memory operations 9029 have 9030 completed before 9031 performing the 9032 store that is being 9033 released. 9034 9035 2. buffer/global/flat_store 9036 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9037 9038 - If CU wavefront execution 9039 mode, omit. 9040 - If OpenCL, omit. 9041 - Could be split into 9042 separate s_waitcnt 9043 vmcnt(0) and s_waitcnt 9044 vscnt(0) to allow 9045 them to be 9046 independently moved 9047 according to the 9048 following rules. 9049 - s_waitcnt vmcnt(0) 9050 must happen after 9051 any preceding 9052 global/generic load/load 9053 atomic/ 9054 atomicrmw-with-return-value. 9055 - s_waitcnt vscnt(0) 9056 must happen after 9057 any preceding 9058 global/generic 9059 store/store atomic/ 9060 atomicrmw-no-return-value. 9061 - Must happen before 9062 the following 9063 store. 9064 - Ensures that all 9065 global memory 9066 operations have 9067 completed before 9068 performing the 9069 store that is being 9070 released. 9071 9072 2. ds_store 9073 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9074 - system - generic vmcnt(0) & vscnt(0) 9075 9076 - If OpenCL and 9077 address space is 9078 not generic, omit 9079 lgkmcnt(0). 9080 - Could be split into 9081 separate s_waitcnt 9082 vmcnt(0), s_waitcnt vscnt(0) 9083 and s_waitcnt 9084 lgkmcnt(0) to allow 9085 them to be 9086 independently moved 9087 according to the 9088 following rules. 9089 - s_waitcnt vmcnt(0) 9090 must happen after 9091 any preceding 9092 global/generic 9093 load/load 9094 atomic/ 9095 atomicrmw-with-return-value. 9096 - s_waitcnt vscnt(0) 9097 must happen after 9098 any preceding 9099 global/generic 9100 store/store atomic/ 9101 atomicrmw-no-return-value. 9102 - s_waitcnt lgkmcnt(0) 9103 must happen after 9104 any preceding 9105 local/generic 9106 load/store/load 9107 atomic/store 9108 atomic/atomicrmw. 9109 - Must happen before 9110 the following 9111 store. 9112 - Ensures that all 9113 memory operations 9114 have 9115 completed before 9116 performing the 9117 store that is being 9118 released. 9119 9120 2. buffer/global/flat_store 9121 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9122 - wavefront - local 9123 - generic 9124 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9125 - generic vmcnt(0) & vscnt(0) 9126 9127 - If CU wavefront execution 9128 mode, omit vmcnt(0) and 9129 vscnt(0). 9130 - If OpenCL, omit lgkmcnt(0). 9131 - Could be split into 9132 separate s_waitcnt 9133 vmcnt(0), s_waitcnt 9134 vscnt(0) and s_waitcnt 9135 lgkmcnt(0) to allow 9136 them to be 9137 independently moved 9138 according to the 9139 following rules. 9140 - s_waitcnt vmcnt(0) 9141 must happen after 9142 any preceding 9143 global/generic load/load 9144 atomic/ 9145 atomicrmw-with-return-value. 9146 - s_waitcnt vscnt(0) 9147 must happen after 9148 any preceding 9149 global/generic 9150 store/store 9151 atomic/ 9152 atomicrmw-no-return-value. 9153 - s_waitcnt lgkmcnt(0) 9154 must happen after 9155 any preceding 9156 local/generic 9157 load/store/load 9158 atomic/store 9159 atomic/atomicrmw. 9160 - Must happen before 9161 the following 9162 atomicrmw. 9163 - Ensures that all 9164 memory operations 9165 have 9166 completed before 9167 performing the 9168 atomicrmw that is 9169 being released. 9170 9171 2. buffer/global/flat_atomic 9172 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9173 9174 - If CU wavefront execution 9175 mode, omit. 9176 - If OpenCL, omit. 9177 - Could be split into 9178 separate s_waitcnt 9179 vmcnt(0) and s_waitcnt 9180 vscnt(0) to allow 9181 them to be 9182 independently moved 9183 according to the 9184 following rules. 9185 - s_waitcnt vmcnt(0) 9186 must happen after 9187 any preceding 9188 global/generic load/load 9189 atomic/ 9190 atomicrmw-with-return-value. 9191 - s_waitcnt vscnt(0) 9192 must happen after 9193 any preceding 9194 global/generic 9195 store/store atomic/ 9196 atomicrmw-no-return-value. 9197 - Must happen before 9198 the following 9199 store. 9200 - Ensures that all 9201 global memory 9202 operations have 9203 completed before 9204 performing the 9205 store that is being 9206 released. 9207 9208 2. ds_atomic 9209 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9210 - system - generic vmcnt(0) & vscnt(0) 9211 9212 - If OpenCL, omit 9213 lgkmcnt(0). 9214 - Could be split into 9215 separate s_waitcnt 9216 vmcnt(0), s_waitcnt 9217 vscnt(0) and s_waitcnt 9218 lgkmcnt(0) to allow 9219 them to be 9220 independently moved 9221 according to the 9222 following rules. 9223 - s_waitcnt vmcnt(0) 9224 must happen after 9225 any preceding 9226 global/generic 9227 load/load atomic/ 9228 atomicrmw-with-return-value. 9229 - s_waitcnt vscnt(0) 9230 must happen after 9231 any preceding 9232 global/generic 9233 store/store atomic/ 9234 atomicrmw-no-return-value. 9235 - s_waitcnt lgkmcnt(0) 9236 must happen after 9237 any preceding 9238 local/generic 9239 load/store/load 9240 atomic/store 9241 atomic/atomicrmw. 9242 - Must happen before 9243 the following 9244 atomicrmw. 9245 - Ensures that all 9246 memory operations 9247 to global and local 9248 have completed 9249 before performing 9250 the atomicrmw that 9251 is being released. 9252 9253 2. buffer/global/flat_atomic 9254 fence release - singlethread *none* *none* 9255 - wavefront 9256 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9257 vmcnt(0) & vscnt(0) 9258 9259 - If CU wavefront execution 9260 mode, omit vmcnt(0) and 9261 vscnt(0). 9262 - If OpenCL and 9263 address space is 9264 not generic, omit 9265 lgkmcnt(0). 9266 - If OpenCL and 9267 address space is 9268 local, omit 9269 vmcnt(0) and vscnt(0). 9270 - However, since LLVM 9271 currently has no 9272 address space on 9273 the fence need to 9274 conservatively 9275 always generate. If 9276 fence had an 9277 address space then 9278 set to address 9279 space of OpenCL 9280 fence flag, or to 9281 generic if both 9282 local and global 9283 flags are 9284 specified. 9285 - Could be split into 9286 separate s_waitcnt 9287 vmcnt(0), s_waitcnt 9288 vscnt(0) and s_waitcnt 9289 lgkmcnt(0) to allow 9290 them to be 9291 independently moved 9292 according to the 9293 following rules. 9294 - s_waitcnt vmcnt(0) 9295 must happen after 9296 any preceding 9297 global/generic 9298 load/load 9299 atomic/ 9300 atomicrmw-with-return-value. 9301 - s_waitcnt vscnt(0) 9302 must happen after 9303 any preceding 9304 global/generic 9305 store/store atomic/ 9306 atomicrmw-no-return-value. 9307 - s_waitcnt lgkmcnt(0) 9308 must happen after 9309 any preceding 9310 local/generic 9311 load/store/load 9312 atomic/store atomic/ 9313 atomicrmw. 9314 - Must happen before 9315 any following store 9316 atomic/atomicrmw 9317 with an equal or 9318 wider sync scope 9319 and memory ordering 9320 stronger than 9321 unordered (this is 9322 termed the 9323 fence-paired-atomic). 9324 - Ensures that all 9325 memory operations 9326 have 9327 completed before 9328 performing the 9329 following 9330 fence-paired-atomic. 9331 9332 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9333 - system vmcnt(0) & vscnt(0) 9334 9335 - If OpenCL and 9336 address space is 9337 not generic, omit 9338 lgkmcnt(0). 9339 - If OpenCL and 9340 address space is 9341 local, omit 9342 vmcnt(0) and vscnt(0). 9343 - However, since LLVM 9344 currently has no 9345 address space on 9346 the fence need to 9347 conservatively 9348 always generate. If 9349 fence had an 9350 address space then 9351 set to address 9352 space of OpenCL 9353 fence flag, or to 9354 generic if both 9355 local and global 9356 flags are 9357 specified. 9358 - Could be split into 9359 separate s_waitcnt 9360 vmcnt(0), s_waitcnt 9361 vscnt(0) and s_waitcnt 9362 lgkmcnt(0) to allow 9363 them to be 9364 independently moved 9365 according to the 9366 following rules. 9367 - s_waitcnt vmcnt(0) 9368 must happen after 9369 any preceding 9370 global/generic 9371 load/load atomic/ 9372 atomicrmw-with-return-value. 9373 - s_waitcnt vscnt(0) 9374 must happen after 9375 any preceding 9376 global/generic 9377 store/store atomic/ 9378 atomicrmw-no-return-value. 9379 - s_waitcnt lgkmcnt(0) 9380 must happen after 9381 any preceding 9382 local/generic 9383 load/store/load 9384 atomic/store 9385 atomic/atomicrmw. 9386 - Must happen before 9387 any following store 9388 atomic/atomicrmw 9389 with an equal or 9390 wider sync scope 9391 and memory ordering 9392 stronger than 9393 unordered (this is 9394 termed the 9395 fence-paired-atomic). 9396 - Ensures that all 9397 memory operations 9398 have 9399 completed before 9400 performing the 9401 following 9402 fence-paired-atomic. 9403 9404 **Acquire-Release Atomic** 9405 ------------------------------------------------------------------------------------ 9406 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9407 - wavefront - local 9408 - generic 9409 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9410 vmcnt(0) & vscnt(0) 9411 9412 - If CU wavefront execution 9413 mode, omit vmcnt(0) and 9414 vscnt(0). 9415 - If OpenCL, omit 9416 lgkmcnt(0). 9417 - Must happen after 9418 any preceding 9419 local/generic 9420 load/store/load 9421 atomic/store 9422 atomic/atomicrmw. 9423 - Could be split into 9424 separate s_waitcnt 9425 vmcnt(0), s_waitcnt 9426 vscnt(0), and s_waitcnt 9427 lgkmcnt(0) to allow 9428 them to be 9429 independently moved 9430 according to the 9431 following rules. 9432 - s_waitcnt vmcnt(0) 9433 must happen after 9434 any preceding 9435 global/generic load/load 9436 atomic/ 9437 atomicrmw-with-return-value. 9438 - s_waitcnt vscnt(0) 9439 must happen after 9440 any preceding 9441 global/generic 9442 store/store 9443 atomic/ 9444 atomicrmw-no-return-value. 9445 - s_waitcnt lgkmcnt(0) 9446 must happen after 9447 any preceding 9448 local/generic 9449 load/store/load 9450 atomic/store 9451 atomic/atomicrmw. 9452 - Must happen before 9453 the following 9454 atomicrmw. 9455 - Ensures that all 9456 memory operations 9457 have 9458 completed before 9459 performing the 9460 atomicrmw that is 9461 being released. 9462 9463 2. buffer/global_atomic 9464 3. s_waitcnt vm/vscnt(0) 9465 9466 - If CU wavefront execution 9467 mode, omit. 9468 - Use vmcnt(0) if atomic with 9469 return and vscnt(0) if 9470 atomic with no-return. 9471 - Must happen before 9472 the following 9473 buffer_gl0_inv. 9474 - Ensures any 9475 following global 9476 data read is no 9477 older than the 9478 atomicrmw value 9479 being acquired. 9480 9481 4. buffer_gl0_inv 9482 9483 - If CU wavefront execution 9484 mode, omit. 9485 - Ensures that 9486 following 9487 loads will not see 9488 stale data. 9489 9490 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9491 9492 - If CU wavefront execution 9493 mode, omit. 9494 - If OpenCL, omit. 9495 - Could be split into 9496 separate s_waitcnt 9497 vmcnt(0) and s_waitcnt 9498 vscnt(0) to allow 9499 them to be 9500 independently moved 9501 according to the 9502 following rules. 9503 - s_waitcnt vmcnt(0) 9504 must happen after 9505 any preceding 9506 global/generic load/load 9507 atomic/ 9508 atomicrmw-with-return-value. 9509 - s_waitcnt vscnt(0) 9510 must happen after 9511 any preceding 9512 global/generic 9513 store/store atomic/ 9514 atomicrmw-no-return-value. 9515 - Must happen before 9516 the following 9517 store. 9518 - Ensures that all 9519 global memory 9520 operations have 9521 completed before 9522 performing the 9523 store that is being 9524 released. 9525 9526 2. ds_atomic 9527 3. s_waitcnt lgkmcnt(0) 9528 9529 - If OpenCL, omit. 9530 - Must happen before 9531 the following 9532 buffer_gl0_inv. 9533 - Ensures any 9534 following global 9535 data read is no 9536 older than the local load 9537 atomic value being 9538 acquired. 9539 9540 4. buffer_gl0_inv 9541 9542 - If CU wavefront execution 9543 mode, omit. 9544 - If OpenCL omit. 9545 - Ensures that 9546 following 9547 loads will not see 9548 stale data. 9549 9550 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9551 vmcnt(0) & vscnt(0) 9552 9553 - If CU wavefront execution 9554 mode, omit vmcnt(0) and 9555 vscnt(0). 9556 - If OpenCL, omit lgkmcnt(0). 9557 - Could be split into 9558 separate s_waitcnt 9559 vmcnt(0), s_waitcnt 9560 vscnt(0) and s_waitcnt 9561 lgkmcnt(0) to allow 9562 them to be 9563 independently moved 9564 according to the 9565 following rules. 9566 - s_waitcnt vmcnt(0) 9567 must happen after 9568 any preceding 9569 global/generic load/load 9570 atomic/ 9571 atomicrmw-with-return-value. 9572 - s_waitcnt vscnt(0) 9573 must happen after 9574 any preceding 9575 global/generic 9576 store/store 9577 atomic/ 9578 atomicrmw-no-return-value. 9579 - s_waitcnt lgkmcnt(0) 9580 must happen after 9581 any preceding 9582 local/generic 9583 load/store/load 9584 atomic/store 9585 atomic/atomicrmw. 9586 - Must happen before 9587 the following 9588 atomicrmw. 9589 - Ensures that all 9590 memory operations 9591 have 9592 completed before 9593 performing the 9594 atomicrmw that is 9595 being released. 9596 9597 2. flat_atomic 9598 3. s_waitcnt lgkmcnt(0) & 9599 vmcnt(0) & vscnt(0) 9600 9601 - If CU wavefront execution 9602 mode, omit vmcnt(0) and 9603 vscnt(0). 9604 - If OpenCL, omit lgkmcnt(0). 9605 - Must happen before 9606 the following 9607 buffer_gl0_inv. 9608 - Ensures any 9609 following global 9610 data read is no 9611 older than the load 9612 atomic value being 9613 acquired. 9614 9615 3. buffer_gl0_inv 9616 9617 - If CU wavefront execution 9618 mode, omit. 9619 - Ensures that 9620 following 9621 loads will not see 9622 stale data. 9623 9624 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9625 - system vmcnt(0) & vscnt(0) 9626 9627 - If OpenCL, omit 9628 lgkmcnt(0). 9629 - Could be split into 9630 separate s_waitcnt 9631 vmcnt(0), s_waitcnt 9632 vscnt(0) and s_waitcnt 9633 lgkmcnt(0) to allow 9634 them to be 9635 independently moved 9636 according to the 9637 following rules. 9638 - s_waitcnt vmcnt(0) 9639 must happen after 9640 any preceding 9641 global/generic 9642 load/load atomic/ 9643 atomicrmw-with-return-value. 9644 - s_waitcnt vscnt(0) 9645 must happen after 9646 any preceding 9647 global/generic 9648 store/store atomic/ 9649 atomicrmw-no-return-value. 9650 - s_waitcnt lgkmcnt(0) 9651 must happen after 9652 any preceding 9653 local/generic 9654 load/store/load 9655 atomic/store 9656 atomic/atomicrmw. 9657 - Must happen before 9658 the following 9659 atomicrmw. 9660 - Ensures that all 9661 memory operations 9662 to global have 9663 completed before 9664 performing the 9665 atomicrmw that is 9666 being released. 9667 9668 2. buffer/global_atomic 9669 3. s_waitcnt vm/vscnt(0) 9670 9671 - Use vmcnt(0) if atomic with 9672 return and vscnt(0) if 9673 atomic with no-return. 9674 - Must happen before 9675 following 9676 buffer_gl*_inv. 9677 - Ensures the 9678 atomicrmw has 9679 completed before 9680 invalidating the 9681 caches. 9682 9683 4. buffer_gl0_inv; 9684 buffer_gl1_inv 9685 9686 - Must happen before 9687 any following 9688 global/generic 9689 load/load 9690 atomic/atomicrmw. 9691 - Ensures that 9692 following loads 9693 will not see stale 9694 global data. 9695 9696 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9697 - system vmcnt(0) & vscnt(0) 9698 9699 - If OpenCL, omit 9700 lgkmcnt(0). 9701 - Could be split into 9702 separate s_waitcnt 9703 vmcnt(0), s_waitcnt 9704 vscnt(0), and s_waitcnt 9705 lgkmcnt(0) to allow 9706 them to be 9707 independently moved 9708 according to the 9709 following rules. 9710 - s_waitcnt vmcnt(0) 9711 must happen after 9712 any preceding 9713 global/generic 9714 load/load atomic 9715 atomicrmw-with-return-value. 9716 - s_waitcnt vscnt(0) 9717 must happen after 9718 any preceding 9719 global/generic 9720 store/store atomic/ 9721 atomicrmw-no-return-value. 9722 - s_waitcnt lgkmcnt(0) 9723 must happen after 9724 any preceding 9725 local/generic 9726 load/store/load 9727 atomic/store 9728 atomic/atomicrmw. 9729 - Must happen before 9730 the following 9731 atomicrmw. 9732 - Ensures that all 9733 memory operations 9734 have 9735 completed before 9736 performing the 9737 atomicrmw that is 9738 being released. 9739 9740 2. flat_atomic 9741 3. s_waitcnt vm/vscnt(0) & 9742 lgkmcnt(0) 9743 9744 - If OpenCL, omit 9745 lgkmcnt(0). 9746 - Use vmcnt(0) if atomic with 9747 return and vscnt(0) if 9748 atomic with no-return. 9749 - Must happen before 9750 following 9751 buffer_gl*_inv. 9752 - Ensures the 9753 atomicrmw has 9754 completed before 9755 invalidating the 9756 caches. 9757 9758 4. buffer_gl0_inv; 9759 buffer_gl1_inv 9760 9761 - Must happen before 9762 any following 9763 global/generic 9764 load/load 9765 atomic/atomicrmw. 9766 - Ensures that 9767 following loads 9768 will not see stale 9769 global data. 9770 9771 fence acq_rel - singlethread *none* *none* 9772 - wavefront 9773 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9774 vmcnt(0) & vscnt(0) 9775 9776 - If CU wavefront execution 9777 mode, omit vmcnt(0) and 9778 vscnt(0). 9779 - If OpenCL and 9780 address space is 9781 not generic, omit 9782 lgkmcnt(0). 9783 - If OpenCL and 9784 address space is 9785 local, omit 9786 vmcnt(0) and vscnt(0). 9787 - However, 9788 since LLVM 9789 currently has no 9790 address space on 9791 the fence need to 9792 conservatively 9793 always generate 9794 (see comment for 9795 previous fence). 9796 - Could be split into 9797 separate s_waitcnt 9798 vmcnt(0), s_waitcnt 9799 vscnt(0) and s_waitcnt 9800 lgkmcnt(0) to allow 9801 them to be 9802 independently moved 9803 according to the 9804 following rules. 9805 - s_waitcnt vmcnt(0) 9806 must happen after 9807 any preceding 9808 global/generic 9809 load/load 9810 atomic/ 9811 atomicrmw-with-return-value. 9812 - s_waitcnt vscnt(0) 9813 must happen after 9814 any preceding 9815 global/generic 9816 store/store atomic/ 9817 atomicrmw-no-return-value. 9818 - s_waitcnt lgkmcnt(0) 9819 must happen after 9820 any preceding 9821 local/generic 9822 load/store/load 9823 atomic/store atomic/ 9824 atomicrmw. 9825 - Must happen before 9826 any following 9827 global/generic 9828 load/load 9829 atomic/store/store 9830 atomic/atomicrmw. 9831 - Ensures that all 9832 memory operations 9833 have 9834 completed before 9835 performing any 9836 following global 9837 memory operations. 9838 - Ensures that the 9839 preceding 9840 local/generic load 9841 atomic/atomicrmw 9842 with an equal or 9843 wider sync scope 9844 and memory ordering 9845 stronger than 9846 unordered (this is 9847 termed the 9848 acquire-fence-paired-atomic) 9849 has completed 9850 before following 9851 global memory 9852 operations. This 9853 satisfies the 9854 requirements of 9855 acquire. 9856 - Ensures that all 9857 previous memory 9858 operations have 9859 completed before a 9860 following 9861 local/generic store 9862 atomic/atomicrmw 9863 with an equal or 9864 wider sync scope 9865 and memory ordering 9866 stronger than 9867 unordered (this is 9868 termed the 9869 release-fence-paired-atomic). 9870 This satisfies the 9871 requirements of 9872 release. 9873 - Must happen before 9874 the following 9875 buffer_gl0_inv. 9876 - Ensures that the 9877 acquire-fence-paired 9878 atomic has completed 9879 before invalidating 9880 the 9881 cache. Therefore 9882 any following 9883 locations read must 9884 be no older than 9885 the value read by 9886 the 9887 acquire-fence-paired-atomic. 9888 9889 3. buffer_gl0_inv 9890 9891 - If CU wavefront execution 9892 mode, omit. 9893 - Ensures that 9894 following 9895 loads will not see 9896 stale data. 9897 9898 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 9899 - system vmcnt(0) & vscnt(0) 9900 9901 - If OpenCL and 9902 address space is 9903 not generic, omit 9904 lgkmcnt(0). 9905 - If OpenCL and 9906 address space is 9907 local, omit 9908 vmcnt(0) and vscnt(0). 9909 - However, since LLVM 9910 currently has no 9911 address space on 9912 the fence need to 9913 conservatively 9914 always generate 9915 (see comment for 9916 previous fence). 9917 - Could be split into 9918 separate s_waitcnt 9919 vmcnt(0), s_waitcnt 9920 vscnt(0) and s_waitcnt 9921 lgkmcnt(0) to allow 9922 them to be 9923 independently moved 9924 according to the 9925 following rules. 9926 - s_waitcnt vmcnt(0) 9927 must happen after 9928 any preceding 9929 global/generic 9930 load/load 9931 atomic/ 9932 atomicrmw-with-return-value. 9933 - s_waitcnt vscnt(0) 9934 must happen after 9935 any preceding 9936 global/generic 9937 store/store atomic/ 9938 atomicrmw-no-return-value. 9939 - s_waitcnt lgkmcnt(0) 9940 must happen after 9941 any preceding 9942 local/generic 9943 load/store/load 9944 atomic/store 9945 atomic/atomicrmw. 9946 - Must happen before 9947 the following 9948 buffer_gl*_inv. 9949 - Ensures that the 9950 preceding 9951 global/local/generic 9952 load 9953 atomic/atomicrmw 9954 with an equal or 9955 wider sync scope 9956 and memory ordering 9957 stronger than 9958 unordered (this is 9959 termed the 9960 acquire-fence-paired-atomic) 9961 has completed 9962 before invalidating 9963 the caches. This 9964 satisfies the 9965 requirements of 9966 acquire. 9967 - Ensures that all 9968 previous memory 9969 operations have 9970 completed before a 9971 following 9972 global/local/generic 9973 store 9974 atomic/atomicrmw 9975 with an equal or 9976 wider sync scope 9977 and memory ordering 9978 stronger than 9979 unordered (this is 9980 termed the 9981 release-fence-paired-atomic). 9982 This satisfies the 9983 requirements of 9984 release. 9985 9986 2. buffer_gl0_inv; 9987 buffer_gl1_inv 9988 9989 - Must happen before 9990 any following 9991 global/generic 9992 load/load 9993 atomic/store/store 9994 atomic/atomicrmw. 9995 - Ensures that 9996 following loads 9997 will not see stale 9998 global data. This 9999 satisfies the 10000 requirements of 10001 acquire. 10002 10003 **Sequential Consistent Atomic** 10004 ------------------------------------------------------------------------------------ 10005 load atomic seq_cst - singlethread - global *Same as corresponding 10006 - wavefront - local load atomic acquire, 10007 - generic except must generated 10008 all instructions even 10009 for OpenCL.* 10010 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10011 - generic vmcnt(0) & vscnt(0) 10012 10013 - If CU wavefront execution 10014 mode, omit vmcnt(0) and 10015 vscnt(0). 10016 - Could be split into 10017 separate s_waitcnt 10018 vmcnt(0), s_waitcnt 10019 vscnt(0), and s_waitcnt 10020 lgkmcnt(0) to allow 10021 them to be 10022 independently moved 10023 according to the 10024 following rules. 10025 - s_waitcnt lgkmcnt(0) must 10026 happen after 10027 preceding 10028 local/generic load 10029 atomic/store 10030 atomic/atomicrmw 10031 with memory 10032 ordering of seq_cst 10033 and with equal or 10034 wider sync scope. 10035 (Note that seq_cst 10036 fences have their 10037 own s_waitcnt 10038 lgkmcnt(0) and so do 10039 not need to be 10040 considered.) 10041 - s_waitcnt vmcnt(0) 10042 must happen after 10043 preceding 10044 global/generic load 10045 atomic/ 10046 atomicrmw-with-return-value 10047 with memory 10048 ordering of seq_cst 10049 and with equal or 10050 wider sync scope. 10051 (Note that seq_cst 10052 fences have their 10053 own s_waitcnt 10054 vmcnt(0) and so do 10055 not need to be 10056 considered.) 10057 - s_waitcnt vscnt(0) 10058 Must happen after 10059 preceding 10060 global/generic store 10061 atomic/ 10062 atomicrmw-no-return-value 10063 with memory 10064 ordering of seq_cst 10065 and with equal or 10066 wider sync scope. 10067 (Note that seq_cst 10068 fences have their 10069 own s_waitcnt 10070 vscnt(0) and so do 10071 not need to be 10072 considered.) 10073 - Ensures any 10074 preceding 10075 sequential 10076 consistent global/local 10077 memory instructions 10078 have completed 10079 before executing 10080 this sequentially 10081 consistent 10082 instruction. This 10083 prevents reordering 10084 a seq_cst store 10085 followed by a 10086 seq_cst load. (Note 10087 that seq_cst is 10088 stronger than 10089 acquire/release as 10090 the reordering of 10091 load acquire 10092 followed by a store 10093 release is 10094 prevented by the 10095 s_waitcnt of 10096 the release, but 10097 there is nothing 10098 preventing a store 10099 release followed by 10100 load acquire from 10101 completing out of 10102 order. The s_waitcnt 10103 could be placed after 10104 seq_store or before 10105 the seq_load. We 10106 choose the load to 10107 make the s_waitcnt be 10108 as late as possible 10109 so that the store 10110 may have already 10111 completed.) 10112 10113 2. *Following 10114 instructions same as 10115 corresponding load 10116 atomic acquire, 10117 except must generated 10118 all instructions even 10119 for OpenCL.* 10120 load atomic seq_cst - workgroup - local 10121 10122 1. s_waitcnt vmcnt(0) & vscnt(0) 10123 10124 - If CU wavefront execution 10125 mode, omit. 10126 - Could be split into 10127 separate s_waitcnt 10128 vmcnt(0) and s_waitcnt 10129 vscnt(0) to allow 10130 them to be 10131 independently moved 10132 according to the 10133 following rules. 10134 - s_waitcnt vmcnt(0) 10135 Must happen after 10136 preceding 10137 global/generic load 10138 atomic/ 10139 atomicrmw-with-return-value 10140 with memory 10141 ordering of seq_cst 10142 and with equal or 10143 wider sync scope. 10144 (Note that seq_cst 10145 fences have their 10146 own s_waitcnt 10147 vmcnt(0) and so do 10148 not need to be 10149 considered.) 10150 - s_waitcnt vscnt(0) 10151 Must happen after 10152 preceding 10153 global/generic store 10154 atomic/ 10155 atomicrmw-no-return-value 10156 with memory 10157 ordering of seq_cst 10158 and with equal or 10159 wider sync scope. 10160 (Note that seq_cst 10161 fences have their 10162 own s_waitcnt 10163 vscnt(0) and so do 10164 not need to be 10165 considered.) 10166 - Ensures any 10167 preceding 10168 sequential 10169 consistent global 10170 memory instructions 10171 have completed 10172 before executing 10173 this sequentially 10174 consistent 10175 instruction. This 10176 prevents reordering 10177 a seq_cst store 10178 followed by a 10179 seq_cst load. (Note 10180 that seq_cst is 10181 stronger than 10182 acquire/release as 10183 the reordering of 10184 load acquire 10185 followed by a store 10186 release is 10187 prevented by the 10188 s_waitcnt of 10189 the release, but 10190 there is nothing 10191 preventing a store 10192 release followed by 10193 load acquire from 10194 completing out of 10195 order. The s_waitcnt 10196 could be placed after 10197 seq_store or before 10198 the seq_load. We 10199 choose the load to 10200 make the s_waitcnt be 10201 as late as possible 10202 so that the store 10203 may have already 10204 completed.) 10205 10206 2. *Following 10207 instructions same as 10208 corresponding load 10209 atomic acquire, 10210 except must generated 10211 all instructions even 10212 for OpenCL.* 10213 10214 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10215 - system - generic vmcnt(0) & vscnt(0) 10216 10217 - Could be split into 10218 separate s_waitcnt 10219 vmcnt(0), s_waitcnt 10220 vscnt(0) and s_waitcnt 10221 lgkmcnt(0) to allow 10222 them to be 10223 independently moved 10224 according to the 10225 following rules. 10226 - s_waitcnt lgkmcnt(0) 10227 must happen after 10228 preceding 10229 local load 10230 atomic/store 10231 atomic/atomicrmw 10232 with memory 10233 ordering of seq_cst 10234 and with equal or 10235 wider sync scope. 10236 (Note that seq_cst 10237 fences have their 10238 own s_waitcnt 10239 lgkmcnt(0) and so do 10240 not need to be 10241 considered.) 10242 - s_waitcnt vmcnt(0) 10243 must happen after 10244 preceding 10245 global/generic load 10246 atomic/ 10247 atomicrmw-with-return-value 10248 with memory 10249 ordering of seq_cst 10250 and with equal or 10251 wider sync scope. 10252 (Note that seq_cst 10253 fences have their 10254 own s_waitcnt 10255 vmcnt(0) and so do 10256 not need to be 10257 considered.) 10258 - s_waitcnt vscnt(0) 10259 Must happen after 10260 preceding 10261 global/generic store 10262 atomic/ 10263 atomicrmw-no-return-value 10264 with memory 10265 ordering of seq_cst 10266 and with equal or 10267 wider sync scope. 10268 (Note that seq_cst 10269 fences have their 10270 own s_waitcnt 10271 vscnt(0) and so do 10272 not need to be 10273 considered.) 10274 - Ensures any 10275 preceding 10276 sequential 10277 consistent global 10278 memory instructions 10279 have completed 10280 before executing 10281 this sequentially 10282 consistent 10283 instruction. This 10284 prevents reordering 10285 a seq_cst store 10286 followed by a 10287 seq_cst load. (Note 10288 that seq_cst is 10289 stronger than 10290 acquire/release as 10291 the reordering of 10292 load acquire 10293 followed by a store 10294 release is 10295 prevented by the 10296 s_waitcnt of 10297 the release, but 10298 there is nothing 10299 preventing a store 10300 release followed by 10301 load acquire from 10302 completing out of 10303 order. The s_waitcnt 10304 could be placed after 10305 seq_store or before 10306 the seq_load. We 10307 choose the load to 10308 make the s_waitcnt be 10309 as late as possible 10310 so that the store 10311 may have already 10312 completed.) 10313 10314 2. *Following 10315 instructions same as 10316 corresponding load 10317 atomic acquire, 10318 except must generated 10319 all instructions even 10320 for OpenCL.* 10321 store atomic seq_cst - singlethread - global *Same as corresponding 10322 - wavefront - local store atomic release, 10323 - workgroup - generic except must generated 10324 - agent all instructions even 10325 - system for OpenCL.* 10326 atomicrmw seq_cst - singlethread - global *Same as corresponding 10327 - wavefront - local atomicrmw acq_rel, 10328 - workgroup - generic except must generated 10329 - agent all instructions even 10330 - system for OpenCL.* 10331 fence seq_cst - singlethread *none* *Same as corresponding 10332 - wavefront fence acq_rel, 10333 - workgroup except must generated 10334 - agent all instructions even 10335 - system for OpenCL.* 10336 ============ ============ ============== ========== ================================ 10337 10338Trap Handler ABI 10339~~~~~~~~~~~~~~~~ 10340 10341For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10342runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10343supports the ``s_trap`` instruction. For usage see: 10344 10345- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10346- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10347- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10348 10349 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10350 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10351 10352 =================== =============== =============== ======================================= 10353 Usage Code Sequence Trap Handler Description 10354 Inputs 10355 =================== =============== =============== ======================================= 10356 reserved ``s_trap 0x00`` Reserved by hardware. 10357 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10358 ``queue_ptr`` intrinsic (not implemented). 10359 ``VGPR0``: 10360 ``arg`` 10361 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10362 ``queue_ptr`` the trap instruction. The associated 10363 queue is signalled to put it into the 10364 error state. When the queue is put in 10365 the error state, the waves executing 10366 dispatches on the queue will be 10367 terminated. 10368 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10369 as a no-operation. The trap handler 10370 is entered and immediately returns to 10371 continue execution of the wavefront. 10372 - If the debugger is enabled, causes 10373 the debug trap to be reported by the 10374 debugger and the wavefront is put in 10375 the halt state with the PC at the 10376 instruction. The debugger must 10377 increment the PC and resume the wave. 10378 reserved ``s_trap 0x04`` Reserved. 10379 reserved ``s_trap 0x05`` Reserved. 10380 reserved ``s_trap 0x06`` Reserved. 10381 reserved ``s_trap 0x07`` Reserved. 10382 reserved ``s_trap 0x08`` Reserved. 10383 reserved ``s_trap 0xfe`` Reserved. 10384 reserved ``s_trap 0xff`` Reserved. 10385 =================== =============== =============== ======================================= 10386 10387.. 10388 10389 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10390 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10391 10392 =================== =============== =============== ======================================= 10393 Usage Code Sequence Trap Handler Description 10394 Inputs 10395 =================== =============== =============== ======================================= 10396 reserved ``s_trap 0x00`` Reserved by hardware. 10397 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10398 breakpoints. Causes wave to be halted 10399 with the PC at the trap instruction. 10400 The debugger is responsible to resume 10401 the wave, including the instruction 10402 that the breakpoint overwrote. 10403 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10404 ``queue_ptr`` the trap instruction. The associated 10405 queue is signalled to put it into the 10406 error state. When the queue is put in 10407 the error state, the waves executing 10408 dispatches on the queue will be 10409 terminated. 10410 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10411 as a no-operation. The trap handler 10412 is entered and immediately returns to 10413 continue execution of the wavefront. 10414 - If the debugger is enabled, causes 10415 the debug trap to be reported by the 10416 debugger and the wavefront is put in 10417 the halt state with the PC at the 10418 instruction. The debugger must 10419 increment the PC and resume the wave. 10420 reserved ``s_trap 0x04`` Reserved. 10421 reserved ``s_trap 0x05`` Reserved. 10422 reserved ``s_trap 0x06`` Reserved. 10423 reserved ``s_trap 0x07`` Reserved. 10424 reserved ``s_trap 0x08`` Reserved. 10425 reserved ``s_trap 0xfe`` Reserved. 10426 reserved ``s_trap 0xff`` Reserved. 10427 =================== =============== =============== ======================================= 10428 10429.. 10430 10431 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10432 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10433 10434 =================== =============== ================ ================= ======================================= 10435 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10436 =================== =============== ================ ================= ======================================= 10437 reserved ``s_trap 0x00`` Reserved by hardware. 10438 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10439 breakpoints. Causes wave to be halted 10440 with the PC at the trap instruction. 10441 The debugger is responsible to resume 10442 the wave, including the instruction 10443 that the breakpoint overwrote. 10444 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10445 ``queue_ptr`` the trap instruction. The associated 10446 queue is signalled to put it into the 10447 error state. When the queue is put in 10448 the error state, the waves executing 10449 dispatches on the queue will be 10450 terminated. 10451 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10452 as a no-operation. The trap handler 10453 is entered and immediately returns to 10454 continue execution of the wavefront. 10455 - If the debugger is enabled, causes 10456 the debug trap to be reported by the 10457 debugger and the wavefront is put in 10458 the halt state with the PC at the 10459 instruction. The debugger must 10460 increment the PC and resume the wave. 10461 reserved ``s_trap 0x04`` Reserved. 10462 reserved ``s_trap 0x05`` Reserved. 10463 reserved ``s_trap 0x06`` Reserved. 10464 reserved ``s_trap 0x07`` Reserved. 10465 reserved ``s_trap 0x08`` Reserved. 10466 reserved ``s_trap 0xfe`` Reserved. 10467 reserved ``s_trap 0xff`` Reserved. 10468 =================== =============== ================ ================= ======================================= 10469 10470.. _amdgpu-amdhsa-function-call-convention: 10471 10472Call Convention 10473~~~~~~~~~~~~~~~ 10474 10475.. note:: 10476 10477 This section is currently incomplete and has inaccuracies. It is WIP that will 10478 be updated as information is determined. 10479 10480See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10481addresses. Unswizzled addresses are normal linear addresses. 10482 10483.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10484 10485Kernel Functions 10486++++++++++++++++ 10487 10488This section describes the call convention ABI for the outer kernel function. 10489 10490See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10491convention. 10492 10493The following is not part of the AMDGPU kernel calling convention but describes 10494how the AMDGPU implements function calls: 10495 104961. Clang decides the kernarg layout to match the *HSA Programmer's Language 10497 Reference* [HSA]_. 10498 10499 - All structs are passed directly. 10500 - Lambda values are passed *TBA*. 10501 10502 .. TODO:: 10503 10504 - Does this really follow HSA rules? Or are structs >16 bytes passed 10505 by-value struct? 10506 - What is ABI for lambda values? 10507 105084. The kernel performs certain setup in its prolog, as described in 10509 :ref:`amdgpu-amdhsa-kernel-prolog`. 10510 10511.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10512 10513Non-Kernel Functions 10514++++++++++++++++++++ 10515 10516This section describes the call convention ABI for functions other than the 10517outer kernel function. 10518 10519If a kernel has function calls then scratch is always allocated and used for 10520the call stack which grows from low address to high address using the swizzled 10521scratch address space. 10522 10523On entry to a function: 10524 105251. SGPR0-3 contain a V# with the following properties (see 10526 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10527 10528 * Base address pointing to the beginning of the wavefront scratch backing 10529 memory. 10530 * Swizzled with dword element size and stride of wavefront size elements. 10531 105322. The FLAT_SCRATCH register pair is setup. See 10533 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 105343. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10535 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 105364. The EXEC register is set to the lanes active on entry to the function. 105375. MODE register: *TBD* 105386. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10539 below. 105407. SGPR30-31 return address (RA). The code address that the function must 10541 return to when it completes. The value is undefined if the function is *no 10542 return*. 105438. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10544 offset relative to the beginning of the wavefront scratch backing memory. 10545 10546 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10547 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10548 manner. 10549 10550 The unswizzled SP value can be converted into the swizzled SP value by: 10551 10552 | swizzled SP = unswizzled SP / wavefront size 10553 10554 This may be used to obtain the private address space address of stack 10555 objects and to convert this address to a flat address by adding the flat 10556 scratch aperture base address. 10557 10558 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10559 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10560 10561 .. note:: 10562 10563 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10564 OpenCL language which has the largest base type defined as 16 bytes. 10565 10566 On entry, the swizzled SP value is the address of the first function 10567 argument passed on the stack. Other stack passed arguments are positive 10568 offsets from the entry swizzled SP value. 10569 10570 The function may use positive offsets beyond the last stack passed argument 10571 for stack allocated local variables and register spill slots. If necessary, 10572 the function may align these to greater alignment than 16 bytes. After these 10573 the function may dynamically allocate space for such things as runtime sized 10574 ``alloca`` local allocations. 10575 10576 If the function calls another function, it will place any stack allocated 10577 arguments after the last local allocation and adjust SGPR32 to the address 10578 after the last local allocation. 10579 105809. All other registers are unspecified. 1058110. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10582 to the function. 10583 10584On exit from a function: 10585 105861. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10587 described below. Any registers used are considered clobbered registers. 105882. The following registers are preserved and have the same value as on entry: 10589 10590 * FLAT_SCRATCH 10591 * EXEC 10592 * GFX6-GFX8: M0 10593 * All SGPR registers except the clobbered registers of SGPR4-31. 10594 * VGPR40-47 10595 * VGPR56-63 10596 * VGPR72-79 10597 * VGPR88-95 10598 * VGPR104-111 10599 * VGPR120-127 10600 * VGPR136-143 10601 * VGPR152-159 10602 * VGPR168-175 10603 * VGPR184-191 10604 * VGPR200-207 10605 * VGPR216-223 10606 * VGPR232-239 10607 * VGPR248-255 10608 10609 .. note:: 10610 10611 Except the argument registers, the VGPRs clobbered and the preserved 10612 registers are intermixed at regular intervals in order to keep a 10613 similar ratio independent of the number of allocated VGPRs. 10614 10615 * Lanes of all VGPRs that are inactive at the call site. 10616 10617 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10618 optimization may mark some of clobbered SGPR and VGPR registers as 10619 preserved if it can be determined that the called function does not change 10620 their value. 10621 106222. The PC is set to the RA provided on entry. 106233. MODE register: *TBD*. 106244. All other registers are clobbered. 106255. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10626 function is available to the caller. 10627 10628.. TODO:: 10629 10630 - On gfx908 are all ACC registers clobbered? 10631 10632 - How are function results returned? The address of structured types is passed 10633 by reference, but what about other types? 10634 10635The function input arguments are made up of the formal arguments explicitly 10636declared by the source language function plus the implicit input arguments used 10637by the implementation. 10638 10639The source language input arguments are: 10640 106411. Any source language implicit ``this`` or ``self`` argument comes first as a 10642 pointer type. 106432. Followed by the function formal arguments in left to right source order. 10644 10645The source language result arguments are: 10646 106471. The function result argument. 10648 10649The source language input or result struct type arguments that are less than or 10650equal to 16 bytes, are decomposed recursively into their base type fields, and 10651each field is passed as if a separate argument. For input arguments, if the 10652called function requires the struct to be in memory, for example because its 10653address is taken, then the function body is responsible for allocating a stack 10654location and copying the field arguments into it. Clang terms this *direct 10655struct*. 10656 10657The source language input struct type arguments that are greater than 16 bytes, 10658are passed by reference. The caller is responsible for allocating a stack 10659location to make a copy of the struct value and pass the address as the input 10660argument. The called function is responsible to perform the dereference when 10661accessing the input argument. Clang terms this *by-value struct*. 10662 10663A source language result struct type argument that is greater than 16 bytes, is 10664returned by reference. The caller is responsible for allocating a stack location 10665to hold the result value and passes the address as the last input argument 10666(before the implicit input arguments). In this case there are no result 10667arguments. The called function is responsible to perform the dereference when 10668storing the result value. Clang terms this *structured return (sret)*. 10669 10670*TODO: correct the ``sret`` definition.* 10671 10672.. TODO:: 10673 10674 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10675 pass as non-decomposed struct as stack argument? Or something else? Is the 10676 memory location in the caller stack frame, or a stack memory argument and so 10677 no address is passed as the caller can directly write to the argument stack 10678 location? But then the stack location is still live after return. If an 10679 argument stack location is it the first stack argument or the last one? 10680 10681Lambda argument types are treated as struct types with an implementation defined 10682set of fields. 10683 10684.. TODO:: 10685 10686 Need to specify the ABI for lambda types for AMDGPU. 10687 10688For AMDGPU backend all source language arguments (including the decomposed 10689struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10690they are passed in SGPRs. 10691 10692The AMDGPU backend walks the function call graph from the leaves to determine 10693which implicit input arguments are used, propagating to each caller of the 10694function. The used implicit arguments are appended to the function arguments 10695after the source language arguments in the following order: 10696 10697.. TODO:: 10698 10699 Is recursion or external functions supported? 10700 107011. Work-Item ID (1 VGPR) 10702 10703 The X, Y and Z work-item ID are packed into a single VGRP with the following 10704 layout. Only fields actually used by the function are set. The other bits 10705 are undefined. 10706 10707 The values come from the initial kernel execution state. See 10708 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10709 10710 .. table:: Work-item implicit argument layout 10711 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10712 10713 ======= ======= ============== 10714 Bits Size Field Name 10715 ======= ======= ============== 10716 9:0 10 bits X Work-Item ID 10717 19:10 10 bits Y Work-Item ID 10718 29:20 10 bits Z Work-Item ID 10719 31:30 2 bits Unused 10720 ======= ======= ============== 10721 107222. Dispatch Ptr (2 SGPRs) 10723 10724 The value comes from the initial kernel execution state. See 10725 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10726 107273. Queue Ptr (2 SGPRs) 10728 10729 The value comes from the initial kernel execution state. See 10730 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10731 107324. Kernarg Segment Ptr (2 SGPRs) 10733 10734 The value comes from the initial kernel execution state. See 10735 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10736 107375. Dispatch id (2 SGPRs) 10738 10739 The value comes from the initial kernel execution state. See 10740 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10741 107426. Work-Group ID X (1 SGPR) 10743 10744 The value comes from the initial kernel execution state. See 10745 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10746 107477. Work-Group ID Y (1 SGPR) 10748 10749 The value comes from the initial kernel execution state. See 10750 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10751 107528. Work-Group ID Z (1 SGPR) 10753 10754 The value comes from the initial kernel execution state. See 10755 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10756 107579. Implicit Argument Ptr (2 SGPRs) 10758 10759 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10760 global address space pointer to the first kernarg implicit argument. 10761 10762The input and result arguments are assigned in order in the following manner: 10763 10764.. note:: 10765 10766 There are likely some errors and omissions in the following description that 10767 need correction. 10768 10769 .. TODO:: 10770 10771 Check the Clang source code to decipher how function arguments and return 10772 results are handled. Also see the AMDGPU specific values used. 10773 10774* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 10775 VGPR31. 10776 10777 If there are more arguments than will fit in these registers, the remaining 10778 arguments are allocated on the stack in order on naturally aligned 10779 addresses. 10780 10781 .. TODO:: 10782 10783 How are overly aligned structures allocated on the stack? 10784 10785* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 10786 SGPR29. 10787 10788 If there are more arguments than will fit in these registers, the remaining 10789 arguments are allocated on the stack in order on naturally aligned 10790 addresses. 10791 10792Note that decomposed struct type arguments may have some fields passed in 10793registers and some in memory. 10794 10795.. TODO:: 10796 10797 So, a struct which can pass some fields as decomposed register arguments, will 10798 pass the rest as decomposed stack elements? But an argument that will not start 10799 in registers will not be decomposed and will be passed as a non-decomposed 10800 stack value? 10801 10802The following is not part of the AMDGPU function calling convention but 10803describes how the AMDGPU implements function calls: 10804 108051. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 10806 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 10807 are used, or for the reasons defined in ``SIFrameLowering``. 108082. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 10809 to access the incoming stack arguments in the function. The BP is needed 10810 only when the function requires the runtime stack alignment. 10811 108123. Allocating SGPR arguments on the stack are not supported. 10813 108144. No CFI is currently generated. See 10815 :ref:`amdgpu-dwarf-call-frame-information`. 10816 10817 .. note:: 10818 10819 CFI will be generated that defines the CFA as the unswizzled address 10820 relative to the wave scratch base in the unswizzled private address space 10821 of the lowest address stack allocated local variable. 10822 10823 ``DW_AT_frame_base`` will be defined as the swizzled address in the 10824 swizzled private address space by dividing the CFA by the wavefront size 10825 (since CFA is always at least dword aligned which matches the scratch 10826 swizzle element size). 10827 10828 If no dynamic stack alignment was performed, the stack allocated arguments 10829 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 10830 local variables and register spill slots are accessed as positive offsets 10831 relative to ``DW_AT_frame_base``. 10832 108335. Function argument passing is implemented by copying the input physical 10834 registers to virtual registers on entry. The register allocator can spill if 10835 necessary. These are copied back to physical registers at call sites. The 10836 net effect is that each function call can have these values in entirely 10837 distinct locations. The IPRA can help avoid shuffling argument registers. 108386. Call sites are implemented by setting up the arguments at positive offsets 10839 from SP. Then SP is incremented to account for the known frame size before 10840 the call and decremented after the call. 10841 10842 .. note:: 10843 10844 The CFI will reflect the changed calculation needed to compute the CFA 10845 from SP. 10846 108477. 4 byte spill slots are used in the stack frame. One slot is allocated for an 10848 emergency spill slot. Buffer instructions are used for stack accesses and 10849 not the ``flat_scratch`` instruction. 10850 10851 .. TODO:: 10852 10853 Explain when the emergency spill slot is used. 10854 10855.. TODO:: 10856 10857 Possible broken issues: 10858 10859 - Stack arguments must be aligned to required alignment. 10860 - Stack is aligned to max(16, max formal argument alignment) 10861 - Direct argument < 64 bits should check register budget. 10862 - Register budget calculation should respect ``inreg`` for SGPR. 10863 - SGPR overflow is not handled. 10864 - struct with 1 member unpeeling is not checking size of member. 10865 - ``sret`` is after ``this`` pointer. 10866 - Caller is not implementing stack realignment: need an extra pointer. 10867 - Should say AMDGPU passes FP rather than SP. 10868 - Should CFI define CFA as address of locals or arguments. Difference is 10869 apparent when have implemented dynamic alignment. 10870 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 10871 highest address of stack frame and use negative offset for locals. Would 10872 allow SP to be the same as FP and could support signal-handler-like as now 10873 have a real SP for the top of the stack. 10874 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 10875 arguments? 10876 10877AMDPAL 10878------ 10879 10880This section provides code conventions used when the target triple OS is 10881``amdpal`` (see :ref:`amdgpu-target-triples`). 10882 10883.. _amdgpu-amdpal-code-object-metadata-section: 10884 10885Code Object Metadata 10886~~~~~~~~~~~~~~~~~~~~ 10887 10888.. note:: 10889 10890 The metadata is currently in development and is subject to major 10891 changes. Only the current version is supported. *When this document 10892 was generated the version was 2.6.* 10893 10894Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 10895record (see :ref:`amdgpu-note-records-v3-v4`). 10896 10897The metadata is represented as Message Pack formatted binary data (see 10898[MsgPack]_). The top level is a Message Pack map that includes the keys 10899defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 10900and referenced tables. 10901 10902Additional information can be added to the maps. To avoid conflicts, any 10903key names should be prefixed by "*vendor-name*." where ``vendor-name`` 10904can be the name of the vendor and specific vendor tool that generates the 10905information. The prefix is abbreviated to simply "." when it appears 10906within a map that has been added by the same *vendor-name*. 10907 10908 .. table:: AMDPAL Code Object Metadata Map 10909 :name: amdgpu-amdpal-code-object-metadata-map-table 10910 10911 =================== ============== ========= ====================================================================== 10912 String Key Value Type Required? Description 10913 =================== ============== ========= ====================================================================== 10914 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 10915 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 10916 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 10917 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 10918 definition of the keys included in that map. 10919 =================== ============== ========= ====================================================================== 10920 10921.. 10922 10923 .. table:: AMDPAL Code Object Pipeline Metadata Map 10924 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 10925 10926 ====================================== ============== ========= =================================================== 10927 String Key Value Type Required? Description 10928 ====================================== ============== ========= =================================================== 10929 ".name" string Source name of the pipeline. 10930 ".type" string Pipeline type, e.g. VsPs. Values include: 10931 10932 - "VsPs" 10933 - "Gs" 10934 - "Cs" 10935 - "Ngg" 10936 - "Tess" 10937 - "GsTess" 10938 - "NggTess" 10939 10940 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 10941 2 integers 64 bits is the "stable" portion of the hash, used 10942 for e.g. shader replacement lookup. Upper 64 bits 10943 is the "unique" portion of the hash, used for 10944 e.g. pipeline cache lookup. The value is 10945 implementation defined, and can not be relied on 10946 between different builds of the compiler. 10947 ".shaders" map Per-API shader metadata. See 10948 :ref:`amdgpu-amdpal-code-object-shader-map-table` 10949 for the definition of the keys included in that 10950 map. 10951 ".hardware_stages" map Per-hardware stage metadata. See 10952 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 10953 for the definition of the keys included in that 10954 map. 10955 ".shader_functions" map Per-shader function metadata. See 10956 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 10957 for the definition of the keys included in that 10958 map. 10959 ".registers" map Required Hardware register configuration. See 10960 :ref:`amdgpu-amdpal-code-object-register-map-table` 10961 for the definition of the keys included in that 10962 map. 10963 ".user_data_limit" integer Number of user data entries accessed by this 10964 pipeline. 10965 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 10966 NoUserDataSpilling. 10967 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 10968 viewport array index feature. Pipelines which use 10969 this feature can render into all 16 viewports, 10970 whereas pipelines which do not use it are 10971 restricted to viewport #0. 10972 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 10973 handling data-passing between the ES and GS 10974 shader stages. This can be zero if the data is 10975 passed using off-chip buffers. This value should 10976 be used to program all user-SGPRs which have been 10977 marked with "UserDataMapping::EsGsLdsSize" 10978 (typically only the GS and VS HW stages will ever 10979 have a user-SGPR so marked). 10980 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 10981 (maximum number of threads in a subgroup). 10982 ".num_interpolants" integer Graphics only. Number of PS interpolants. 10983 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 10984 ".api" string Name of the client graphics API. 10985 ".api_create_info" binary Graphics API shader create info binary blob. Can 10986 be defined by the driver using the compiler if 10987 they want to be able to correlate API-specific 10988 information used during creation at a later time. 10989 ====================================== ============== ========= =================================================== 10990 10991.. 10992 10993 .. table:: AMDPAL Code Object Shader Map 10994 :name: amdgpu-amdpal-code-object-shader-map-table 10995 10996 10997 +-------------+--------------+-------------------------------------------------------------------+ 10998 |String Key |Value Type |Description | 10999 +=============+==============+===================================================================+ 11000 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 11001 |- ".vertex" | |for the definition of the keys included in that map. | 11002 |- ".hull" | | | 11003 |- ".domain" | | | 11004 |- ".geometry"| | | 11005 |- ".pixel" | | | 11006 +-------------+--------------+-------------------------------------------------------------------+ 11007 11008.. 11009 11010 .. table:: AMDPAL Code Object API Shader Metadata Map 11011 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 11012 11013 ==================== ============== ========= ===================================================================== 11014 String Key Value Type Required? Description 11015 ==================== ============== ========= ===================================================================== 11016 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 11017 2 integers is implementation defined, and can not be relied on between 11018 different builds of the compiler. 11019 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 11020 string include: 11021 11022 - ".ls" 11023 - ".hs" 11024 - ".es" 11025 - ".gs" 11026 - ".vs" 11027 - ".ps" 11028 - ".cs" 11029 11030 ==================== ============== ========= ===================================================================== 11031 11032.. 11033 11034 .. table:: AMDPAL Code Object Hardware Stage Map 11035 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 11036 11037 +-------------+--------------+-----------------------------------------------------------------------+ 11038 |String Key |Value Type |Description | 11039 +=============+==============+=======================================================================+ 11040 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 11041 |- ".hs" | |for the definition of the keys included in that map. | 11042 |- ".es" | | | 11043 |- ".gs" | | | 11044 |- ".vs" | | | 11045 |- ".ps" | | | 11046 |- ".cs" | | | 11047 +-------------+--------------+-----------------------------------------------------------------------+ 11048 11049.. 11050 11051 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 11052 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 11053 11054 ========================== ============== ========= =============================================================== 11055 String Key Value Type Required? Description 11056 ========================== ============== ========= =============================================================== 11057 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 11058 ".scratch_memory_size" integer Scratch memory size in bytes. 11059 ".lds_size" integer Local Data Share size in bytes. 11060 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 11061 ".vgpr_count" integer Number of VGPRs used. 11062 ".sgpr_count" integer Number of SGPRs used. 11063 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 11064 directive to instruct the compiler to limit the VGPR usage to 11065 be less than or equal to the specified value (only set if 11066 different from HW default). 11067 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 11068 default). 11069 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 11070 3 integers 11071 ".wavefront_size" integer Wavefront size (only set if different from HW default). 11072 ".uses_uavs" boolean The shader reads or writes UAVs. 11073 ".uses_rovs" boolean The shader reads or writes ROVs. 11074 ".writes_uavs" boolean The shader writes to one or more UAVs. 11075 ".writes_depth" boolean The shader writes out a depth value. 11076 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 11077 memory or GDS. 11078 ".uses_prim_id" boolean The shader uses PrimID. 11079 ========================== ============== ========= =============================================================== 11080 11081.. 11082 11083 .. table:: AMDPAL Code Object Shader Function Map 11084 :name: amdgpu-amdpal-code-object-shader-function-map-table 11085 11086 =============== ============== ==================================================================== 11087 String Key Value Type Description 11088 =============== ============== ==================================================================== 11089 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 11090 entry address. The value is the function's metadata. See 11091 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 11092 =============== ============== ==================================================================== 11093 11094.. 11095 11096 .. table:: AMDPAL Code Object Shader Function Metadata Map 11097 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 11098 11099 ============================= ============== ================================================================= 11100 String Key Value Type Description 11101 ============================= ============== ================================================================= 11102 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 11103 2 integers is implementation defined, and can not be relied on between 11104 different builds of the compiler. 11105 ".scratch_memory_size" sequence of Size in bytes of scratch memory used by the shader. 11106 2 integers 11107 ".lds_size" sequence of Size in bytes of LDS memory. 11108 2 integers 11109 ".vgpr_count" integer Number of VGPRs used by the shader. 11110 ".sgpr_count" integer Number of SGPRs used by the shader. 11111 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 11112 ".shader_subtype" string Shader subtype/kind. Values include: 11113 11114 - "Unknown" 11115 11116 ============================= ============== ================================================================= 11117 11118.. 11119 11120 .. table:: AMDPAL Code Object Register Map 11121 :name: amdgpu-amdpal-code-object-register-map-table 11122 11123 ========================== ============== ==================================================================== 11124 32-bit Integer Key Value Type Description 11125 ========================== ============== ==================================================================== 11126 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 11127 a GRBM register (i.e., driver accessible GPU register number, not 11128 shader GPR register number). The driver is required to program each 11129 specified register to the corresponding specified value when 11130 executing this pipeline. Typically, the ``reg offsets`` are the 11131 ``uint16_t`` offsets to each register as defined by the hardware 11132 chip headers. The register is set to the provided value. However, a 11133 ``reg offset`` that specifies a user data register (e.g., 11134 COMPUTE_USER_DATA_0) needs special treatment. See 11135 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 11136 information. 11137 ========================== ============== ==================================================================== 11138 11139.. _amdgpu-amdpal-code-object-user-data-section: 11140 11141User Data 11142+++++++++ 11143 11144Each hardware stage has a set of 32-bit physical SPI *user data registers* 11145(either 16 or 32 based on graphics IP and the stage) which can be 11146written from a command buffer and then loaded into SGPRs when waves are 11147launched via a subsequent dispatch or draw operation. This is the way 11148most arguments are passed from the application/runtime to a hardware 11149shader. 11150 11151PAL abstracts this functionality by exposing a set of 128 *user data 11152entries* per pipeline a client can use to pass arguments from a command 11153buffer to one or more shaders in that pipeline. The ELF code object must 11154specify a mapping from virtualized *user data entries* to physical *user 11155data registers*, and PAL is responsible for implementing that mapping, 11156including spilling overflow *user data entries* to memory if needed. 11157 11158Since the *user data registers* are GRBM-accessible SPI registers, this 11159mapping is actually embedded in the ``.registers`` metadata entry. For 11160most registers, the value in that map is a literal 32-bit value that 11161should be written to the register by the driver. However, when the 11162register is a *user data register* (any USER_DATA register e.g., 11163SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 11164the driver to write either a *user data entry* value or one of several 11165driver-internal values to the register. This encoding is described in 11166the following table: 11167 11168.. note:: 11169 11170 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 11171 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 11172 always be programmed to the address of the GlobalTable, and *user data 11173 register* 1 must always be programmed to the address of the PerShaderTable. 11174 11175.. 11176 11177 .. table:: AMDPAL User Data Mapping 11178 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 11179 11180 ========== ================= =============================================================================== 11181 Value Name Description 11182 ========== ================= =============================================================================== 11183 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 11184 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 11185 always point to *user data register* 0). 11186 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 11187 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 11188 for more detail (should always point to *user data register* 1). 11189 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 11190 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 11191 more detail. 11192 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 11193 reference the draw index in the vertex shader. Only supported by the first 11194 stage in a graphics pipeline. 11195 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 11196 a graphics pipeline. 11197 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 11198 graphics pipeline. 11199 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 11200 a buffer containing the grid dimensions for a Compute dispatch operation. The 11201 high half of the address is stored in the next sequential user-SGPR. Only 11202 supported by compute pipelines. 11203 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 11204 space used for the ES/GS pseudo-ring-buffer for passing data between shader 11205 stages. 11206 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 11207 pipeline instancing. 11208 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 11209 can only appear for one shader stage per pipeline. 11210 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 11211 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 11212 only appear for one shader stage per pipeline. 11213 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 11214 only appear for one shader stage per pipeline (PS). These replace color targets 11215 and are completely separate from any UAVs used by the shader. This is optional, 11216 and only used by the PS when UAV exports are used to replace color-target 11217 exports to optimize specific shaders. 11218 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 11219 some NGG pipelines to perform culling. This value contains the address of the 11220 first of two consecutive registers which provide the full GPU address. 11221 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 11222 ========== ================= =============================================================================== 11223 11224.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 11225 11226Per-Shader Table 11227################ 11228 11229Low 32 bits of the GPU address for an optional buffer in the ``.data`` 11230section of the ELF. The high 32 bits of the address match the high 32 bits 11231of the shader's program counter. 11232 11233The buffer can be anything the shader compiler needs it for, and 11234allows each shader to have its own region of the ``.data`` section. 11235Typically, this could be a table of buffer SRD's and the data pointed to 11236by the buffer SRD's, but it could be a flat-address region of memory as 11237well. Its layout and usage are defined by the shader compiler. 11238 11239Each shader's table in the ``.data`` section is referenced by the symbol 11240``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 11241hardware shader stage the data is for. E.g., 11242``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 11243 11244.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 11245 11246Spill Table 11247########### 11248 11249It is possible for a hardware shader to need access to more *user data 11250entries* than there are slots available in user data registers for one 11251or more hardware shader stages. In that case, the PAL runtime expects 11252the necessary *user data entries* to be spilled to GPU memory and use 11253one user data register to point to the spilled user data memory. The 11254value of the *user data entry* must then represent the location where 11255a shader expects to read the low 32-bits of the table's GPU virtual 11256address. The *spill table* itself represents a set of 32-bit values 11257managed by the PAL runtime in GPU-accessible memory that can be made 11258indirectly accessible to a hardware shader. 11259 11260Unspecified OS 11261-------------- 11262 11263This section provides code conventions used when the target triple OS is 11264empty (see :ref:`amdgpu-target-triples`). 11265 11266Trap Handler ABI 11267~~~~~~~~~~~~~~~~ 11268 11269For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11270not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11271instructions are handled as follows: 11272 11273 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11274 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11275 11276 =============== =============== =========================================== 11277 Usage Code Sequence Description 11278 =============== =============== =========================================== 11279 llvm.trap s_endpgm Causes wavefront to be terminated. 11280 llvm.debugtrap *none* Compiler warning given that there is no 11281 trap handler installed. 11282 =============== =============== =========================================== 11283 11284Source Languages 11285================ 11286 11287.. _amdgpu-opencl: 11288 11289OpenCL 11290------ 11291 11292When the language is OpenCL the following differences occur: 11293 112941. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 112952. The AMDGPU backend appends additional arguments to the kernel's explicit 11296 arguments for the AMDHSA OS (see 11297 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 112983. Additional metadata is generated 11299 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11300 11301 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11302 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11303 11304 ======== ==== ========= =========================================== 11305 Position Byte Byte Description 11306 Size Alignment 11307 ======== ==== ========= =========================================== 11308 1 8 8 OpenCL Global Offset X 11309 2 8 8 OpenCL Global Offset Y 11310 3 8 8 OpenCL Global Offset Z 11311 4 8 8 OpenCL address of printf buffer 11312 5 8 8 OpenCL address of virtual queue used by 11313 enqueue_kernel. 11314 6 8 8 OpenCL address of AqlWrap struct used by 11315 enqueue_kernel. 11316 7 8 8 Pointer argument used for Multi-gird 11317 synchronization. 11318 ======== ==== ========= =========================================== 11319 11320.. _amdgpu-hcc: 11321 11322HCC 11323--- 11324 11325When the language is HCC the following differences occur: 11326 113271. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11328 11329.. _amdgpu-assembler: 11330 11331Assembler 11332--------- 11333 11334AMDGPU backend has LLVM-MC based assembler which is currently in development. 11335It supports AMDGCN GFX6-GFX10. 11336 11337This section describes general syntax for instructions and operands. 11338 11339Instructions 11340~~~~~~~~~~~~ 11341 11342An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11343 11344 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11345 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11346 11347:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11348:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11349 11350The order of operands and modifiers is fixed. 11351Most modifiers are optional and may be omitted. 11352 11353Links to detailed instruction syntax description may be found in the following 11354table. Note that features under development are not included 11355in this description. 11356 11357 =================================== ======================================= 11358 Core ISA ISA Extensions 11359 =================================== ======================================= 11360 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11361 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11362 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11363 11364 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11365 11366 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11367 11368 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11369 11370 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11371 11372 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11373 11374 :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 11375 11376 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11377 11378 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11379 =================================== ======================================= 11380 11381For more information about instructions, their semantics and supported 11382combinations of operands, refer to one of instruction set architecture manuals 11383[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_, 11384[AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11385 11386Operands 11387~~~~~~~~ 11388 11389Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11390 11391Modifiers 11392~~~~~~~~~ 11393 11394Detailed description of modifiers may be found 11395:doc:`here<AMDGPUModifierSyntax>`. 11396 11397Instruction Examples 11398~~~~~~~~~~~~~~~~~~~~ 11399 11400DS 11401++ 11402 11403.. code-block:: nasm 11404 11405 ds_add_u32 v2, v4 offset:16 11406 ds_write_src2_b64 v2 offset0:4 offset1:8 11407 ds_cmpst_f32 v2, v4, v6 11408 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11409 11410For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11411Manual. 11412 11413FLAT 11414++++ 11415 11416.. code-block:: nasm 11417 11418 flat_load_dword v1, v[3:4] 11419 flat_store_dwordx3 v[3:4], v[5:7] 11420 flat_atomic_swap v1, v[3:4], v5 glc 11421 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11422 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11423 11424For full list of supported instructions, refer to "FLAT instructions" in ISA 11425Manual. 11426 11427MUBUF 11428+++++ 11429 11430.. code-block:: nasm 11431 11432 buffer_load_dword v1, off, s[4:7], s1 11433 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11434 buffer_store_format_xy v[1:2], off, s[4:7], s1 11435 buffer_wbinvl1 11436 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11437 11438For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11439Manual. 11440 11441SMRD/SMEM 11442+++++++++ 11443 11444.. code-block:: nasm 11445 11446 s_load_dword s1, s[2:3], 0xfc 11447 s_load_dwordx8 s[8:15], s[2:3], s4 11448 s_load_dwordx16 s[88:103], s[2:3], s4 11449 s_dcache_inv_vol 11450 s_memtime s[4:5] 11451 11452For full list of supported instructions, refer to "Scalar Memory Operations" in 11453ISA Manual. 11454 11455SOP1 11456++++ 11457 11458.. code-block:: nasm 11459 11460 s_mov_b32 s1, s2 11461 s_mov_b64 s[0:1], 0x80000000 11462 s_cmov_b32 s1, 200 11463 s_wqm_b64 s[2:3], s[4:5] 11464 s_bcnt0_i32_b64 s1, s[2:3] 11465 s_swappc_b64 s[2:3], s[4:5] 11466 s_cbranch_join s[4:5] 11467 11468For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11469Manual. 11470 11471SOP2 11472++++ 11473 11474.. code-block:: nasm 11475 11476 s_add_u32 s1, s2, s3 11477 s_and_b64 s[2:3], s[4:5], s[6:7] 11478 s_cselect_b32 s1, s2, s3 11479 s_andn2_b32 s2, s4, s6 11480 s_lshr_b64 s[2:3], s[4:5], s6 11481 s_ashr_i32 s2, s4, s6 11482 s_bfm_b64 s[2:3], s4, s6 11483 s_bfe_i64 s[2:3], s[4:5], s6 11484 s_cbranch_g_fork s[4:5], s[6:7] 11485 11486For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11487Manual. 11488 11489SOPC 11490++++ 11491 11492.. code-block:: nasm 11493 11494 s_cmp_eq_i32 s1, s2 11495 s_bitcmp1_b32 s1, s2 11496 s_bitcmp0_b64 s[2:3], s4 11497 s_setvskip s3, s5 11498 11499For full list of supported instructions, refer to "SOPC Instructions" in ISA 11500Manual. 11501 11502SOPP 11503++++ 11504 11505.. code-block:: nasm 11506 11507 s_barrier 11508 s_nop 2 11509 s_endpgm 11510 s_waitcnt 0 ; Wait for all counters to be 0 11511 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11512 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11513 s_sethalt 9 11514 s_sleep 10 11515 s_sendmsg 0x1 11516 s_sendmsg sendmsg(MSG_INTERRUPT) 11517 s_trap 1 11518 11519For full list of supported instructions, refer to "SOPP Instructions" in ISA 11520Manual. 11521 11522Unless otherwise mentioned, little verification is performed on the operands 11523of SOPP Instructions, so it is up to the programmer to be familiar with the 11524range or acceptable values. 11525 11526VALU 11527++++ 11528 11529For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11530the assembler will automatically use optimal encoding based on its operands. To 11531force specific encoding, one can add a suffix to the opcode of the instruction: 11532 11533* _e32 for 32-bit VOP1/VOP2/VOPC 11534* _e64 for 64-bit VOP3 11535* _dpp for VOP_DPP 11536* _sdwa for VOP_SDWA 11537 11538VOP1/VOP2/VOP3/VOPC examples: 11539 11540.. code-block:: nasm 11541 11542 v_mov_b32 v1, v2 11543 v_mov_b32_e32 v1, v2 11544 v_nop 11545 v_cvt_f64_i32_e32 v[1:2], v2 11546 v_floor_f32_e32 v1, v2 11547 v_bfrev_b32_e32 v1, v2 11548 v_add_f32_e32 v1, v2, v3 11549 v_mul_i32_i24_e64 v1, v2, 3 11550 v_mul_i32_i24_e32 v1, -3, v3 11551 v_mul_i32_i24_e32 v1, -100, v3 11552 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11553 v_max_f16_e32 v1, v2, v3 11554 11555VOP_DPP examples: 11556 11557.. code-block:: nasm 11558 11559 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11560 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11561 v_mov_b32 v0, v0 wave_shl:1 11562 v_mov_b32 v0, v0 row_mirror 11563 v_mov_b32 v0, v0 row_bcast:31 11564 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11565 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11566 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11567 11568VOP_SDWA examples: 11569 11570.. code-block:: nasm 11571 11572 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11573 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11574 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11575 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11576 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11577 11578For full list of supported instructions, refer to "Vector ALU instructions". 11579 11580.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11581 11582Code Object V2 Predefined Symbols 11583~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11584 11585.. warning:: 11586 Code object V2 is not the default code object version emitted by 11587 this version of LLVM. 11588 11589The AMDGPU assembler defines and updates some symbols automatically. These 11590symbols do not affect code generation. 11591 11592.option.machine_version_major 11593+++++++++++++++++++++++++++++ 11594 11595Set to the GFX major generation number of the target being assembled for. For 11596example, when assembling for a "GFX9" target this will be set to the integer 11597value "9". The possible GFX major generation numbers are presented in 11598:ref:`amdgpu-processors`. 11599 11600.option.machine_version_minor 11601+++++++++++++++++++++++++++++ 11602 11603Set to the GFX minor generation number of the target being assembled for. For 11604example, when assembling for a "GFX810" target this will be set to the integer 11605value "1". The possible GFX minor generation numbers are presented in 11606:ref:`amdgpu-processors`. 11607 11608.option.machine_version_stepping 11609++++++++++++++++++++++++++++++++ 11610 11611Set to the GFX stepping generation number of the target being assembled for. 11612For example, when assembling for a "GFX704" target this will be set to the 11613integer value "4". The possible GFX stepping generation numbers are presented 11614in :ref:`amdgpu-processors`. 11615 11616.kernel.vgpr_count 11617++++++++++++++++++ 11618 11619Set to zero each time a 11620:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11621encountered. At each instruction, if the current value of this symbol is less 11622than or equal to the maximum VGPR number explicitly referenced within that 11623instruction then the symbol value is updated to equal that VGPR number plus 11624one. 11625 11626.kernel.sgpr_count 11627++++++++++++++++++ 11628 11629Set to zero each time a 11630:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11631encountered. At each instruction, if the current value of this symbol is less 11632than or equal to the maximum VGPR number explicitly referenced within that 11633instruction then the symbol value is updated to equal that SGPR number plus 11634one. 11635 11636.. _amdgpu-amdhsa-assembler-directives-v2: 11637 11638Code Object V2 Directives 11639~~~~~~~~~~~~~~~~~~~~~~~~~ 11640 11641.. warning:: 11642 Code object V2 is not the default code object version emitted by 11643 this version of LLVM. 11644 11645AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11646one can specify them with assembler directives. 11647 11648.hsa_code_object_version major, minor 11649+++++++++++++++++++++++++++++++++++++ 11650 11651*major* and *minor* are integers that specify the version of the HSA code 11652object that will be generated by the assembler. 11653 11654.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11655+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11656 11657 11658*major*, *minor*, and *stepping* are all integers that describe the instruction 11659set architecture (ISA) version of the assembly program. 11660 11661*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11662"AMD" and *arch* should always be equal to "AMDGPU". 11663 11664By default, the assembler will derive the ISA version, *vendor*, and *arch* 11665from the value of the -mcpu option that is passed to the assembler. 11666 11667.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11668 11669.amdgpu_hsa_kernel (name) 11670+++++++++++++++++++++++++ 11671 11672This directives specifies that the symbol with given name is a kernel entry 11673point (label) and the object should contain corresponding symbol of type 11674STT_AMDGPU_HSA_KERNEL. 11675 11676.amd_kernel_code_t 11677++++++++++++++++++ 11678 11679This directive marks the beginning of a list of key / value pairs that are used 11680to specify the amd_kernel_code_t object that will be emitted by the assembler. 11681The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11682amd_kernel_code_t values that are unspecified a default value will be used. The 11683default value for all keys is 0, with the following exceptions: 11684 11685- *amd_code_version_major* defaults to 1. 11686- *amd_kernel_code_version_minor* defaults to 2. 11687- *amd_machine_kind* defaults to 1. 11688- *amd_machine_version_major*, *machine_version_minor*, and 11689 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11690 that is passed to the assembler. 11691- *kernel_code_entry_byte_offset* defaults to 256. 11692- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11693 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11694 Note that wavefront size is specified as a power of two, so a value of **n** 11695 means a size of 2^ **n**. 11696- *call_convention* defaults to -1. 11697- *kernarg_segment_alignment*, *group_segment_alignment*, and 11698 *private_segment_alignment* default to 4. Note that alignments are specified 11699 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11700- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11701 GFX90A onwards. 11702- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11703 GFX10 onwards. 11704- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11705 11706The *.amd_kernel_code_t* directive must be placed immediately after the 11707function label and before any instructions. 11708 11709For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11710comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11711 11712.. _amdgpu-amdhsa-assembler-example-v2: 11713 11714Code Object V2 Example Source Code 11715~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11716 11717.. warning:: 11718 Code Object V2 is not the default code object version emitted by 11719 this version of LLVM. 11720 11721Here is an example of a minimal assembly source file, defining one HSA kernel: 11722 11723.. code:: 11724 :number-lines: 11725 11726 .hsa_code_object_version 1,0 11727 .hsa_code_object_isa 11728 11729 .hsatext 11730 .globl hello_world 11731 .p2align 8 11732 .amdgpu_hsa_kernel hello_world 11733 11734 hello_world: 11735 11736 .amd_kernel_code_t 11737 enable_sgpr_kernarg_segment_ptr = 1 11738 is_ptr64 = 1 11739 compute_pgm_rsrc1_vgprs = 0 11740 compute_pgm_rsrc1_sgprs = 0 11741 compute_pgm_rsrc2_user_sgpr = 2 11742 compute_pgm_rsrc1_wgp_mode = 0 11743 compute_pgm_rsrc1_mem_ordered = 0 11744 compute_pgm_rsrc1_fwd_progress = 1 11745 .end_amd_kernel_code_t 11746 11747 s_load_dwordx2 s[0:1], s[0:1] 0x0 11748 v_mov_b32 v0, 3.14159 11749 s_waitcnt lgkmcnt(0) 11750 v_mov_b32 v1, s0 11751 v_mov_b32 v2, s1 11752 flat_store_dword v[1:2], v0 11753 s_endpgm 11754 .Lfunc_end0: 11755 .size hello_world, .Lfunc_end0-hello_world 11756 11757.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11758 11759Code Object V3 to V4 Predefined Symbols 11760~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11761 11762The AMDGPU assembler defines and updates some symbols automatically. These 11763symbols do not affect code generation. 11764 11765.amdgcn.gfx_generation_number 11766+++++++++++++++++++++++++++++ 11767 11768Set to the GFX major generation number of the target being assembled for. For 11769example, when assembling for a "GFX9" target this will be set to the integer 11770value "9". The possible GFX major generation numbers are presented in 11771:ref:`amdgpu-processors`. 11772 11773.amdgcn.gfx_generation_minor 11774++++++++++++++++++++++++++++ 11775 11776Set to the GFX minor generation number of the target being assembled for. For 11777example, when assembling for a "GFX810" target this will be set to the integer 11778value "1". The possible GFX minor generation numbers are presented in 11779:ref:`amdgpu-processors`. 11780 11781.amdgcn.gfx_generation_stepping 11782+++++++++++++++++++++++++++++++ 11783 11784Set to the GFX stepping generation number of the target being assembled for. 11785For example, when assembling for a "GFX704" target this will be set to the 11786integer value "4". The possible GFX stepping generation numbers are presented 11787in :ref:`amdgpu-processors`. 11788 11789.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 11790 11791.amdgcn.next_free_vgpr 11792++++++++++++++++++++++ 11793 11794Set to zero before assembly begins. At each instruction, if the current value 11795of this symbol is less than or equal to the maximum VGPR number explicitly 11796referenced within that instruction then the symbol value is updated to equal 11797that VGPR number plus one. 11798 11799May be used to set the `.amdhsa_next_free_vgpr` directive in 11800:ref:`amdhsa-kernel-directives-table`. 11801 11802May be set at any time, e.g. manually set to zero at the start of each kernel. 11803 11804.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 11805 11806.amdgcn.next_free_sgpr 11807++++++++++++++++++++++ 11808 11809Set to zero before assembly begins. At each instruction, if the current value 11810of this symbol is less than or equal the maximum SGPR number explicitly 11811referenced within that instruction then the symbol value is updated to equal 11812that SGPR number plus one. 11813 11814May be used to set the `.amdhsa_next_free_spgr` directive in 11815:ref:`amdhsa-kernel-directives-table`. 11816 11817May be set at any time, e.g. manually set to zero at the start of each kernel. 11818 11819.. _amdgpu-amdhsa-assembler-directives-v3-v4: 11820 11821Code Object V3 to V4 Directives 11822~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11823 11824Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 11825architecture processors, and are not OS-specific. Directives which begin with 11826``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 11827``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 11828:ref:`amdgpu-processors`. 11829 11830.. _amdgpu-assembler-directive-amdgcn-target: 11831 11832.amdgcn_target <target-triple> "-" <target-id> 11833++++++++++++++++++++++++++++++++++++++++++++++ 11834 11835Optional directive which declares the ``<target-triple>-<target-id>`` supported 11836by the containing assembler source file. Used by the assembler to validate 11837command-line options such as ``-triple``, ``-mcpu``, and 11838``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 11839:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 11840 11841.. note:: 11842 11843 The target ID syntax used for code object V2 to V3 for this directive differs 11844 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 11845 11846.amdhsa_kernel <name> 11847+++++++++++++++++++++ 11848 11849Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 11850``<name>.kd``, in the current location of the current section. Only valid when 11851the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 11852instruction to execute, and does not need to be previously defined. 11853 11854Marks the beginning of a list of directives used to generate the bytes of a 11855kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 11856Directives which may appear in this list are described in 11857:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 11858be valid for the target being assembled for, and cannot be repeated. Directives 11859support the range of values specified by the field they reference in 11860:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 11861assumed to have its default value, unless it is marked as "Required", in which 11862case it is an error to omit the directive. This list of directives is 11863terminated by an ``.end_amdhsa_kernel`` directive. 11864 11865 .. table:: AMDHSA Kernel Assembler Directives 11866 :name: amdhsa-kernel-directives-table 11867 11868 ======================================================== =================== ============ =================== 11869 Directive Default Supported On Description 11870 ======================================================== =================== ============ =================== 11871 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 11872 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11873 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 11874 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11875 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 11876 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11877 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 11878 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11879 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 11880 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11881 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 11882 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11883 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 11884 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11885 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 11886 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11887 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 11888 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11889 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 11890 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11891 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 11892 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11893 Specific 11894 (wavefrontsize64) 11895 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 11896 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11897 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 11898 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11899 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 11900 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11901 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 11902 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11903 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 11904 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11905 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 11906 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11907 Possible values are defined in 11908 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 11909 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 11910 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 11911 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11912 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 11913 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11914 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11915 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 11916 Used to calculate ACCUM_OFFSET in 11917 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11918 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 11919 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11920 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11921 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 11922 scratch memory. Used to calculate 11923 GRANULATED_WAVEFRONT_SGPR_COUNT in 11924 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11925 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 11926 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11927 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11928 (xnack) 11929 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 11930 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11931 Possible values are defined in 11932 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11933 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 11934 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11935 Possible values are defined in 11936 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11937 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 11938 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11939 Possible values are defined in 11940 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11941 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 11942 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11943 Possible values are defined in 11944 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11945 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 11946 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11947 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 11948 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11949 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 11950 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11951 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 11952 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11953 Specific 11954 (tgsplit) 11955 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 11956 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11957 Specific 11958 (cumode) 11959 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 11960 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11961 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 11962 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11963 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 11964 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11965 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 11966 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11967 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 11968 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11969 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 11970 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11971 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 11972 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11973 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 11974 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11975 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 11976 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11977 ======================================================== =================== ============ =================== 11978 11979.amdgpu_metadata 11980++++++++++++++++ 11981 11982Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 11983note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 11984 11985The contents must be in the [YAML]_ markup format, with the same structure and 11986semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 11987:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 11988 11989This directive is terminated by an ``.end_amdgpu_metadata`` directive. 11990 11991.. _amdgpu-amdhsa-assembler-example-v3-v4: 11992 11993Code Object V3 to V4 Example Source Code 11994~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11995 11996Here is an example of a minimal assembly source file, defining one HSA kernel: 11997 11998.. code:: 11999 :number-lines: 12000 12001 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12002 12003 .text 12004 .globl hello_world 12005 .p2align 8 12006 .type hello_world,@function 12007 hello_world: 12008 s_load_dwordx2 s[0:1], s[0:1] 0x0 12009 v_mov_b32 v0, 3.14159 12010 s_waitcnt lgkmcnt(0) 12011 v_mov_b32 v1, s0 12012 v_mov_b32 v2, s1 12013 flat_store_dword v[1:2], v0 12014 s_endpgm 12015 .Lfunc_end0: 12016 .size hello_world, .Lfunc_end0-hello_world 12017 12018 .rodata 12019 .p2align 6 12020 .amdhsa_kernel hello_world 12021 .amdhsa_user_sgpr_kernarg_segment_ptr 1 12022 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12023 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12024 .end_amdhsa_kernel 12025 12026 .amdgpu_metadata 12027 --- 12028 amdhsa.version: 12029 - 1 12030 - 0 12031 amdhsa.kernels: 12032 - .name: hello_world 12033 .symbol: hello_world.kd 12034 .kernarg_segment_size: 48 12035 .group_segment_fixed_size: 0 12036 .private_segment_fixed_size: 0 12037 .kernarg_segment_align: 4 12038 .wavefront_size: 64 12039 .sgpr_count: 2 12040 .vgpr_count: 3 12041 .max_flat_workgroup_size: 256 12042 ... 12043 .end_amdgpu_metadata 12044 12045If an assembly source file contains multiple kernels and/or functions, the 12046:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 12047:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 12048the ``.set <symbol>, <expression>`` directive. For example, in the case of two 12049kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 12050to group the function with the kernel that calls it and reset the symbols 12051between the two connected components: 12052 12053.. code:: 12054 :number-lines: 12055 12056 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12057 12058 // gpr tracking symbols are implicitly set to zero 12059 12060 .text 12061 .globl kern0 12062 .p2align 8 12063 .type kern0,@function 12064 kern0: 12065 // ... 12066 s_endpgm 12067 .Lkern0_end: 12068 .size kern0, .Lkern0_end-kern0 12069 12070 .rodata 12071 .p2align 6 12072 .amdhsa_kernel kern0 12073 // ... 12074 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12075 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12076 .end_amdhsa_kernel 12077 12078 // reset symbols to begin tracking usage in func1 and kern1 12079 .set .amdgcn.next_free_vgpr, 0 12080 .set .amdgcn.next_free_sgpr, 0 12081 12082 .text 12083 .hidden func1 12084 .global func1 12085 .p2align 2 12086 .type func1,@function 12087 func1: 12088 // ... 12089 s_setpc_b64 s[30:31] 12090 .Lfunc1_end: 12091 .size func1, .Lfunc1_end-func1 12092 12093 .globl kern1 12094 .p2align 8 12095 .type kern1,@function 12096 kern1: 12097 // ... 12098 s_getpc_b64 s[4:5] 12099 s_add_u32 s4, s4, func1@rel32@lo+4 12100 s_addc_u32 s5, s5, func1@rel32@lo+4 12101 s_swappc_b64 s[30:31], s[4:5] 12102 // ... 12103 s_endpgm 12104 .Lkern1_end: 12105 .size kern1, .Lkern1_end-kern1 12106 12107 .rodata 12108 .p2align 6 12109 .amdhsa_kernel kern1 12110 // ... 12111 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12112 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12113 .end_amdhsa_kernel 12114 12115These symbols cannot identify connected components in order to automatically 12116track the usage for each kernel. However, in some cases careful organization of 12117the kernels and functions in the source file means there is minimal additional 12118effort required to accurately calculate GPR usage. 12119 12120Additional Documentation 12121======================== 12122 12123.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 12124.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 12125.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 12126.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 12127.. [AMD-GCN-GFX10-RDNA1] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 12128.. [AMD-GCN-GFX10-RDNA2] `AMD "RDNA 2" Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 12129.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 12130.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 12131.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 12132.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 12133.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 12134.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12135.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12136.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12137.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12138.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12139.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12140.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12141.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12142.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12143.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12144.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12145