1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPUModifierSyntax 22 AMDGPUOperandSyntax 23 AMDGPUInstructionSyntax 24 AMDGPUInstructionNotation 25 AMDGPUDwarfExtensionsForHeterogeneousDebugging 26 27Introduction 28============ 29 30The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 31R600 family up until the current GCN families. It lives in the 32``llvm/lib/Target/AMDGPU`` directory. 33 34LLVM 35==== 36 37.. _amdgpu-target-triples: 38 39Target Triples 40-------------- 41 42Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 43to specify the target triple: 44 45 .. table:: AMDGPU Architectures 46 :name: amdgpu-architecture-table 47 48 ============ ============================================================== 49 Architecture Description 50 ============ ============================================================== 51 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 52 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 53 ============ ============================================================== 54 55 .. table:: AMDGPU Vendors 56 :name: amdgpu-vendor-table 57 58 ============ ============================================================== 59 Vendor Description 60 ============ ============================================================== 61 ``amd`` Can be used for all AMD GPU usage. 62 ``mesa3d`` Can be used if the OS is ``mesa3d``. 63 ============ ============================================================== 64 65 .. table:: AMDGPU Operating Systems 66 :name: amdgpu-os 67 68 ============== ============================================================ 69 OS Description 70 ============== ============================================================ 71 *<empty>* Defaults to the *unknown* OS. 72 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 73 such as: 74 75 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 76 loader on Linux. See *AMD ROCm Platform Release Notes* 77 [AMD-ROCm-Release-Notes]_ for supported hardware and 78 software. 79 - AMD's PAL runtime using the *pal-amdhsa* loader on 80 Windows. 81 82 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 83 runtime using the *pal-amdpal* loader on Windows and Linux 84 Pro. 85 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 86 3D runtime using the *mesa-mesa3d* loader on Linux. 87 ============== ============================================================ 88 89 .. table:: AMDGPU Environments 90 :name: amdgpu-environment-table 91 92 ============ ============================================================== 93 Environment Description 94 ============ ============================================================== 95 *<empty>* Default. 96 ============ ============================================================== 97 98.. _amdgpu-processors: 99 100Processors 101---------- 102 103Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 104specify the AMDGPU processor together with optional target features. See 105:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 106specific information. 107 108Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 109 110* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 111 112 113 .. table:: AMDGPU Processors 114 :name: amdgpu-processor-table 115 116 =========== =============== ============ ===== ================= =============== =============== ====================== 117 Processor Alternative Target dGPU/ Target Target OS Support Example 118 Processor Triple APU Features Properties *(see* Products 119 Architecture Supported `amdgpu-os`_ 120 *and 121 corresponding 122 runtime release 123 notes for 124 current 125 information and 126 level of 127 support)* 128 =========== =============== ============ ===== ================= =============== =============== ====================== 129 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 130 ----------------------------------------------------------------------------------------------------------------------- 131 ``r600`` ``r600`` dGPU - Does not 132 support 133 generic 134 address 135 space 136 ``r630`` ``r600`` dGPU - Does not 137 support 138 generic 139 address 140 space 141 ``rs880`` ``r600`` dGPU - Does not 142 support 143 generic 144 address 145 space 146 ``rv670`` ``r600`` dGPU - Does not 147 support 148 generic 149 address 150 space 151 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 152 ----------------------------------------------------------------------------------------------------------------------- 153 ``rv710`` ``r600`` dGPU - Does not 154 support 155 generic 156 address 157 space 158 ``rv730`` ``r600`` dGPU - Does not 159 support 160 generic 161 address 162 space 163 ``rv770`` ``r600`` dGPU - Does not 164 support 165 generic 166 address 167 space 168 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 169 ----------------------------------------------------------------------------------------------------------------------- 170 ``cedar`` ``r600`` dGPU - Does not 171 support 172 generic 173 address 174 space 175 ``cypress`` ``r600`` dGPU - Does not 176 support 177 generic 178 address 179 space 180 ``juniper`` ``r600`` dGPU - Does not 181 support 182 generic 183 address 184 space 185 ``redwood`` ``r600`` dGPU - Does not 186 support 187 generic 188 address 189 space 190 ``sumo`` ``r600`` dGPU - Does not 191 support 192 generic 193 address 194 space 195 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 196 ----------------------------------------------------------------------------------------------------------------------- 197 ``barts`` ``r600`` dGPU - Does not 198 support 199 generic 200 address 201 space 202 ``caicos`` ``r600`` dGPU - Does not 203 support 204 generic 205 address 206 space 207 ``cayman`` ``r600`` dGPU - Does not 208 support 209 generic 210 address 211 space 212 ``turks`` ``r600`` dGPU - Does not 213 support 214 generic 215 address 216 space 217 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 218 ----------------------------------------------------------------------------------------------------------------------- 219 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 220 support 221 generic 222 address 223 space 224 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 225 - ``verde`` support 226 generic 227 address 228 space 229 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 230 - ``oland`` support 231 generic 232 address 233 space 234 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 235 ----------------------------------------------------------------------------------------------------------------------- 236 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 237 flat - *pal-amdhsa* - A6 Pro-7050B 238 scratch - *pal-amdpal* - A8-7100 239 - A8 Pro-7150B 240 - A10-7300 241 - A10 Pro-7350B 242 - FX-7500 243 - A8-7200P 244 - A10-7400P 245 - FX-7600P 246 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 247 flat - *pal-amdhsa* - FirePro W9100 248 scratch - *pal-amdpal* - FirePro S9150 249 - FirePro S9170 250 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 251 flat - *pal-amdhsa* - Radeon R9 290x 252 scratch - *pal-amdpal* - Radeon R390 253 - Radeon R390x 254 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 255 - ``mullins`` flat - *pal-amdpal* - E1-2200 256 scratch - E1-2500 257 - E2-3000 258 - E2-3800 259 - A4-5000 260 - A4-5100 261 - A6-5200 262 - A4 Pro-3340B 263 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 264 flat - *pal-amdpal* - Radeon HD 8770 265 scratch - R7 260 266 - R7 260X 267 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 268 flat - *pal-amdpal* 269 scratch .. TODO:: 270 271 Add product 272 names. 273 274 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 275 ----------------------------------------------------------------------------------------------------------------------- 276 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 277 flat - *pal-amdhsa* - Pro A6-8500B 278 scratch - *pal-amdpal* - A8-8600P 279 - Pro A8-8600B 280 - FX-8800P 281 - Pro A12-8800B 282 - A10-8700P 283 - Pro A10-8700B 284 - A10-8780P 285 - A10-9600P 286 - A10-9630P 287 - A12-9700P 288 - A12-9730P 289 - FX-9800P 290 - FX-9830P 291 - E2-9010 292 - A6-9210 293 - A9-9410 294 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 295 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 296 scratch - *pal-amdpal* - Radeon R9 385 297 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 298 - *pal-amdhsa* - Radeon R9 Fury 299 - *pal-amdpal* - Radeon R9 FuryX 300 - Radeon Pro Duo 301 - FirePro S9300x2 302 - Radeon Instinct MI8 303 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 304 flat - *pal-amdhsa* - Radeon RX 480 305 scratch - *pal-amdpal* - Radeon Instinct MI6 306 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 307 flat - *pal-amdhsa* 308 scratch - *pal-amdpal* 309 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 310 flat - *pal-amdhsa* - FirePro S7100 311 scratch - *pal-amdpal* - FirePro W7100 312 - Mobile FirePro 313 M7170 314 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 315 flat - *pal-amdhsa* 316 scratch - *pal-amdpal* .. TODO:: 317 318 Add product 319 names. 320 321 **GCN GFX9 (Vega)** [AMD-GCN-GFX9]_ [AMD-GCN-GFX908-CDNA1]_ 322 ----------------------------------------------------------------------------------------------------------------------- 323 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 324 flat - *pal-amdhsa* Frontier Edition 325 scratch - *pal-amdpal* - Radeon RX Vega 56 326 - Radeon RX Vega 64 327 - Radeon RX Vega 64 328 Liquid 329 - Radeon Instinct MI25 330 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 331 flat - *pal-amdhsa* - Ryzen 5 2400G 332 scratch - *pal-amdpal* 333 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 334 - *pal-amdhsa* 335 - *pal-amdpal* .. TODO:: 336 337 Add product 338 names. 339 340 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 341 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 342 scratch - *pal-amdpal* - Radeon VII 343 - Radeon Pro VII 344 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 345 - xnack - Absolute 346 flat 347 scratch 348 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 349 flat 350 scratch .. TODO:: 351 352 Add product 353 names. 354 355 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 356 - tgsplit flat 357 - xnack scratch .. TODO:: 358 - Packed 359 work-item Add product 360 IDs names. 361 362 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 363 flat - Ryzen 7 4700GE 364 scratch - Ryzen 5 4600G 365 - Ryzen 5 4600GE 366 - Ryzen 3 4300G 367 - Ryzen 3 4300GE 368 - Ryzen Pro 4000G 369 - Ryzen 7 Pro 4700G 370 - Ryzen 7 Pro 4750GE 371 - Ryzen 5 Pro 4650G 372 - Ryzen 5 Pro 4650GE 373 - Ryzen 3 Pro 4350G 374 - Ryzen 3 Pro 4350GE 375 376 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 377 ----------------------------------------------------------------------------------------------------------------------- 378 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 379 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 380 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 381 - Radeon Pro 5600M 382 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 383 - wavefrontsize64 - Absolute - *pal-amdhsa* 384 - xnack flat - *pal-amdpal* 385 scratch 386 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 387 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 388 - xnack scratch - *pal-amdpal* 389 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 390 - wavefrontsize64 flat - *pal-amdhsa* 391 - xnack scratch - *pal-amdpal* .. TODO:: 392 393 Add product 394 names. 395 396 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 397 ----------------------------------------------------------------------------------------------------------------------- 398 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 399 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 400 scratch - *pal-amdpal* - Radeon RX 6900 XT 401 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 402 - wavefrontsize64 flat - *pal-amdhsa* 403 scratch - *pal-amdpal* 404 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 405 - wavefrontsize64 flat - *pal-amdhsa* 406 scratch - *pal-amdpal* .. TODO:: 407 408 Add product 409 names. 410 411 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 412 - wavefrontsize64 flat 413 scratch .. TODO:: 414 415 Add product 416 names. 417 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 418 - wavefrontsize64 flat 419 scratch .. TODO:: 420 421 Add product 422 names. 423 424 =========== =============== ============ ===== ================= =============== =============== ====================== 425 426.. _amdgpu-target-features: 427 428Target Features 429--------------- 430 431Target features control how code is generated to support certain 432processor specific features. Not all target features are supported by 433all processors. The runtime must ensure that the features supported by 434the device used to execute the code match the features enabled when 435generating the code. A mismatch of features may result in incorrect 436execution, or a reduction in performance. 437 438The target features supported by each processor is listed in 439:ref:`amdgpu-processor-table`. 440 441Target features are controlled by exactly one of the following Clang 442options: 443 444``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 445 446 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 447 optional components of the target ID. If omitted, the target feature has the 448 ``any`` value. See :ref:`amdgpu-target-id`. 449 450``-m[no-]<target-feature>`` 451 452 Target features not specified by the target ID are specified using a 453 separate option. These target features can have an ``on`` or ``off`` 454 value. ``on`` is specified by omitting the ``no-`` prefix, and 455 ``off`` is specified by including the ``no-`` prefix. The default 456 if not specified is ``off``. 457 458For example: 459 460``-mcpu=gfx908:xnack+`` 461 Enable the ``xnack`` feature. 462``-mcpu=gfx908:xnack-`` 463 Disable the ``xnack`` feature. 464``-mcumode`` 465 Enable the ``cumode`` feature. 466``-mno-cumode`` 467 Disable the ``cumode`` feature. 468 469 .. table:: AMDGPU Target Features 470 :name: amdgpu-target-features-table 471 472 =============== ============================ ================================================== 473 Target Feature Clang Option to Control Description 474 Name 475 =============== ============================ ================================================== 476 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 477 when generating code for kernels. When disabled 478 native WGP wavefront execution mode is used, 479 when enabled CU wavefront execution mode is used 480 (see :ref:`amdgpu-amdhsa-memory-model`). 481 482 sramecc - ``-mcpu`` If specified, generate code that can only be 483 - ``--offload-arch`` loaded and executed in a process that has a 484 matching setting for SRAMECC. 485 486 If not specified for code object V2 to V3, generate 487 code that can be loaded and executed in a process 488 with SRAMECC enabled. 489 490 If not specified for code object V4, generate 491 code that can be loaded and executed in a process 492 with either setting of SRAMECC. 493 494 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 495 work-groups are launched in threadgroup split mode. 496 When enabled the waves of a work-group may be 497 launched in different CUs. 498 499 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 500 generating code for kernels. When disabled 501 native wavefront size 32 is used, when enabled 502 wavefront size 64 is used. 503 504 xnack - ``-mcpu`` If specified, generate code that can only be 505 - ``--offload-arch`` loaded and executed in a process that has a 506 matching setting for XNACK replay. 507 508 If not specified for code object V2 to V3, generate 509 code that can be loaded and executed in a process 510 with XNACK replay enabled. 511 512 If not specified for code object V4, generate 513 code that can be loaded and executed in a process 514 with either setting of XNACK replay. 515 516 XNACK replay can be used for demand paging and 517 page migration. If enabled in the device, then if 518 a page fault occurs the code may execute 519 incorrectly unless generated with XNACK replay 520 enabled, or generated for code object V4 without 521 specifying XNACK replay. Executing code that was 522 generated with XNACK replay enabled, or generated 523 for code object V4 without specifying XNACK replay, 524 on a device that does not have XNACK replay 525 enabled will execute correctly but may be less 526 performant than code generated for XNACK replay 527 disabled. 528 =============== ============================ ================================================== 529 530.. _amdgpu-target-id: 531 532Target ID 533--------- 534 535AMDGPU supports target IDs. See `Clang Offload Bundler 536<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 537description. The AMDGPU target specific information is: 538 539**processor** 540 Is an AMDGPU processor or alternative processor name specified in 541 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 542 the primary processor and alternative processor names. The canonical form 543 target ID only allow the primary processor name. 544 545**target-feature** 546 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 547 is supported by the processor. The target features supported by each processor 548 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 549 a target ID are marked as being controlled by ``-mcpu`` and 550 ``--offload-arch``. Each target feature must appear at most once in a target 551 ID. The non-canonical form target ID allows the target features to be 552 specified in any order. The canonical form target ID requires the target 553 features to be specified in alphabetic order. 554 555.. _amdgpu-target-id-v2-v3: 556 557Code Object V2 to V3 Target ID 558~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 559 560The target ID syntax for code object V2 to V3 is the same as defined in `Clang 561Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 562when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 563directive and the bundle entry ID. In those cases it has the following BNF 564syntax: 565 566.. code:: 567 568 <target-id> ::== <processor> ( "+" <target-feature> )* 569 570Where a target feature is omitted if *Off* and present if *On* or *Any*. 571 572.. note:: 573 574 The code object V2 to V3 cannot represent *Any* and treats it the same as 575 *On*. 576 577.. _amdgpu-embedding-bundled-objects: 578 579Embedding Bundled Code Objects 580------------------------------ 581 582AMDGPU supports the HIP and OpenMP languages that perform code object embedding 583as described in `Clang Offload Bundler 584<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 585 586.. note:: 587 588 The target ID syntax used for code object V2 to V3 for a bundle entry ID 589 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 590 591.. _amdgpu-address-spaces: 592 593Address Spaces 594-------------- 595 596The AMDGPU architecture supports a number of memory address spaces. The address 597space names use the OpenCL standard names, with some additions. 598 599The AMDGPU address spaces correspond to target architecture specific LLVM 600address space numbers used in LLVM IR. 601 602The AMDGPU address spaces are described in 603:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 604supported for the ``amdgcn`` target. 605 606 .. table:: AMDGPU Address Spaces 607 :name: amdgpu-address-spaces-table 608 609 ================================= =============== =========== ================ ======= ============================ 610 .. 64-Bit Process Address Space 611 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 612 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 613 Space Number Name Name Size 614 ================================= =============== =========== ================ ======= ============================ 615 Generic 0 flat flat 64 0x0000000000000000 616 Global 1 global global 64 0x0000000000000000 617 Region 2 N/A GDS 32 *not implemented for AMDHSA* 618 Local 3 group LDS 32 0xFFFFFFFF 619 Constant 4 constant *same as global* 64 0x0000000000000000 620 Private 5 private scratch 32 0xFFFFFFFF 621 Constant 32-bit 6 *TODO* 0x00000000 622 Buffer Fat Pointer (experimental) 7 *TODO* 623 ================================= =============== =========== ================ ======= ============================ 624 625**Generic** 626 The generic address space is supported unless the *Target Properties* column 627 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 628 space*. 629 630 The generic address space uses the hardware flat address support for two fixed 631 ranges of virtual addresses (the private and local apertures), that are 632 outside the range of addressable global memory, to map from a flat address to 633 a private or local address. This uses FLAT instructions that can take a flat 634 address and access global, private (scratch), and group (LDS) memory depending 635 on if the address is within one of the aperture ranges. 636 637 Flat access to scratch requires hardware aperture setup and setup in the 638 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 639 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 640 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 641 642 To convert between a private or group address space address (termed a segment 643 address) and a flat address the base address of the corresponding aperture 644 can be used. For GFX7-GFX8 these are available in the 645 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 646 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 647 GFX9-GFX10 the aperture base addresses are directly available as inline 648 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 649 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 650 aligned to 2^32 which makes it easier to convert from flat to segment or 651 segment to flat. 652 653 A global address space address has the same value when used as a flat address 654 so no conversion is needed. 655 656**Global and Constant** 657 The global and constant address spaces both use global virtual addresses, 658 which are the same virtual address space used by the CPU. However, some 659 virtual addresses may only be accessible to the CPU, some only accessible 660 by the GPU, and some by both. 661 662 Using the constant address space indicates that the data will not change 663 during the execution of the kernel. This allows scalar read instructions to 664 be used. As the constant address space could only be modified on the host 665 side, a generic pointer loaded from the constant address space is safe to be 666 assumed as a global pointer since only the device global memory is visible 667 and managed on the host side. The vector and scalar L1 caches are invalidated 668 of volatile data before each kernel dispatch execution to allow constant 669 memory to change values between kernel dispatches. 670 671**Region** 672 The region address space uses the hardware Global Data Store (GDS). All 673 wavefronts executing on the same device will access the same memory for any 674 given region address. However, the same region address accessed by wavefronts 675 executing on different devices will access different memory. It is higher 676 performance than global memory. It is allocated by the runtime. The data 677 store (DS) instructions can be used to access it. 678 679**Local** 680 The local address space uses the hardware Local Data Store (LDS) which is 681 automatically allocated when the hardware creates the wavefronts of a 682 work-group, and freed when all the wavefronts of a work-group have 683 terminated. All wavefronts belonging to the same work-group will access the 684 same memory for any given local address. However, the same local address 685 accessed by wavefronts belonging to different work-groups will access 686 different memory. It is higher performance than global memory. The data store 687 (DS) instructions can be used to access it. 688 689**Private** 690 The private address space uses the hardware scratch memory support which 691 automatically allocates memory when it creates a wavefront and frees it when 692 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 693 given private address will be different to the memory accessed by another lane 694 of the same or different wavefront for the same private address. 695 696 If a kernel dispatch uses scratch, then the hardware allocates memory from a 697 pool of backing memory allocated by the runtime for each wavefront. The lanes 698 of the wavefront access this using dword (4 byte) interleaving. The mapping 699 used from private address to backing memory address is: 700 701 ``wavefront-scratch-base + 702 ((private-address / 4) * wavefront-size * 4) + 703 (wavefront-lane-id * 4) + (private-address % 4)`` 704 705 If each lane of a wavefront accesses the same private address, the 706 interleaving results in adjacent dwords being accessed and hence requires 707 fewer cache lines to be fetched. 708 709 There are different ways that the wavefront scratch base address is 710 determined by a wavefront (see 711 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 712 713 Scratch memory can be accessed in an interleaved manner using buffer 714 instructions with the scratch buffer descriptor and per wavefront scratch 715 offset, by the scratch instructions, or by flat instructions. Multi-dword 716 access is not supported except by flat and scratch instructions in 717 GFX9-GFX10. 718 719**Constant 32-bit** 720 *TODO* 721 722**Buffer Fat Pointer** 723 The buffer fat pointer is an experimental address space that is currently 724 unsupported in the backend. It exposes a non-integral pointer that is in 725 the future intended to support the modelling of 128-bit buffer descriptors 726 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 727 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 728 model the buffer descriptors used heavily in graphics workloads targeting 729 the backend. 730 731.. _amdgpu-memory-scopes: 732 733Memory Scopes 734------------- 735 736This section provides LLVM memory synchronization scopes supported by the AMDGPU 737backend memory model when the target triple OS is ``amdhsa`` (see 738:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 739 740The memory model supported is based on the HSA memory model [HSA]_ which is 741based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 742relation is transitive over the synchronizes-with relation independent of scope 743and synchronizes-with allows the memory scope instances to be inclusive (see 744table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 745 746This is different to the OpenCL [OpenCL]_ memory model which does not have scope 747inclusion and requires the memory scopes to exactly match. However, this 748is conservatively correct for OpenCL. 749 750 .. table:: AMDHSA LLVM Sync Scopes 751 :name: amdgpu-amdhsa-llvm-sync-scopes-table 752 753 ======================= =================================================== 754 LLVM Sync Scope Description 755 ======================= =================================================== 756 *none* The default: ``system``. 757 758 Synchronizes with, and participates in modification 759 and seq_cst total orderings with, other operations 760 (except image operations) for all address spaces 761 (except private, or generic that accesses private) 762 provided the other operation's sync scope is: 763 764 - ``system``. 765 - ``agent`` and executed by a thread on the same 766 agent. 767 - ``workgroup`` and executed by a thread in the 768 same work-group. 769 - ``wavefront`` and executed by a thread in the 770 same wavefront. 771 772 ``agent`` Synchronizes with, and participates in modification 773 and seq_cst total orderings with, other operations 774 (except image operations) for all address spaces 775 (except private, or generic that accesses private) 776 provided the other operation's sync scope is: 777 778 - ``system`` or ``agent`` and executed by a thread 779 on the same agent. 780 - ``workgroup`` and executed by a thread in the 781 same work-group. 782 - ``wavefront`` and executed by a thread in the 783 same wavefront. 784 785 ``workgroup`` Synchronizes with, and participates in modification 786 and seq_cst total orderings with, other operations 787 (except image operations) for all address spaces 788 (except private, or generic that accesses private) 789 provided the other operation's sync scope is: 790 791 - ``system``, ``agent`` or ``workgroup`` and 792 executed by a thread in the same work-group. 793 - ``wavefront`` and executed by a thread in the 794 same wavefront. 795 796 ``wavefront`` Synchronizes with, and participates in modification 797 and seq_cst total orderings with, other operations 798 (except image operations) for all address spaces 799 (except private, or generic that accesses private) 800 provided the other operation's sync scope is: 801 802 - ``system``, ``agent``, ``workgroup`` or 803 ``wavefront`` and executed by a thread in the 804 same wavefront. 805 806 ``singlethread`` Only synchronizes with and participates in 807 modification and seq_cst total orderings with, 808 other operations (except image operations) running 809 in the same thread for all address spaces (for 810 example, in signal handlers). 811 812 ``one-as`` Same as ``system`` but only synchronizes with other 813 operations within the same address space. 814 815 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 816 operations within the same address space. 817 818 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 819 other operations within the same address space. 820 821 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 822 other operations within the same address space. 823 824 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 825 other operations within the same address space. 826 ======================= =================================================== 827 828LLVM IR Intrinsics 829------------------ 830 831The AMDGPU backend implements the following LLVM IR intrinsics. 832 833*This section is WIP.* 834 835.. TODO:: 836 837 List AMDGPU intrinsics. 838 839LLVM IR Attributes 840------------------ 841 842The AMDGPU backend supports the following LLVM IR attributes. 843 844 .. table:: AMDGPU LLVM IR Attributes 845 :name: amdgpu-llvm-ir-attributes-table 846 847 ======================================= ========================================================== 848 LLVM Attribute Description 849 ======================================= ========================================================== 850 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 851 will be specified when the kernel is dispatched. Generated 852 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 853 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 854 argument block size for the implicit arguments. This 855 varies by OS and language (for OpenCL see 856 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 857 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 858 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 859 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 860 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 861 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 862 execution unit. Generated by the ``amdgpu_waves_per_eu`` 863 CLANG attribute [CLANG-ATTR]_. 864 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 865 mode register to be set on entry. Overrides the default for 866 the calling convention. 867 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 868 the mode register to be set on entry. Overrides the default 869 for the calling convention. 870 ======================================= ========================================================== 871 872.. _amdgpu-elf-code-object: 873 874ELF Code Object 875=============== 876 877The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 878can be linked by ``lld`` to produce a standard ELF shared code object which can 879be loaded and executed on an AMDGPU target. 880 881.. _amdgpu-elf-header: 882 883Header 884------ 885 886The AMDGPU backend uses the following ELF header: 887 888 .. table:: AMDGPU ELF Header 889 :name: amdgpu-elf-header-table 890 891 ========================== =============================== 892 Field Value 893 ========================== =============================== 894 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 895 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 896 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 897 - ``ELFOSABI_AMDGPU_HSA`` 898 - ``ELFOSABI_AMDGPU_PAL`` 899 - ``ELFOSABI_AMDGPU_MESA3D`` 900 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 901 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 902 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 903 - ``ELFABIVERSION_AMDGPU_PAL`` 904 - ``ELFABIVERSION_AMDGPU_MESA3D`` 905 ``e_type`` - ``ET_REL`` 906 - ``ET_DYN`` 907 ``e_machine`` ``EM_AMDGPU`` 908 ``e_entry`` 0 909 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 910 :ref:`amdgpu-elf-header-e_flags-table-v3`, 911 and :ref:`amdgpu-elf-header-e_flags-table-v4` 912 ========================== =============================== 913 914.. 915 916 .. table:: AMDGPU ELF Header Enumeration Values 917 :name: amdgpu-elf-header-enumeration-values-table 918 919 =============================== ===== 920 Name Value 921 =============================== ===== 922 ``EM_AMDGPU`` 224 923 ``ELFOSABI_NONE`` 0 924 ``ELFOSABI_AMDGPU_HSA`` 64 925 ``ELFOSABI_AMDGPU_PAL`` 65 926 ``ELFOSABI_AMDGPU_MESA3D`` 66 927 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 928 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 929 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 930 ``ELFABIVERSION_AMDGPU_PAL`` 0 931 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 932 =============================== ===== 933 934``e_ident[EI_CLASS]`` 935 The ELF class is: 936 937 * ``ELFCLASS32`` for ``r600`` architecture. 938 939 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 940 process address space applications. 941 942``e_ident[EI_DATA]`` 943 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 944 945``e_ident[EI_OSABI]`` 946 One of the following AMDGPU target architecture specific OS ABIs 947 (see :ref:`amdgpu-os`): 948 949 * ``ELFOSABI_NONE`` for *unknown* OS. 950 951 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 952 953 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 954 955 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 956 957``e_ident[EI_ABIVERSION]`` 958 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 959 object conforms: 960 961 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 962 runtime ABI for code object V2. Specify using the Clang option 963 ``-mcode-object-version=2``. 964 965 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 966 runtime ABI for code object V3. Specify using the Clang option 967 ``-mcode-object-version=3``. This is the default code object 968 version if not specified. 969 970 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 971 runtime ABI for code object V4. Specify using the Clang option 972 ``-mcode-object-version=4``. 973 974 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 975 runtime ABI. 976 977 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 978 3D runtime ABI. 979 980``e_type`` 981 Can be one of the following values: 982 983 984 ``ET_REL`` 985 The type produced by the AMDGPU backend compiler as it is relocatable code 986 object. 987 988 ``ET_DYN`` 989 The type produced by the linker as it is a shared code object. 990 991 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 992 993``e_machine`` 994 The value ``EM_AMDGPU`` is used for the machine for all processors supported 995 by the ``r600`` and ``amdgcn`` architectures (see 996 :ref:`amdgpu-processor-table`). The specific processor is specified in the 997 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 998 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 999 ``e_flags`` for code object V3 to V4 (see 1000 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1001 :ref:`amdgpu-elf-header-e_flags-table-v4`). 1002 1003``e_entry`` 1004 The entry point is 0 as the entry points for individual kernels must be 1005 selected in order to invoke them through AQL packets. 1006 1007``e_flags`` 1008 The AMDGPU backend uses the following ELF header flags: 1009 1010 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1011 :name: amdgpu-elf-header-e_flags-v2-table 1012 1013 ===================================== ===== ============================= 1014 Name Value Description 1015 ===================================== ===== ============================= 1016 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1017 target feature is 1018 enabled for all code 1019 contained in the code object. 1020 If the processor 1021 does not support the 1022 ``xnack`` target 1023 feature then must 1024 be 0. 1025 See 1026 :ref:`amdgpu-target-features`. 1027 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1028 handler is enabled for all 1029 code contained in the code 1030 object. If the processor 1031 does not support a trap 1032 handler then must be 0. 1033 See 1034 :ref:`amdgpu-target-features`. 1035 ===================================== ===== ============================= 1036 1037 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1038 :name: amdgpu-elf-header-e_flags-table-v3 1039 1040 ================================= ===== ============================= 1041 Name Value Description 1042 ================================= ===== ============================= 1043 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1044 mask for 1045 ``EF_AMDGPU_MACH_xxx`` values 1046 defined in 1047 :ref:`amdgpu-ef-amdgpu-mach-table`. 1048 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1049 target feature is 1050 enabled for all code 1051 contained in the code object. 1052 If the processor 1053 does not support the 1054 ``xnack`` target 1055 feature then must 1056 be 0. 1057 See 1058 :ref:`amdgpu-target-features`. 1059 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1060 target feature is 1061 enabled for all code 1062 contained in the code object. 1063 If the processor 1064 does not support the 1065 ``sramecc`` target 1066 feature then must 1067 be 0. 1068 See 1069 :ref:`amdgpu-target-features`. 1070 ================================= ===== ============================= 1071 1072 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1073 :name: amdgpu-elf-header-e_flags-table-v4 1074 1075 ============================================ ===== =================================== 1076 Name Value Description 1077 ============================================ ===== =================================== 1078 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1079 mask for 1080 ``EF_AMDGPU_MACH_xxx`` values 1081 defined in 1082 :ref:`amdgpu-ef-amdgpu-mach-table`. 1083 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1084 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1085 values. 1086 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1087 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1088 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1089 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1090 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1091 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1092 values. 1093 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1094 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1095 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1096 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1097 ============================================ ===== =================================== 1098 1099 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1100 :name: amdgpu-ef-amdgpu-mach-table 1101 1102 ==================================== ========== ============================= 1103 Name Value Description (see 1104 :ref:`amdgpu-processor-table`) 1105 ==================================== ========== ============================= 1106 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1107 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1108 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1109 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1110 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1111 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1112 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1113 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1114 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1115 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1116 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1117 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1118 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1119 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1120 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1121 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1122 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1123 *reserved* 0x011 - Reserved for ``r600`` 1124 0x01f architecture processors. 1125 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1126 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1127 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1128 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1129 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1130 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1131 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1132 *reserved* 0x027 Reserved. 1133 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1134 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1135 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1136 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1137 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1138 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1139 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1140 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1141 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1142 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1143 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1144 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1145 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1146 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1147 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1148 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1149 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1150 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1151 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1152 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1153 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1154 *reserved* 0x03d Reserved. 1155 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1156 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1157 *reserved* 0x040 Reserved. 1158 *reserved* 0x041 Reserved. 1159 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1160 ==================================== ========== ============================= 1161 1162Sections 1163-------- 1164 1165An AMDGPU target ELF code object has the standard ELF sections which include: 1166 1167 .. table:: AMDGPU ELF Sections 1168 :name: amdgpu-elf-sections-table 1169 1170 ================== ================ ================================= 1171 Name Type Attributes 1172 ================== ================ ================================= 1173 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1174 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1175 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1176 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1177 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1178 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1179 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1180 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1181 ``.note`` ``SHT_NOTE`` *none* 1182 ``.rela``\ *name* ``SHT_RELA`` *none* 1183 ``.rela.dyn`` ``SHT_RELA`` *none* 1184 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1185 ``.shstrtab`` ``SHT_STRTAB`` *none* 1186 ``.strtab`` ``SHT_STRTAB`` *none* 1187 ``.symtab`` ``SHT_SYMTAB`` *none* 1188 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1189 ================== ================ ================================= 1190 1191These sections have their standard meanings (see [ELF]_) and are only generated 1192if needed. 1193 1194``.debug``\ *\** 1195 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1196 information on the DWARF produced by the AMDGPU backend. 1197 1198``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1199 The standard sections used by a dynamic loader. 1200 1201``.note`` 1202 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1203 backend. 1204 1205``.rela``\ *name*, ``.rela.dyn`` 1206 For relocatable code objects, *name* is the name of the section that the 1207 relocation records apply. For example, ``.rela.text`` is the section name for 1208 relocation records associated with the ``.text`` section. 1209 1210 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1211 records from each of the relocatable code object's ``.rela``\ *name* sections. 1212 1213 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1214 the AMDGPU backend. 1215 1216``.text`` 1217 The executable machine code for the kernels and functions they call. Generated 1218 as position independent code. See :ref:`amdgpu-code-conventions` for 1219 information on conventions used in the isa generation. 1220 1221.. _amdgpu-note-records: 1222 1223Note Records 1224------------ 1225 1226The AMDGPU backend code object contains ELF note records in the ``.note`` 1227section. The set of generated notes and their semantics depend on the code 1228object version; see :ref:`amdgpu-note-records-v2` and 1229:ref:`amdgpu-note-records-v3-v4`. 1230 1231As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1232must be generated after the ``name`` field to ensure the ``desc`` field is 4 1233byte aligned. In addition, minimal zero-byte padding must be generated to 1234ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1235field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1236alignment. 1237 1238.. _amdgpu-note-records-v2: 1239 1240Code Object V2 Note Records 1241~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1242 1243.. warning:: 1244 Code object V2 is not the default code object version emitted by 1245 this version of LLVM. 1246 1247The AMDGPU backend code object uses the following ELF note record in the 1248``.note`` section when compiling for code object V2. 1249 1250The note record vendor field is "AMD". 1251 1252Additional note records may be present, but any which are not documented here 1253are deprecated and should not be used. 1254 1255 .. table:: AMDGPU Code Object V2 ELF Note Records 1256 :name: amdgpu-elf-note-records-v2-table 1257 1258 ===== ===================================== ====================================== 1259 Name Type Description 1260 ===== ===================================== ====================================== 1261 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1262 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1263 Finalizer and not the LLVM compiler. 1264 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1265 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1266 YAML [YAML]_ textual format. 1267 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1268 ===== ===================================== ====================================== 1269 1270.. 1271 1272 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1273 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1274 1275 ===================================== ===== 1276 Name Value 1277 ===================================== ===== 1278 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1279 ``NT_AMD_HSA_HSAIL`` 2 1280 ``NT_AMD_HSA_ISA_VERSION`` 3 1281 *reserved* 4-9 1282 ``NT_AMD_HSA_METADATA`` 10 1283 ``NT_AMD_HSA_ISA_NAME`` 11 1284 ===================================== ===== 1285 1286``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1287 Specifies the code object version number. The description field has the 1288 following layout: 1289 1290 .. code:: c 1291 1292 struct amdgpu_hsa_note_code_object_version_s { 1293 uint32_t major_version; 1294 uint32_t minor_version; 1295 }; 1296 1297 The ``major_version`` has a value less than or equal to 2. 1298 1299``NT_AMD_HSA_HSAIL`` 1300 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1301 field has the following layout: 1302 1303 .. code:: c 1304 1305 struct amdgpu_hsa_note_hsail_s { 1306 uint32_t hsail_major_version; 1307 uint32_t hsail_minor_version; 1308 uint8_t profile; 1309 uint8_t machine_model; 1310 uint8_t default_float_round; 1311 }; 1312 1313``NT_AMD_HSA_ISA_VERSION`` 1314 Specifies the target ISA version. The description field has the following layout: 1315 1316 .. code:: c 1317 1318 struct amdgpu_hsa_note_isa_s { 1319 uint16_t vendor_name_size; 1320 uint16_t architecture_name_size; 1321 uint32_t major; 1322 uint32_t minor; 1323 uint32_t stepping; 1324 char vendor_and_architecture_name[1]; 1325 }; 1326 1327 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1328 vendor and architecture names respectively, including the NUL character. 1329 1330 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1331 vendor, immediately followed by the NUL terminated string for the 1332 architecture. 1333 1334 This note record is used by the HSA runtime loader. 1335 1336 Code object V2 only supports a limited number of processors and has fixed 1337 settings for target features. See 1338 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1339 processors and the corresponding target ID. In the table the note record ISA 1340 name is a concatenation of the vendor name, architecture name, major, minor, 1341 and stepping separated by a ":". 1342 1343 The target ID column shows the processor name and fixed target features used 1344 by the LLVM compiler. The LLVM compiler does not generate a 1345 ``NT_AMD_HSA_HSAIL`` note record. 1346 1347 A code object generated by the Finalizer also uses code object V2 and always 1348 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1349 ``sramecc`` target feature is as shown in 1350 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1351 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1352 bit. 1353 1354``NT_AMD_HSA_ISA_NAME`` 1355 Specifies the target ISA name as a non-NUL terminated string. 1356 1357 This note record is not used by the HSA runtime loader. 1358 1359 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1360 V2's limited support of processors and fixed settings for target features. 1361 1362 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1363 from the string to the corresponding target ID. If the ``xnack`` target 1364 feature is supported and enabled, the string produced by the LLVM compiler 1365 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1366 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1367 1368``NT_AMD_HSA_METADATA`` 1369 Specifies extensible metadata associated with the code objects executed on HSA 1370 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1371 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1372 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1373 metadata string. 1374 1375 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1376 :name: amdgpu-elf-note-record-supported_processors-v2-table 1377 1378 ===================== ========================== 1379 Note Record ISA Name Target ID 1380 ===================== ========================== 1381 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1382 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1383 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1384 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1385 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1386 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1387 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1388 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1389 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1390 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1391 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1392 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1393 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1394 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1395 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1396 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1397 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1398 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1399 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1400 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1401 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1402 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1403 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1404 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1405 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1406 ===================== ========================== 1407 1408.. _amdgpu-note-records-v3-v4: 1409 1410Code Object V3 to V4 Note Records 1411~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1412 1413The AMDGPU backend code object uses the following ELF note record in the 1414``.note`` section when compiling for code object V3 to V4. 1415 1416The note record vendor field is "AMDGPU". 1417 1418Additional note records may be present, but any which are not documented here 1419are deprecated and should not be used. 1420 1421 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1422 :name: amdgpu-elf-note-records-table-v3-v4 1423 1424 ======== ============================== ====================================== 1425 Name Type Description 1426 ======== ============================== ====================================== 1427 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1428 binary format. 1429 ======== ============================== ====================================== 1430 1431.. 1432 1433 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1434 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1435 1436 ============================== ===== 1437 Name Value 1438 ============================== ===== 1439 *reserved* 0-31 1440 ``NT_AMDGPU_METADATA`` 32 1441 ============================== ===== 1442 1443``NT_AMDGPU_METADATA`` 1444 Specifies extensible metadata associated with an AMDGPU code object. It is 1445 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1446 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1447 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1448 ``amdhsa`` OS. 1449 1450.. _amdgpu-symbols: 1451 1452Symbols 1453------- 1454 1455Symbols include the following: 1456 1457 .. table:: AMDGPU ELF Symbols 1458 :name: amdgpu-elf-symbols-table 1459 1460 ===================== ================== ================ ================== 1461 Name Type Section Description 1462 ===================== ================== ================ ================== 1463 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1464 - ``.rodata`` 1465 - ``.bss`` 1466 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1467 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1468 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1469 ===================== ================== ================ ================== 1470 1471Global variable 1472 Global variables both used and defined by the compilation unit. 1473 1474 If the symbol is defined in the compilation unit then it is allocated in the 1475 appropriate section according to if it has initialized data or is readonly. 1476 1477 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1478 will resolve relocations using the definition provided by another code object 1479 or explicitly defined by the runtime. 1480 1481 If the symbol resides in local/group memory (LDS) then its section is the 1482 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1483 ``st_value`` field describes alignment requirements as it does for common 1484 symbols. 1485 1486 .. TODO:: 1487 1488 Add description of linked shared object symbols. Seems undefined symbols 1489 are marked as STT_NOTYPE. 1490 1491Kernel descriptor 1492 Every HSA kernel has an associated kernel descriptor. It is the address of the 1493 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1494 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1495 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1496 1497Kernel entry point 1498 Every HSA kernel also has a symbol for its machine code entry point. 1499 1500.. _amdgpu-relocation-records: 1501 1502Relocation Records 1503------------------ 1504 1505AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1506relocatable fields are: 1507 1508``word32`` 1509 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1510 alignment. These values use the same byte order as other word values in the 1511 AMDGPU architecture. 1512 1513``word64`` 1514 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1515 alignment. These values use the same byte order as other word values in the 1516 AMDGPU architecture. 1517 1518Following notations are used for specifying relocation calculations: 1519 1520**A** 1521 Represents the addend used to compute the value of the relocatable field. 1522 1523**G** 1524 Represents the offset into the global offset table at which the relocation 1525 entry's symbol will reside during execution. 1526 1527**GOT** 1528 Represents the address of the global offset table. 1529 1530**P** 1531 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1532 of the storage unit being relocated (computed using ``r_offset``). 1533 1534**S** 1535 Represents the value of the symbol whose index resides in the relocation 1536 entry. Relocations not using this must specify a symbol index of 1537 ``STN_UNDEF``. 1538 1539**B** 1540 Represents the base address of a loaded executable or shared object which is 1541 the difference between the ELF address and the actual load address. 1542 Relocations using this are only valid in executable or shared objects. 1543 1544The following relocation types are supported: 1545 1546 .. table:: AMDGPU ELF Relocation Records 1547 :name: amdgpu-elf-relocation-records-table 1548 1549 ========================== ======= ===== ========== ============================== 1550 Relocation Type Kind Value Field Calculation 1551 ========================== ======= ===== ========== ============================== 1552 ``R_AMDGPU_NONE`` 0 *none* *none* 1553 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1554 Dynamic 1555 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1556 Dynamic 1557 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1558 Dynamic 1559 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1560 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1561 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1562 Dynamic 1563 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1564 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1565 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1566 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1567 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1568 *reserved* 12 1569 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1570 ========================== ======= ===== ========== ============================== 1571 1572``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1573the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1574 1575There is no current OS loader support for 32-bit programs and so 1576``R_AMDGPU_ABS32`` is not used. 1577 1578.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1579 1580Loaded Code Object Path Uniform Resource Identifier (URI) 1581--------------------------------------------------------- 1582 1583The AMD GPU code object loader represents the path of the ELF shared object from 1584which the code object was loaded as a textual Unifom Resource Identifier (URI). 1585Note that the code object is the in memory loaded relocated form of the ELF 1586shared object. Multiple code objects may be loaded at different memory 1587addresses in the same process from the same ELF shared object. 1588 1589The loaded code object path URI syntax is defined by the following BNF syntax: 1590 1591.. code:: 1592 1593 code_object_uri ::== file_uri | memory_uri 1594 file_uri ::== "file://" file_path [ range_specifier ] 1595 memory_uri ::== "memory://" process_id range_specifier 1596 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1597 file_path ::== URI_ENCODED_OS_FILE_PATH 1598 process_id ::== DECIMAL_NUMBER 1599 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1600 1601**number** 1602 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1603 and octal values by "0". 1604 1605**file_path** 1606 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1607 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1608 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1609 the path are separated by "/". 1610 1611**offset** 1612 Is a 0-based byte offset to the start of the code object. For a file URI, it 1613 is from the start of the file specified by the ``file_path``, and if omitted 1614 defaults to 0. For a memory URI, it is the memory address and is required. 1615 1616**size** 1617 Is the number of bytes in the code object. For a file URI, if omitted it 1618 defaults to the size of the file. It is required for a memory URI. 1619 1620**process_id** 1621 Is the identity of the process owning the memory. For Linux it is the C 1622 unsigned integral decimal literal for the process ID (PID). 1623 1624For example: 1625 1626.. code:: 1627 1628 file:///dir1/dir2/file1 1629 file:///dir3/dir4/file2#offset=0x2000&size=3000 1630 memory://1234#offset=0x20000&size=3000 1631 1632.. _amdgpu-dwarf-debug-information: 1633 1634DWARF Debug Information 1635======================= 1636 1637.. warning:: 1638 1639 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1640 is not currently fully implemented and is subject to change. 1641 1642AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1643:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1644object executable code and data to the source language constructs. It can be 1645used by tools such as debuggers and profilers. It uses features defined in 1646:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1647DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1648 1649This section defines the AMDGPU target architecture specific DWARF mappings. 1650 1651.. _amdgpu-dwarf-register-identifier: 1652 1653Register Identifier 1654------------------- 1655 1656This section defines the AMDGPU target architecture register numbers used in 1657DWARF operation expressions (see DWARF Version 5 section 2.5 and 1658:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1659instructions (see DWARF Version 5 section 6.4 and 1660:ref:`amdgpu-dwarf-call-frame-information`). 1661 1662A single code object can contain code for kernels that have different wavefront 1663sizes. The vector registers and some scalar registers are based on the wavefront 1664size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1665simplifies the consumer of the DWARF so that each register has a fixed size, 1666rather than being dynamic according to the wavefront size mode. Similarly, 1667distinct DWARF registers are defined for those registers that vary in size 1668according to the process address size. This allows a consumer to treat a 1669specific AMDGPU processor as a single architecture regardless of how it is 1670configured at run time. The compiler explicitly specifies the DWARF registers 1671that match the mode in which the code it is generating will be executed. 1672 1673DWARF registers are encoded as numbers, which are mapped to architecture 1674registers. The mapping for AMDGPU is defined in 1675:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1676mapping. 1677 1678.. table:: AMDGPU DWARF Register Mapping 1679 :name: amdgpu-dwarf-register-mapping-table 1680 1681 ============== ================= ======== ================================== 1682 DWARF Register AMDGPU Register Bit Size Description 1683 ============== ================= ======== ================================== 1684 0 PC_32 32 Program Counter (PC) when 1685 executing in a 32-bit process 1686 address space. Used in the CFI to 1687 describe the PC of the calling 1688 frame. 1689 1 EXEC_MASK_32 32 Execution Mask Register when 1690 executing in wavefront 32 mode. 1691 2-15 *Reserved* *Reserved for highly accessed 1692 registers using DWARF shortcut.* 1693 16 PC_64 64 Program Counter (PC) when 1694 executing in a 64-bit process 1695 address space. Used in the CFI to 1696 describe the PC of the calling 1697 frame. 1698 17 EXEC_MASK_64 64 Execution Mask Register when 1699 executing in wavefront 64 mode. 1700 18-31 *Reserved* *Reserved for highly accessed 1701 registers using DWARF shortcut.* 1702 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1703 Registers. 1704 96-127 *Reserved* *Reserved for frequently accessed 1705 registers using DWARF 1-byte ULEB.* 1706 128 STATUS 32 Status Register. 1707 129-511 *Reserved* *Reserved for future Scalar 1708 Architectural Registers.* 1709 512 VCC_32 32 Vector Condition Code Register 1710 when executing in wavefront 32 1711 mode. 1712 513-1023 *Reserved* *Reserved for future Vector 1713 Architectural Registers when 1714 executing in wavefront 32 mode.* 1715 768 VCC_64 64 Vector Condition Code Register 1716 when executing in wavefront 64 1717 mode. 1718 769-1023 *Reserved* *Reserved for future Vector 1719 Architectural Registers when 1720 executing in wavefront 64 mode.* 1721 1024-1087 *Reserved* *Reserved for padding.* 1722 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1723 1130-1535 *Reserved* *Reserved for future Scalar 1724 General Purpose Registers.* 1725 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1726 when executing in wavefront 32 1727 mode. 1728 1792-2047 *Reserved* *Reserved for future Vector 1729 General Purpose Registers when 1730 executing in wavefront 32 mode.* 1731 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1732 when executing in wavefront 32 1733 mode. 1734 2304-2559 *Reserved* *Reserved for future Vector 1735 Accumulation Registers when 1736 executing in wavefront 32 mode.* 1737 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1738 when executing in wavefront 64 1739 mode. 1740 2816-3071 *Reserved* *Reserved for future Vector 1741 General Purpose Registers when 1742 executing in wavefront 64 mode.* 1743 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1744 when executing in wavefront 64 1745 mode. 1746 3328-3583 *Reserved* *Reserved for future Vector 1747 Accumulation Registers when 1748 executing in wavefront 64 mode.* 1749 ============== ================= ======== ================================== 1750 1751The vector registers are represented as the full size for the wavefront. They 1752are organized as consecutive dwords (32-bits), one per lane, with the dword at 1753the least significant bit position corresponding to lane 0 and so forth. DWARF 1754location expressions involving the ``DW_OP_LLVM_offset`` and 1755``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1756register corresponding to the lane that is executing the current thread of 1757execution in languages that are implemented using a SIMD or SIMT execution 1758model. 1759 1760If the wavefront size is 32 lanes then the wavefront 32 mode register 1761definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1762mode register definitions are used. Some AMDGPU targets support executing in 1763both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1764to the wavefront mode of the generated code will be used. 1765 1766If code is generated to execute in a 32-bit process address space, then the 176732-bit process address space register definitions are used. If code is generated 1768to execute in a 64-bit process address space, then the 64-bit process address 1769space register definitions are used. The ``amdgcn`` target only supports the 177064-bit process address space. 1771 1772.. _amdgpu-dwarf-address-class-identifier: 1773 1774Address Class Identifier 1775------------------------ 1776 1777The DWARF address class represents the source language memory space. See DWARF 1778Version 5 section 2.12 which is updated by the *DWARF Extensions For 1779Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1780 1781The DWARF address class mapping used for AMDGPU is defined in 1782:ref:`amdgpu-dwarf-address-class-mapping-table`. 1783 1784.. table:: AMDGPU DWARF Address Class Mapping 1785 :name: amdgpu-dwarf-address-class-mapping-table 1786 1787 ========================= ====== ================= 1788 DWARF AMDGPU 1789 -------------------------------- ----------------- 1790 Address Class Name Value Address Space 1791 ========================= ====== ================= 1792 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1793 ``DW_ADDR_LLVM_global`` 0x0001 Global 1794 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1795 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1796 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1797 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1798 ========================= ====== ================= 1799 1800The DWARF address class values defined in the *DWARF Extensions For 1801Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1802 1803In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1804available for use for the AMD extension for access to the hardware GDS memory 1805which is scratchpad memory allocated per device. 1806 1807For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1808address class of ``DW_ADDR_none`` is used. 1809 1810See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1811mapping of DWARF address classes to DWARF address spaces, including address size 1812and NULL value. 1813 1814.. _amdgpu-dwarf-address-space-identifier: 1815 1816Address Space Identifier 1817------------------------ 1818 1819DWARF address spaces correspond to target architecture specific linear 1820addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1821For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1822 1823The DWARF address space mapping used for AMDGPU is defined in 1824:ref:`amdgpu-dwarf-address-space-mapping-table`. 1825 1826.. table:: AMDGPU DWARF Address Space Mapping 1827 :name: amdgpu-dwarf-address-space-mapping-table 1828 1829 ======================================= ===== ======= ======== ================= ======================= 1830 DWARF AMDGPU Notes 1831 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1832 Address Space Name Value Address Bit Size Address Space 1833 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1834 .. 64-bit 32-bit 1835 process process 1836 address address 1837 space space 1838 ======================================= ===== ======= ======== ================= ======================= 1839 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1840 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1841 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1842 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1843 *Reserved* 0x04 1844 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1845 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1846 ======================================= ===== ======= ======== ================= ======================= 1847 1848See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1849including address size and NULL value. 1850 1851The ``DW_ASPACE_none`` address space is the default target architecture address 1852space used in DWARF operations that do not specify an address space. It 1853therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1854related operations can refer to addresses in the program code. 1855 1856The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1857specify the flat address space. If the address corresponds to an address in the 1858local address space, then it corresponds to the wavefront that is executing the 1859focused thread of execution. If the address corresponds to an address in the 1860private address space, then it corresponds to the lane that is executing the 1861focused thread of execution for languages that are implemented using a SIMD or 1862SIMT execution model. 1863 1864.. note:: 1865 1866 CUDA-like languages such as HIP that do not have address spaces in the 1867 language type system, but do allow variables to be allocated in different 1868 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1869 address space in the DWARF expression operations as the default address space 1870 is the global address space. 1871 1872The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1873specify the local address space corresponding to the wavefront that is executing 1874the focused thread of execution. 1875 1876The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1877to specify the private address space corresponding to the lane that is executing 1878the focused thread of execution for languages that are implemented using a SIMD 1879or SIMT execution model. 1880 1881The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1882to specify the unswizzled private address space corresponding to the wavefront 1883that is executing the focused thread of execution. The wavefront view of private 1884memory is the per wavefront unswizzled backing memory layout defined in 1885:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1886location for the backing memory of the wavefront (namely the address is not 1887offset by ``wavefront-scratch-base``). The following formula can be used to 1888convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1889``DW_ASPACE_AMDGPU_private_wave`` address: 1890 1891:: 1892 1893 private-address-wavefront = 1894 ((private-address-lane / 4) * wavefront-size * 4) + 1895 (wavefront-lane-id * 4) + (private-address-lane % 4) 1896 1897If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1898of the dwords for each lane starting with lane 0 is required, then this 1899simplifies to: 1900 1901:: 1902 1903 private-address-wavefront = 1904 private-address-lane * wavefront-size 1905 1906A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1907complete spilled vector register back into a complete vector register in the 1908CFI. The frame pointer can be a private lane address which is dword aligned, 1909which can be shifted to multiply by the wavefront size, and then used to form a 1910private wavefront address that gives a location for a contiguous set of dwords, 1911one per lane, where the vector register dwords are spilled. The compiler knows 1912the wavefront size since it generates the code. Note that the type of the 1913address may have to be converted as the size of a 1914``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1915``DW_ASPACE_AMDGPU_private_wave`` address. 1916 1917.. _amdgpu-dwarf-lane-identifier: 1918 1919Lane identifier 1920--------------- 1921 1922DWARF lane identifies specify a target architecture lane position for hardware 1923that executes in a SIMD or SIMT manner, and on which a source language maps its 1924threads of execution onto those lanes. The DWARF lane identifier is pushed by 1925the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1926section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1927section :ref:`amdgpu-dwarf-operation-expressions`. 1928 1929For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1930wavefront. It is numbered from 0 to the wavefront size minus 1. 1931 1932Operation Expressions 1933--------------------- 1934 1935DWARF expressions are used to compute program values and the locations of 1936program objects. See DWARF Version 5 section 2.5 and 1937:ref:`amdgpu-dwarf-operation-expressions`. 1938 1939DWARF location descriptions describe how to access storage which includes memory 1940and registers. When accessing storage on AMDGPU, bytes are ordered with least 1941significant bytes first, and bits are ordered within bytes with least 1942significant bits first. 1943 1944For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1945unwinding vector registers that are spilled under the execution mask to memory: 1946the zero-single location description is the vector register, and the one-single 1947location description is the spilled memory location description. The 1948``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1949memory location description. 1950 1951In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1952``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1953controlled by the execution mask. An undefined location description together 1954with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1955to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1956 1957Debugger Information Entry Attributes 1958------------------------------------- 1959 1960This section describes how certain debugger information entry attributes are 1961used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1962by *DWARF Extensions For Heterogeneous Debugging* section 1963:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1964 1965.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1966 1967``DW_AT_LLVM_lane_pc`` 1968~~~~~~~~~~~~~~~~~~~~~~ 1969 1970For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1971location of the separate lanes of a SIMT thread. 1972 1973If the lane is an active lane then this will be the same as the current program 1974location. 1975 1976If the lane is inactive, but was active on entry to the subprogram, then this is 1977the program location in the subprogram at which execution of the lane is 1978conceptual positioned. 1979 1980If the lane was not active on entry to the subprogram, then this will be the 1981undefined location. A client debugger can check if the lane is part of a valid 1982work-group by checking that the lane is in the range of the associated 1983work-group within the grid, accounting for partial work-groups. If it is not, 1984then the debugger can omit any information for the lane. Otherwise, the debugger 1985may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1986calling subprogram until it finds a non-undefined location. Conceptually the 1987lane only has the call frames that it has a non-undefined 1988``DW_AT_LLVM_lane_pc``. 1989 1990The following example illustrates how the AMDGPU backend can generate a DWARF 1991location list expression for the nested ``IF/THEN/ELSE`` structures of the 1992following subprogram pseudo code for a target with 64 lanes per wavefront. 1993 1994.. code:: 1995 :number-lines: 1996 1997 SUBPROGRAM X 1998 BEGIN 1999 a; 2000 IF (c1) THEN 2001 b; 2002 IF (c2) THEN 2003 c; 2004 ELSE 2005 d; 2006 ENDIF 2007 e; 2008 ELSE 2009 f; 2010 ENDIF 2011 g; 2012 END 2013 2014The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2015execution mask (``EXEC``) to linearize the control flow. The condition is 2016evaluated to make a mask of the lanes for which the condition evaluates to true. 2017First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2018logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2019``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2020the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2021region the ``EXEC`` mask is restored to the value it had at the beginning of the 2022region. This is shown below. Other approaches are possible, but the basic 2023concept is the same. 2024 2025.. code:: 2026 :number-lines: 2027 2028 $lex_start: 2029 a; 2030 %1 = EXEC 2031 %2 = c1 2032 $lex_1_start: 2033 EXEC = %1 & %2 2034 $if_1_then: 2035 b; 2036 %3 = EXEC 2037 %4 = c2 2038 $lex_1_1_start: 2039 EXEC = %3 & %4 2040 $lex_1_1_then: 2041 c; 2042 EXEC = ~EXEC & %3 2043 $lex_1_1_else: 2044 d; 2045 EXEC = %3 2046 $lex_1_1_end: 2047 e; 2048 EXEC = ~EXEC & %1 2049 $lex_1_else: 2050 f; 2051 EXEC = %1 2052 $lex_1_end: 2053 g; 2054 $lex_end: 2055 2056To create the DWARF location list expression that defines the location 2057description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2058pseudo instruction can be used to annotate the linearized control flow. This can 2059be done by defining an artificial variable for the lane PC. The DWARF location 2060list expression created for it is used as the value of the 2061``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2062 2063A DWARF procedure is defined for each well nested structured control flow region 2064which provides the conceptual lane program location for a lane if it is not 2065active (namely it is divergent). The DWARF operation expression for each region 2066conceptually inherits the value of the immediately enclosing region and modifies 2067it according to the semantics of the region. 2068 2069For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2070the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2071region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2072region since the ``THEN`` region has completed. 2073 2074The lane PC artificial variable is assigned at each region transition. It uses 2075the immediately enclosing region's DWARF procedure to compute the program 2076location for each lane assuming they are divergent, and then modifies the result 2077by inserting the current program location for each lane that the ``EXEC`` mask 2078indicates is active. 2079 2080By having separate DWARF procedures for each region, they can be reused to 2081define the value for any nested region. This reduces the total size of the DWARF 2082operation expressions. 2083 2084The following provides an example using pseudo LLVM MIR. 2085 2086.. code:: 2087 :number-lines: 2088 2089 $lex_start: 2090 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2091 DW_AT_name = "__uint64"; 2092 DW_AT_byte_size = 8; 2093 DW_AT_encoding = DW_ATE_unsigned; 2094 ]; 2095 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2096 DW_AT_name = "__active_lane_pc"; 2097 DW_AT_location = [ 2098 DW_OP_regx PC; 2099 DW_OP_LLVM_extend 64, 64; 2100 DW_OP_regval_type EXEC, %uint_64; 2101 DW_OP_LLVM_select_bit_piece 64, 64; 2102 ]; 2103 ]; 2104 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2105 DW_AT_name = "__divergent_lane_pc"; 2106 DW_AT_location = [ 2107 DW_OP_LLVM_undefined; 2108 DW_OP_LLVM_extend 64, 64; 2109 ]; 2110 ]; 2111 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2112 DW_OP_call_ref %__divergent_lane_pc; 2113 DW_OP_call_ref %__active_lane_pc; 2114 ]; 2115 a; 2116 %1 = EXEC; 2117 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2118 %2 = c1; 2119 $lex_1_start: 2120 EXEC = %1 & %2; 2121 $lex_1_then: 2122 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2123 DW_AT_name = "__divergent_lane_pc_1_then"; 2124 DW_AT_location = DIExpression[ 2125 DW_OP_call_ref %__divergent_lane_pc; 2126 DW_OP_addrx &lex_1_start; 2127 DW_OP_stack_value; 2128 DW_OP_LLVM_extend 64, 64; 2129 DW_OP_call_ref %__lex_1_save_exec; 2130 DW_OP_deref_type 64, %__uint_64; 2131 DW_OP_LLVM_select_bit_piece 64, 64; 2132 ]; 2133 ]; 2134 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2135 DW_OP_call_ref %__divergent_lane_pc_1_then; 2136 DW_OP_call_ref %__active_lane_pc; 2137 ]; 2138 b; 2139 %3 = EXEC; 2140 DBG_VALUE %3, %__lex_1_1_save_exec; 2141 %4 = c2; 2142 $lex_1_1_start: 2143 EXEC = %3 & %4; 2144 $lex_1_1_then: 2145 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2146 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2147 DW_AT_location = DIExpression[ 2148 DW_OP_call_ref %__divergent_lane_pc_1_then; 2149 DW_OP_addrx &lex_1_1_start; 2150 DW_OP_stack_value; 2151 DW_OP_LLVM_extend 64, 64; 2152 DW_OP_call_ref %__lex_1_1_save_exec; 2153 DW_OP_deref_type 64, %__uint_64; 2154 DW_OP_LLVM_select_bit_piece 64, 64; 2155 ]; 2156 ]; 2157 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2158 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2159 DW_OP_call_ref %__active_lane_pc; 2160 ]; 2161 c; 2162 EXEC = ~EXEC & %3; 2163 $lex_1_1_else: 2164 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2165 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2166 DW_AT_location = DIExpression[ 2167 DW_OP_call_ref %__divergent_lane_pc_1_then; 2168 DW_OP_addrx &lex_1_1_end; 2169 DW_OP_stack_value; 2170 DW_OP_LLVM_extend 64, 64; 2171 DW_OP_call_ref %__lex_1_1_save_exec; 2172 DW_OP_deref_type 64, %__uint_64; 2173 DW_OP_LLVM_select_bit_piece 64, 64; 2174 ]; 2175 ]; 2176 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2177 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2178 DW_OP_call_ref %__active_lane_pc; 2179 ]; 2180 d; 2181 EXEC = %3; 2182 $lex_1_1_end: 2183 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2184 DW_OP_call_ref %__divergent_lane_pc; 2185 DW_OP_call_ref %__active_lane_pc; 2186 ]; 2187 e; 2188 EXEC = ~EXEC & %1; 2189 $lex_1_else: 2190 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2191 DW_AT_name = "__divergent_lane_pc_1_else"; 2192 DW_AT_location = DIExpression[ 2193 DW_OP_call_ref %__divergent_lane_pc; 2194 DW_OP_addrx &lex_1_end; 2195 DW_OP_stack_value; 2196 DW_OP_LLVM_extend 64, 64; 2197 DW_OP_call_ref %__lex_1_save_exec; 2198 DW_OP_deref_type 64, %__uint_64; 2199 DW_OP_LLVM_select_bit_piece 64, 64; 2200 ]; 2201 ]; 2202 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2203 DW_OP_call_ref %__divergent_lane_pc_1_else; 2204 DW_OP_call_ref %__active_lane_pc; 2205 ]; 2206 f; 2207 EXEC = %1; 2208 $lex_1_end: 2209 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2210 DW_OP_call_ref %__divergent_lane_pc; 2211 DW_OP_call_ref %__active_lane_pc; 2212 ]; 2213 g; 2214 $lex_end: 2215 2216The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2217that are active, with the current program location. 2218 2219Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2220the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2221instruction, location list entries will be created that describe where the 2222artificial variables are allocated at any given program location. The compiler 2223may allocate them to registers or spill them to memory. 2224 2225The DWARF procedures for each region use the values of the saved execution mask 2226artificial variables to only update the lanes that are active on entry to the 2227region. All other lanes retain the value of the enclosing region where they were 2228last active. If they were not active on entry to the subprogram, then will have 2229the undefined location description. 2230 2231Other structured control flow regions can be handled similarly. For example, 2232loops would set the divergent program location for the region at the end of the 2233loop. Any lanes active will be in the loop, and any lanes not active must have 2234exited the loop. 2235 2236An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2237``IF/THEN/ELSE`` regions. 2238 2239The DWARF procedures can use the active lane artificial variable described in 2240:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2241``EXEC`` mask in order to support whole or quad wavefront mode. 2242 2243.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2244 2245``DW_AT_LLVM_active_lane`` 2246~~~~~~~~~~~~~~~~~~~~~~~~~~ 2247 2248The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2249entry is used to specify the lanes that are conceptually active for a SIMT 2250thread. 2251 2252The execution mask may be modified to implement whole or quad wavefront mode 2253operations. For example, all lanes may need to temporarily be made active to 2254execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2255update it to enable the necessary lanes, perform the operations, and then 2256restore the ``EXEC`` mask from the saved value. While executing the whole 2257wavefront region, the conceptual execution mask is the saved value, not the 2258``EXEC`` value. 2259 2260This is handled by defining an artificial variable for the active lane mask. The 2261active lane mask artificial variable would be the actual ``EXEC`` mask for 2262normal regions, and the saved execution mask for regions where the mask is 2263temporarily updated. The location list expression created for this artificial 2264variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2265attribute. 2266 2267``DW_AT_LLVM_augmentation`` 2268~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2269 2270For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2271debugger information entry has the following value for the augmentation string: 2272 2273:: 2274 2275 [amdgpu:v0.0] 2276 2277The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2278extensions used in the DWARF of the compilation unit. The version number 2279conforms to [SEMVER]_. 2280 2281Call Frame Information 2282---------------------- 2283 2284DWARF Call Frame Information (CFI) describes how a consumer can virtually 2285*unwind* call frames in a running process or core dump. See DWARF Version 5 2286section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2287 2288For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2289 22901. ``augmentation`` string contains the following null-terminated UTF-8 string: 2291 2292 :: 2293 2294 [amd:v0.0] 2295 2296 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2297 extensions used in this CIE or to the FDEs that use it. The version number 2298 conforms to [SEMVER]_. 2299 23002. ``address_size`` for the ``Global`` address space is defined in 2301 :ref:`amdgpu-dwarf-address-space-identifier`. 2302 23033. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2304 23054. ``code_alignment_factor`` is 4 bytes. 2306 2307 .. TODO:: 2308 2309 Add to :ref:`amdgpu-processor-table` table. 2310 23115. ``data_alignment_factor`` is 4 bytes. 2312 2313 .. TODO:: 2314 2315 Add to :ref:`amdgpu-processor-table` table. 2316 23176. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2318 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2319 23207. ``initial_instructions`` Since a subprogram X with fewer registers can be 2321 called from subprogram Y that has more allocated, X will not change any of 2322 the extra registers as it cannot access them. Therefore, the default rule 2323 for all columns is ``same value``. 2324 2325For AMDGPU the register number follows the numbering defined in 2326:ref:`amdgpu-dwarf-register-identifier`. 2327 2328For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2329the return address to get the address of a byte within the call site 2330instructions. See DWARF Version 5 section 6.4.4. 2331 2332Accelerated Access 2333------------------ 2334 2335See DWARF Version 5 section 6.1. 2336 2337Lookup By Name Section Header 2338~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2339 2340See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2341 2342For AMDGPU the lookup by name section header table: 2343 2344``augmentation_string_size`` (uword) 2345 2346 Set to the length of the ``augmentation_string`` value which is always a 2347 multiple of 4. 2348 2349``augmentation_string`` (sequence of UTF-8 characters) 2350 2351 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2352 2353 :: 2354 2355 [amdgpu:v0.0] 2356 2357 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2358 extensions used in the DWARF of this index. The version number conforms to 2359 [SEMVER]_. 2360 2361 .. note:: 2362 2363 This is different to the DWARF Version 5 definition that requires the first 2364 4 characters to be the vendor ID. But this is consistent with the other 2365 augmentation strings and does allow multiple vendor contributions. However, 2366 backwards compatibility may be more desirable. 2367 2368Lookup By Address Section Header 2369~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2370 2371See DWARF Version 5 section 6.1.2. 2372 2373For AMDGPU the lookup by address section header table: 2374 2375``address_size`` (ubyte) 2376 2377 Match the address size for the ``Global`` address space defined in 2378 :ref:`amdgpu-dwarf-address-space-identifier`. 2379 2380``segment_selector_size`` (ubyte) 2381 2382 AMDGPU does not use a segment selector so this is 0. The entries in the 2383 ``.debug_aranges`` do not have a segment selector. 2384 2385Line Number Information 2386----------------------- 2387 2388See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2389 2390AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2391The instruction set must be obtained from the ELF file header ``e_flags`` field 2392in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2393<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2394 2395.. TODO:: 2396 2397 Should the ``isa`` state machine register be used to indicate if the code is 2398 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2399 2400For AMDGPU the line number program header fields have the following values (see 2401DWARF Version 5 section 6.2.4): 2402 2403``address_size`` (ubyte) 2404 Matches the address size for the ``Global`` address space defined in 2405 :ref:`amdgpu-dwarf-address-space-identifier`. 2406 2407``segment_selector_size`` (ubyte) 2408 AMDGPU does not use a segment selector so this is 0. 2409 2410``minimum_instruction_length`` (ubyte) 2411 For GFX9-GFX10 this is 4. 2412 2413``maximum_operations_per_instruction`` (ubyte) 2414 For GFX9-GFX10 this is 1. 2415 2416Source text for online-compiled programs (for example, those compiled by the 2417OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2418See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2419Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2420<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2421 2422The Clang option used to control source embedding in AMDGPU is defined in 2423:ref:`amdgpu-clang-debug-options-table`. 2424 2425 .. table:: AMDGPU Clang Debug Options 2426 :name: amdgpu-clang-debug-options-table 2427 2428 ==================== ================================================== 2429 Debug Flag Description 2430 ==================== ================================================== 2431 -g[no-]embed-source Enable/disable embedding source text in DWARF 2432 debug sections. Useful for environments where 2433 source cannot be written to disk, such as 2434 when performing online compilation. 2435 ==================== ================================================== 2436 2437For example: 2438 2439``-gembed-source`` 2440 Enable the embedded source. 2441 2442``-gno-embed-source`` 2443 Disable the embedded source. 2444 244532-Bit and 64-Bit DWARF Formats 2446------------------------------- 2447 2448See DWARF Version 5 section 7.4 and 2449:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2450 2451For AMDGPU: 2452 2453* For the ``amdgcn`` target architecture only the 64-bit process address space 2454 is supported. 2455 2456* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2457 the 32-bit DWARF format. 2458 2459Unit Headers 2460------------ 2461 2462For AMDGPU the following values apply for each of the unit headers described in 2463DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2464 2465``address_size`` (ubyte) 2466 Matches the address size for the ``Global`` address space defined in 2467 :ref:`amdgpu-dwarf-address-space-identifier`. 2468 2469.. _amdgpu-code-conventions: 2470 2471Code Conventions 2472================ 2473 2474This section provides code conventions used for each supported target triple OS 2475(see :ref:`amdgpu-target-triples`). 2476 2477AMDHSA 2478------ 2479 2480This section provides code conventions used when the target triple OS is 2481``amdhsa`` (see :ref:`amdgpu-target-triples`). 2482 2483.. _amdgpu-amdhsa-code-object-metadata: 2484 2485Code Object Metadata 2486~~~~~~~~~~~~~~~~~~~~ 2487 2488The code object metadata specifies extensible metadata associated with the code 2489objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2490encoding and semantics of this metadata depends on the code object version; see 2491:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2492:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2493:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2494 2495Code object metadata is specified in a note record (see 2496:ref:`amdgpu-note-records`) and is required when the target triple OS is 2497``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2498information necessary to support the HSA compatible runtime kernel queries. For 2499example, the segment sizes needed in a dispatch packet. In addition, a 2500high-level language runtime may require other information to be included. For 2501example, the AMD OpenCL runtime records kernel argument information. 2502 2503.. _amdgpu-amdhsa-code-object-metadata-v2: 2504 2505Code Object V2 Metadata 2506+++++++++++++++++++++++ 2507 2508.. warning:: 2509 Code object V2 is not the default code object version emitted by this version 2510 of LLVM. 2511 2512Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2513(see :ref:`amdgpu-note-records-v2`). 2514 2515The metadata is specified as a YAML formatted string (see [YAML]_ and 2516:doc:`YamlIO`). 2517 2518.. TODO:: 2519 2520 Is the string null terminated? It probably should not if YAML allows it to 2521 contain null characters, otherwise it should be. 2522 2523The metadata is represented as a single YAML document comprised of the mapping 2524defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2525referenced tables. 2526 2527For boolean values, the string values of ``false`` and ``true`` are used for 2528false and true respectively. 2529 2530Additional information can be added to the mappings. To avoid conflicts, any 2531non-AMD key names should be prefixed by "*vendor-name*.". 2532 2533 .. table:: AMDHSA Code Object V2 Metadata Map 2534 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2535 2536 ========== ============== ========= ======================================= 2537 String Key Value Type Required? Description 2538 ========== ============== ========= ======================================= 2539 "Version" sequence of Required - The first integer is the major 2540 2 integers version. Currently 1. 2541 - The second integer is the minor 2542 version. Currently 0. 2543 "Printf" sequence of Each string is encoded information 2544 strings about a printf function call. The 2545 encoded information is organized as 2546 fields separated by colon (':'): 2547 2548 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2549 2550 where: 2551 2552 ``ID`` 2553 A 32-bit integer as a unique id for 2554 each printf function call 2555 2556 ``N`` 2557 A 32-bit integer equal to the number 2558 of arguments of printf function call 2559 minus 1 2560 2561 ``S[i]`` (where i = 0, 1, ... , N-1) 2562 32-bit integers for the size in bytes 2563 of the i-th FormatString argument of 2564 the printf function call 2565 2566 FormatString 2567 The format string passed to the 2568 printf function call. 2569 "Kernels" sequence of Required Sequence of the mappings for each 2570 mapping kernel in the code object. See 2571 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2572 for the definition of the mapping. 2573 ========== ============== ========= ======================================= 2574 2575.. 2576 2577 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2578 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2579 2580 ================= ============== ========= ================================ 2581 String Key Value Type Required? Description 2582 ================= ============== ========= ================================ 2583 "Name" string Required Source name of the kernel. 2584 "SymbolName" string Required Name of the kernel 2585 descriptor ELF symbol. 2586 "Language" string Source language of the kernel. 2587 Values include: 2588 2589 - "OpenCL C" 2590 - "OpenCL C++" 2591 - "HCC" 2592 - "OpenMP" 2593 2594 "LanguageVersion" sequence of - The first integer is the major 2595 2 integers version. 2596 - The second integer is the 2597 minor version. 2598 "Attrs" mapping Mapping of kernel attributes. 2599 See 2600 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2601 for the mapping definition. 2602 "Args" sequence of Sequence of mappings of the 2603 mapping kernel arguments. See 2604 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2605 for the definition of the mapping. 2606 "CodeProps" mapping Mapping of properties related to 2607 the kernel code. See 2608 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2609 for the mapping definition. 2610 ================= ============== ========= ================================ 2611 2612.. 2613 2614 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2615 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2616 2617 =================== ============== ========= ============================== 2618 String Key Value Type Required? Description 2619 =================== ============== ========= ============================== 2620 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2621 3 integers must be >=1 and the dispatch 2622 work-group size X, Y, Z must 2623 correspond to the specified 2624 values. Defaults to 0, 0, 0. 2625 2626 Corresponds to the OpenCL 2627 ``reqd_work_group_size`` 2628 attribute. 2629 "WorkGroupSizeHint" sequence of The dispatch work-group size 2630 3 integers X, Y, Z is likely to be the 2631 specified values. 2632 2633 Corresponds to the OpenCL 2634 ``work_group_size_hint`` 2635 attribute. 2636 "VecTypeHint" string The name of a scalar or vector 2637 type. 2638 2639 Corresponds to the OpenCL 2640 ``vec_type_hint`` attribute. 2641 2642 "RuntimeHandle" string The external symbol name 2643 associated with a kernel. 2644 OpenCL runtime allocates a 2645 global buffer for the symbol 2646 and saves the kernel's address 2647 to it, which is used for 2648 device side enqueueing. Only 2649 available for device side 2650 enqueued kernels. 2651 =================== ============== ========= ============================== 2652 2653.. 2654 2655 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2656 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2657 2658 ================= ============== ========= ================================ 2659 String Key Value Type Required? Description 2660 ================= ============== ========= ================================ 2661 "Name" string Kernel argument name. 2662 "TypeName" string Kernel argument type name. 2663 "Size" integer Required Kernel argument size in bytes. 2664 "Align" integer Required Kernel argument alignment in 2665 bytes. Must be a power of two. 2666 "ValueKind" string Required Kernel argument kind that 2667 specifies how to set up the 2668 corresponding argument. 2669 Values include: 2670 2671 "ByValue" 2672 The argument is copied 2673 directly into the kernarg. 2674 2675 "GlobalBuffer" 2676 A global address space pointer 2677 to the buffer data is passed 2678 in the kernarg. 2679 2680 "DynamicSharedPointer" 2681 A group address space pointer 2682 to dynamically allocated LDS 2683 is passed in the kernarg. 2684 2685 "Sampler" 2686 A global address space 2687 pointer to a S# is passed in 2688 the kernarg. 2689 2690 "Image" 2691 A global address space 2692 pointer to a T# is passed in 2693 the kernarg. 2694 2695 "Pipe" 2696 A global address space pointer 2697 to an OpenCL pipe is passed in 2698 the kernarg. 2699 2700 "Queue" 2701 A global address space pointer 2702 to an OpenCL device enqueue 2703 queue is passed in the 2704 kernarg. 2705 2706 "HiddenGlobalOffsetX" 2707 The OpenCL grid dispatch 2708 global offset for the X 2709 dimension is passed in the 2710 kernarg. 2711 2712 "HiddenGlobalOffsetY" 2713 The OpenCL grid dispatch 2714 global offset for the Y 2715 dimension is passed in the 2716 kernarg. 2717 2718 "HiddenGlobalOffsetZ" 2719 The OpenCL grid dispatch 2720 global offset for the Z 2721 dimension is passed in the 2722 kernarg. 2723 2724 "HiddenNone" 2725 An argument that is not used 2726 by the kernel. Space needs to 2727 be left for it, but it does 2728 not need to be set up. 2729 2730 "HiddenPrintfBuffer" 2731 A global address space pointer 2732 to the runtime printf buffer 2733 is passed in kernarg. 2734 2735 "HiddenHostcallBuffer" 2736 A global address space pointer 2737 to the runtime hostcall buffer 2738 is passed in kernarg. 2739 2740 "HiddenDefaultQueue" 2741 A global address space pointer 2742 to the OpenCL device enqueue 2743 queue that should be used by 2744 the kernel by default is 2745 passed in the kernarg. 2746 2747 "HiddenCompletionAction" 2748 A global address space pointer 2749 to help link enqueued kernels into 2750 the ancestor tree for determining 2751 when the parent kernel has finished. 2752 2753 "HiddenMultiGridSyncArg" 2754 A global address space pointer for 2755 multi-grid synchronization is 2756 passed in the kernarg. 2757 2758 "ValueType" string Unused and deprecated. This should no longer 2759 be emitted, but is accepted for compatibility. 2760 2761 2762 "PointeeAlign" integer Alignment in bytes of pointee 2763 type for pointer type kernel 2764 argument. Must be a power 2765 of 2. Only present if 2766 "ValueKind" is 2767 "DynamicSharedPointer". 2768 "AddrSpaceQual" string Kernel argument address space 2769 qualifier. Only present if 2770 "ValueKind" is "GlobalBuffer" or 2771 "DynamicSharedPointer". Values 2772 are: 2773 2774 - "Private" 2775 - "Global" 2776 - "Constant" 2777 - "Local" 2778 - "Generic" 2779 - "Region" 2780 2781 .. TODO:: 2782 2783 Is GlobalBuffer only Global 2784 or Constant? Is 2785 DynamicSharedPointer always 2786 Local? Can HCC allow Generic? 2787 How can Private or Region 2788 ever happen? 2789 2790 "AccQual" string Kernel argument access 2791 qualifier. Only present if 2792 "ValueKind" is "Image" or 2793 "Pipe". Values 2794 are: 2795 2796 - "ReadOnly" 2797 - "WriteOnly" 2798 - "ReadWrite" 2799 2800 .. TODO:: 2801 2802 Does this apply to 2803 GlobalBuffer? 2804 2805 "ActualAccQual" string The actual memory accesses 2806 performed by the kernel on the 2807 kernel argument. Only present if 2808 "ValueKind" is "GlobalBuffer", 2809 "Image", or "Pipe". This may be 2810 more restrictive than indicated 2811 by "AccQual" to reflect what the 2812 kernel actual does. If not 2813 present then the runtime must 2814 assume what is implied by 2815 "AccQual" and "IsConst". Values 2816 are: 2817 2818 - "ReadOnly" 2819 - "WriteOnly" 2820 - "ReadWrite" 2821 2822 "IsConst" boolean Indicates if the kernel argument 2823 is const qualified. Only present 2824 if "ValueKind" is 2825 "GlobalBuffer". 2826 2827 "IsRestrict" boolean Indicates if the kernel argument 2828 is restrict qualified. Only 2829 present if "ValueKind" is 2830 "GlobalBuffer". 2831 2832 "IsVolatile" boolean Indicates if the kernel argument 2833 is volatile qualified. Only 2834 present if "ValueKind" is 2835 "GlobalBuffer". 2836 2837 "IsPipe" boolean Indicates if the kernel argument 2838 is pipe qualified. Only present 2839 if "ValueKind" is "Pipe". 2840 2841 .. TODO:: 2842 2843 Can GlobalBuffer be pipe 2844 qualified? 2845 2846 ================= ============== ========= ================================ 2847 2848.. 2849 2850 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2851 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2852 2853 ============================ ============== ========= ===================== 2854 String Key Value Type Required? Description 2855 ============================ ============== ========= ===================== 2856 "KernargSegmentSize" integer Required The size in bytes of 2857 the kernarg segment 2858 that holds the values 2859 of the arguments to 2860 the kernel. 2861 "GroupSegmentFixedSize" integer Required The amount of group 2862 segment memory 2863 required by a 2864 work-group in 2865 bytes. This does not 2866 include any 2867 dynamically allocated 2868 group segment memory 2869 that may be added 2870 when the kernel is 2871 dispatched. 2872 "PrivateSegmentFixedSize" integer Required The amount of fixed 2873 private address space 2874 memory required for a 2875 work-item in 2876 bytes. If the kernel 2877 uses a dynamic call 2878 stack then additional 2879 space must be added 2880 to this value for the 2881 call stack. 2882 "KernargSegmentAlign" integer Required The maximum byte 2883 alignment of 2884 arguments in the 2885 kernarg segment. Must 2886 be a power of 2. 2887 "WavefrontSize" integer Required Wavefront size. Must 2888 be a power of 2. 2889 "NumSGPRs" integer Required Number of scalar 2890 registers used by a 2891 wavefront for 2892 GFX6-GFX10. This 2893 includes the special 2894 SGPRs for VCC, Flat 2895 Scratch (GFX7-GFX10) 2896 and XNACK (for 2897 GFX8-GFX10). It does 2898 not include the 16 2899 SGPR added if a trap 2900 handler is 2901 enabled. It is not 2902 rounded up to the 2903 allocation 2904 granularity. 2905 "NumVGPRs" integer Required Number of vector 2906 registers used by 2907 each work-item for 2908 GFX6-GFX10 2909 "MaxFlatWorkGroupSize" integer Required Maximum flat 2910 work-group size 2911 supported by the 2912 kernel in work-items. 2913 Must be >=1 and 2914 consistent with 2915 ReqdWorkGroupSize if 2916 not 0, 0, 0. 2917 "NumSpilledSGPRs" integer Number of stores from 2918 a scalar register to 2919 a register allocator 2920 created spill 2921 location. 2922 "NumSpilledVGPRs" integer Number of stores from 2923 a vector register to 2924 a register allocator 2925 created spill 2926 location. 2927 ============================ ============== ========= ===================== 2928 2929.. _amdgpu-amdhsa-code-object-metadata-v3: 2930 2931Code Object V3 Metadata 2932+++++++++++++++++++++++ 2933 2934Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2935record (see :ref:`amdgpu-note-records-v3-v4`). 2936 2937The metadata is represented as Message Pack formatted binary data (see 2938[MsgPack]_). The top level is a Message Pack map that includes the 2939keys defined in table 2940:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2941tables. 2942 2943Additional information can be added to the maps. To avoid conflicts, 2944any key names should be prefixed by "*vendor-name*." where 2945``vendor-name`` can be the name of the vendor and specific vendor 2946tool that generates the information. The prefix is abbreviated to 2947simply "." when it appears within a map that has been added by the 2948same *vendor-name*. 2949 2950 .. table:: AMDHSA Code Object V3 Metadata Map 2951 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2952 2953 ================= ============== ========= ======================================= 2954 String Key Value Type Required? Description 2955 ================= ============== ========= ======================================= 2956 "amdhsa.version" sequence of Required - The first integer is the major 2957 2 integers version. Currently 1. 2958 - The second integer is the minor 2959 version. Currently 0. 2960 "amdhsa.printf" sequence of Each string is encoded information 2961 strings about a printf function call. The 2962 encoded information is organized as 2963 fields separated by colon (':'): 2964 2965 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2966 2967 where: 2968 2969 ``ID`` 2970 A 32-bit integer as a unique id for 2971 each printf function call 2972 2973 ``N`` 2974 A 32-bit integer equal to the number 2975 of arguments of printf function call 2976 minus 1 2977 2978 ``S[i]`` (where i = 0, 1, ... , N-1) 2979 32-bit integers for the size in bytes 2980 of the i-th FormatString argument of 2981 the printf function call 2982 2983 FormatString 2984 The format string passed to the 2985 printf function call. 2986 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2987 map kernel in the code object. See 2988 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2989 for the definition of the keys included 2990 in that map. 2991 ================= ============== ========= ======================================= 2992 2993.. 2994 2995 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2996 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2997 2998 =================================== ============== ========= ================================ 2999 String Key Value Type Required? Description 3000 =================================== ============== ========= ================================ 3001 ".name" string Required Source name of the kernel. 3002 ".symbol" string Required Name of the kernel 3003 descriptor ELF symbol. 3004 ".language" string Source language of the kernel. 3005 Values include: 3006 3007 - "OpenCL C" 3008 - "OpenCL C++" 3009 - "HCC" 3010 - "HIP" 3011 - "OpenMP" 3012 - "Assembler" 3013 3014 ".language_version" sequence of - The first integer is the major 3015 2 integers version. 3016 - The second integer is the 3017 minor version. 3018 ".args" sequence of Sequence of maps of the 3019 map kernel arguments. See 3020 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3021 for the definition of the keys 3022 included in that map. 3023 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3024 3 integers must be >=1 and the dispatch 3025 work-group size X, Y, Z must 3026 correspond to the specified 3027 values. Defaults to 0, 0, 0. 3028 3029 Corresponds to the OpenCL 3030 ``reqd_work_group_size`` 3031 attribute. 3032 ".workgroup_size_hint" sequence of The dispatch work-group size 3033 3 integers X, Y, Z is likely to be the 3034 specified values. 3035 3036 Corresponds to the OpenCL 3037 ``work_group_size_hint`` 3038 attribute. 3039 ".vec_type_hint" string The name of a scalar or vector 3040 type. 3041 3042 Corresponds to the OpenCL 3043 ``vec_type_hint`` attribute. 3044 3045 ".device_enqueue_symbol" string The external symbol name 3046 associated with a kernel. 3047 OpenCL runtime allocates a 3048 global buffer for the symbol 3049 and saves the kernel's address 3050 to it, which is used for 3051 device side enqueueing. Only 3052 available for device side 3053 enqueued kernels. 3054 ".kernarg_segment_size" integer Required The size in bytes of 3055 the kernarg segment 3056 that holds the values 3057 of the arguments to 3058 the kernel. 3059 ".group_segment_fixed_size" integer Required The amount of group 3060 segment memory 3061 required by a 3062 work-group in 3063 bytes. This does not 3064 include any 3065 dynamically allocated 3066 group segment memory 3067 that may be added 3068 when the kernel is 3069 dispatched. 3070 ".private_segment_fixed_size" integer Required The amount of fixed 3071 private address space 3072 memory required for a 3073 work-item in 3074 bytes. If the kernel 3075 uses a dynamic call 3076 stack then additional 3077 space must be added 3078 to this value for the 3079 call stack. 3080 ".kernarg_segment_align" integer Required The maximum byte 3081 alignment of 3082 arguments in the 3083 kernarg segment. Must 3084 be a power of 2. 3085 ".wavefront_size" integer Required Wavefront size. Must 3086 be a power of 2. 3087 ".sgpr_count" integer Required Number of scalar 3088 registers required by a 3089 wavefront for 3090 GFX6-GFX9. A register 3091 is required if it is 3092 used explicitly, or 3093 if a higher numbered 3094 register is used 3095 explicitly. This 3096 includes the special 3097 SGPRs for VCC, Flat 3098 Scratch (GFX7-GFX9) 3099 and XNACK (for 3100 GFX8-GFX9). It does 3101 not include the 16 3102 SGPR added if a trap 3103 handler is 3104 enabled. It is not 3105 rounded up to the 3106 allocation 3107 granularity. 3108 ".vgpr_count" integer Required Number of vector 3109 registers required by 3110 each work-item for 3111 GFX6-GFX9. A register 3112 is required if it is 3113 used explicitly, or 3114 if a higher numbered 3115 register is used 3116 explicitly. 3117 ".max_flat_workgroup_size" integer Required Maximum flat 3118 work-group size 3119 supported by the 3120 kernel in work-items. 3121 Must be >=1 and 3122 consistent with 3123 ReqdWorkGroupSize if 3124 not 0, 0, 0. 3125 ".sgpr_spill_count" integer Number of stores from 3126 a scalar register to 3127 a register allocator 3128 created spill 3129 location. 3130 ".vgpr_spill_count" integer Number of stores from 3131 a vector register to 3132 a register allocator 3133 created spill 3134 location. 3135 =================================== ============== ========= ================================ 3136 3137.. 3138 3139 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3140 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3141 3142 ====================== ============== ========= ================================ 3143 String Key Value Type Required? Description 3144 ====================== ============== ========= ================================ 3145 ".name" string Kernel argument name. 3146 ".type_name" string Kernel argument type name. 3147 ".size" integer Required Kernel argument size in bytes. 3148 ".offset" integer Required Kernel argument offset in 3149 bytes. The offset must be a 3150 multiple of the alignment 3151 required by the argument. 3152 ".value_kind" string Required Kernel argument kind that 3153 specifies how to set up the 3154 corresponding argument. 3155 Values include: 3156 3157 "by_value" 3158 The argument is copied 3159 directly into the kernarg. 3160 3161 "global_buffer" 3162 A global address space pointer 3163 to the buffer data is passed 3164 in the kernarg. 3165 3166 "dynamic_shared_pointer" 3167 A group address space pointer 3168 to dynamically allocated LDS 3169 is passed in the kernarg. 3170 3171 "sampler" 3172 A global address space 3173 pointer to a S# is passed in 3174 the kernarg. 3175 3176 "image" 3177 A global address space 3178 pointer to a T# is passed in 3179 the kernarg. 3180 3181 "pipe" 3182 A global address space pointer 3183 to an OpenCL pipe is passed in 3184 the kernarg. 3185 3186 "queue" 3187 A global address space pointer 3188 to an OpenCL device enqueue 3189 queue is passed in the 3190 kernarg. 3191 3192 "hidden_global_offset_x" 3193 The OpenCL grid dispatch 3194 global offset for the X 3195 dimension is passed in the 3196 kernarg. 3197 3198 "hidden_global_offset_y" 3199 The OpenCL grid dispatch 3200 global offset for the Y 3201 dimension is passed in the 3202 kernarg. 3203 3204 "hidden_global_offset_z" 3205 The OpenCL grid dispatch 3206 global offset for the Z 3207 dimension is passed in the 3208 kernarg. 3209 3210 "hidden_none" 3211 An argument that is not used 3212 by the kernel. Space needs to 3213 be left for it, but it does 3214 not need to be set up. 3215 3216 "hidden_printf_buffer" 3217 A global address space pointer 3218 to the runtime printf buffer 3219 is passed in kernarg. 3220 3221 "hidden_hostcall_buffer" 3222 A global address space pointer 3223 to the runtime hostcall buffer 3224 is passed in kernarg. 3225 3226 "hidden_default_queue" 3227 A global address space pointer 3228 to the OpenCL device enqueue 3229 queue that should be used by 3230 the kernel by default is 3231 passed in the kernarg. 3232 3233 "hidden_completion_action" 3234 A global address space pointer 3235 to help link enqueued kernels into 3236 the ancestor tree for determining 3237 when the parent kernel has finished. 3238 3239 "hidden_multigrid_sync_arg" 3240 A global address space pointer for 3241 multi-grid synchronization is 3242 passed in the kernarg. 3243 3244 ".value_type" string Unused and deprecated. This should no longer 3245 be emitted, but is accepted for compatibility. 3246 3247 ".pointee_align" integer Alignment in bytes of pointee 3248 type for pointer type kernel 3249 argument. Must be a power 3250 of 2. Only present if 3251 ".value_kind" is 3252 "dynamic_shared_pointer". 3253 ".address_space" string Kernel argument address space 3254 qualifier. Only present if 3255 ".value_kind" is "global_buffer" or 3256 "dynamic_shared_pointer". Values 3257 are: 3258 3259 - "private" 3260 - "global" 3261 - "constant" 3262 - "local" 3263 - "generic" 3264 - "region" 3265 3266 .. TODO:: 3267 3268 Is "global_buffer" only "global" 3269 or "constant"? Is 3270 "dynamic_shared_pointer" always 3271 "local"? Can HCC allow "generic"? 3272 How can "private" or "region" 3273 ever happen? 3274 3275 ".access" string Kernel argument access 3276 qualifier. Only present if 3277 ".value_kind" is "image" or 3278 "pipe". Values 3279 are: 3280 3281 - "read_only" 3282 - "write_only" 3283 - "read_write" 3284 3285 .. TODO:: 3286 3287 Does this apply to 3288 "global_buffer"? 3289 3290 ".actual_access" string The actual memory accesses 3291 performed by the kernel on the 3292 kernel argument. Only present if 3293 ".value_kind" is "global_buffer", 3294 "image", or "pipe". This may be 3295 more restrictive than indicated 3296 by ".access" to reflect what the 3297 kernel actual does. If not 3298 present then the runtime must 3299 assume what is implied by 3300 ".access" and ".is_const" . Values 3301 are: 3302 3303 - "read_only" 3304 - "write_only" 3305 - "read_write" 3306 3307 ".is_const" boolean Indicates if the kernel argument 3308 is const qualified. Only present 3309 if ".value_kind" is 3310 "global_buffer". 3311 3312 ".is_restrict" boolean Indicates if the kernel argument 3313 is restrict qualified. Only 3314 present if ".value_kind" is 3315 "global_buffer". 3316 3317 ".is_volatile" boolean Indicates if the kernel argument 3318 is volatile qualified. Only 3319 present if ".value_kind" is 3320 "global_buffer". 3321 3322 ".is_pipe" boolean Indicates if the kernel argument 3323 is pipe qualified. Only present 3324 if ".value_kind" is "pipe". 3325 3326 .. TODO:: 3327 3328 Can "global_buffer" be pipe 3329 qualified? 3330 3331 ====================== ============== ========= ================================ 3332 3333.. _amdgpu-amdhsa-code-object-metadata-v4: 3334 3335Code Object V4 Metadata 3336+++++++++++++++++++++++ 3337 3338.. warning:: 3339 Code object V4 is not the default code object version emitted by this version 3340 of LLVM. 3341 3342Code object V4 metadata is the same as 3343:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3344defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3345 3346 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3347 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3348 3349 ================= ============== ========= ======================================= 3350 String Key Value Type Required? Description 3351 ================= ============== ========= ======================================= 3352 "amdhsa.version" sequence of Required - The first integer is the major 3353 2 integers version. Currently 1. 3354 - The second integer is the minor 3355 version. Currently 1. 3356 "amdhsa.target" string Required The target name of the code using the syntax: 3357 3358 .. code:: 3359 3360 <target-triple> [ "-" <target-id> ] 3361 3362 A canonical target ID must be 3363 used. See :ref:`amdgpu-target-triples` 3364 and :ref:`amdgpu-target-id`. 3365 ================= ============== ========= ======================================= 3366 3367.. 3368 3369Kernel Dispatch 3370~~~~~~~~~~~~~~~ 3371 3372The HSA architected queuing language (AQL) defines a user space memory interface 3373that can be used to control the dispatch of kernels, in an agent independent 3374way. An agent can have zero or more AQL queues created for it using an HSA 3375compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3376are 64 bytes) can be placed. See the *HSA Platform System Architecture 3377Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3378 3379The packet processor of a kernel agent is responsible for detecting and 3380dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3381packet processor is implemented by the hardware command processor (CP), 3382asynchronous dispatch controller (ADC) and shader processor input controller 3383(SPI). 3384 3385An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3386the kernel mode driver to initialize and register the AQL queue with CP. 3387 3388To dispatch a kernel the following actions are performed. This can occur in the 3389CPU host program, or from an HSA kernel executing on a GPU. 3390 33911. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3392 executed is obtained. 33932. A pointer to the kernel descriptor (see 3394 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3395 It must be for a kernel that is contained in a code object that that was 3396 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3397 queue is associated. 33983. Space is allocated for the kernel arguments using the HSA compatible runtime 3399 allocator for a memory region with the kernarg property for the kernel agent 3400 that will execute the kernel. It must be at least 16-byte aligned. 34014. Kernel argument values are assigned to the kernel argument memory 3402 allocation. The layout is defined in the *HSA Programmer's Language 3403 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3404 kernel argument memory in the same way constant memory is accessed. (Note 3405 that the HSA specification allows an implementation to copy the kernel 3406 argument contents to another location that is accessed by the kernel.) 34075. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3408 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3409 for the packet. The packet must be set up, and the final write must use an 3410 atomic store release to set the packet kind to ensure the packet contents are 3411 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3412 notify the kernel agent that the AQL queue has been updated. These rules, and 3413 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3414 System Architecture Specification* [HSA]_. 34156. A kernel dispatch packet includes information about the actual dispatch, 3416 such as grid and work-group size, together with information from the code 3417 object about the kernel, such as segment sizes. The HSA compatible runtime 3418 queries on the kernel symbol can be used to obtain the code object values 3419 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 34207. CP executes micro-code and is responsible for detecting and setting up the 3421 GPU to execute the wavefronts of a kernel dispatch. 34228. CP ensures that when the a wavefront starts executing the kernel machine 3423 code, the scalar general purpose registers (SGPR) and vector general purpose 3424 registers (VGPR) are set up as required by the machine code. The required 3425 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3426 register state is defined in 3427 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 34289. The prolog of the kernel machine code (see 3429 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3430 before continuing executing the machine code that corresponds to the kernel. 343110. When the kernel dispatch has completed execution, CP signals the completion 3432 signal specified in the kernel dispatch packet if not 0. 3433 3434.. _amdgpu-amdhsa-memory-spaces: 3435 3436Memory Spaces 3437~~~~~~~~~~~~~ 3438 3439The memory space properties are: 3440 3441 .. table:: AMDHSA Memory Spaces 3442 :name: amdgpu-amdhsa-memory-spaces-table 3443 3444 ================= =========== ======== ======= ================== 3445 Memory Space Name HSA Segment Hardware Address NULL Value 3446 Name Name Size 3447 ================= =========== ======== ======= ================== 3448 Private private scratch 32 0x00000000 3449 Local group LDS 32 0xFFFFFFFF 3450 Global global global 64 0x0000000000000000 3451 Constant constant *same as 64 0x0000000000000000 3452 global* 3453 Generic flat flat 64 0x0000000000000000 3454 Region N/A GDS 32 *not implemented 3455 for AMDHSA* 3456 ================= =========== ======== ======= ================== 3457 3458The global and constant memory spaces both use global virtual addresses, which 3459are the same virtual address space used by the CPU. However, some virtual 3460addresses may only be accessible to the CPU, some only accessible by the GPU, 3461and some by both. 3462 3463Using the constant memory space indicates that the data will not change during 3464the execution of the kernel. This allows scalar read instructions to be 3465used. The vector and scalar L1 caches are invalidated of volatile data before 3466each kernel dispatch execution to allow constant memory to change values between 3467kernel dispatches. 3468 3469The local memory space uses the hardware Local Data Store (LDS) which is 3470automatically allocated when the hardware creates work-groups of wavefronts, and 3471freed when all the wavefronts of a work-group have terminated. The data store 3472(DS) instructions can be used to access it. 3473 3474The private memory space uses the hardware scratch memory support. If the kernel 3475uses scratch, then the hardware allocates memory that is accessed using 3476wavefront lane dword (4 byte) interleaving. The mapping used from private 3477address to physical address is: 3478 3479 ``wavefront-scratch-base + 3480 (private-address * wavefront-size * 4) + 3481 (wavefront-lane-id * 4)`` 3482 3483There are different ways that the wavefront scratch base address is determined 3484by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3485memory can be accessed in an interleaved manner using buffer instruction with 3486the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3487instructions, or by flat instructions. If each lane of a wavefront accesses the 3488same private address, the interleaving results in adjacent dwords being accessed 3489and hence requires fewer cache lines to be fetched. Multi-dword access is not 3490supported except by flat and scratch instructions in GFX9-GFX10. 3491 3492The generic address space uses the hardware flat address support available in 3493GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3494local apertures), that are outside the range of addressible global memory, to 3495map from a flat address to a private or local address. 3496 3497FLAT instructions can take a flat address and access global, private (scratch) 3498and group (LDS) memory depending in if the address is within one of the 3499aperture ranges. Flat access to scratch requires hardware aperture setup and 3500setup in the kernel prologue (see 3501:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3502hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3503:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3504 3505To convert between a segment address and a flat address the base address of the 3506apertures address can be used. For GFX7-GFX8 these are available in the 3507:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3508Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3509GFX9-GFX10 the aperture base addresses are directly available as inline constant 3510registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3511address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3512which makes it easier to convert from flat to segment or segment to flat. 3513 3514Image and Samplers 3515~~~~~~~~~~~~~~~~~~ 3516 3517Image and sample handles created by an HSA compatible runtime (see 3518:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3519object respectively. In order to support the HSA ``query_sampler`` operations 3520two extra dwords are used to store the HSA BRIG enumeration values for the 3521queries that are not trivially deducible from the S# representation. 3522 3523HSA Signals 3524~~~~~~~~~~~ 3525 3526HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3527are 64-bit addresses of a structure allocated in memory accessible from both the 3528CPU and GPU. The structure is defined by the runtime and subject to change 3529between releases. For example, see [AMD-ROCm-github]_. 3530 3531.. _amdgpu-amdhsa-hsa-aql-queue: 3532 3533HSA AQL Queue 3534~~~~~~~~~~~~~ 3535 3536The HSA AQL queue structure is defined by an HSA compatible runtime (see 3537:ref:`amdgpu-os`) and subject to change between releases. For example, see 3538[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3539certain language features such as the flat address aperture bases. It also 3540contains fields used by CP such as managing the allocation of scratch memory. 3541 3542.. _amdgpu-amdhsa-kernel-descriptor: 3543 3544Kernel Descriptor 3545~~~~~~~~~~~~~~~~~ 3546 3547A kernel descriptor consists of the information needed by CP to initiate the 3548execution of a kernel, including the entry point address of the machine code 3549that implements the kernel. 3550 3551Code Object V3 Kernel Descriptor 3552++++++++++++++++++++++++++++++++ 3553 3554CP microcode requires the Kernel descriptor to be allocated on 64-byte 3555alignment. 3556 3557The fields used by CP for code objects before V3 also match those specified in 3558:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3559 3560 .. table:: Code Object V3 Kernel Descriptor 3561 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3562 3563 ======= ======= =============================== ============================ 3564 Bits Size Field Name Description 3565 ======= ======= =============================== ============================ 3566 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3567 address space memory 3568 required for a work-group 3569 in bytes. This does not 3570 include any dynamically 3571 allocated local address 3572 space memory that may be 3573 added when the kernel is 3574 dispatched. 3575 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3576 private address space 3577 memory required for a 3578 work-item in bytes. 3579 Additional space may need to 3580 be added to this value if 3581 the call stack has 3582 non-inlined function calls. 3583 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3584 memory pointed to by the 3585 AQL dispatch packet. The 3586 kernarg memory is used to 3587 pass arguments to the 3588 kernel. 3589 3590 * If the kernarg pointer in 3591 the dispatch packet is NULL 3592 then there are no kernel 3593 arguments. 3594 * If the kernarg pointer in 3595 the dispatch packet is 3596 not NULL and this value 3597 is 0 then the kernarg 3598 memory size is 3599 unspecified. 3600 * If the kernarg pointer in 3601 the dispatch packet is 3602 not NULL and this value 3603 is not 0 then the value 3604 specifies the kernarg 3605 memory size in bytes. It 3606 is recommended to provide 3607 a value as it may be used 3608 by CP to optimize making 3609 the kernarg memory 3610 visible to the kernel 3611 code. 3612 3613 127:96 4 bytes Reserved, must be 0. 3614 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3615 negative) from base 3616 address of kernel 3617 descriptor to kernel's 3618 entry point instruction 3619 which must be 256 byte 3620 aligned. 3621 351:272 20 Reserved, must be 0. 3622 bytes 3623 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3624 Reserved, must be 0. 3625 GFX90A 3626 Compute Shader (CS) 3627 program settings used by 3628 CP to set up 3629 ``COMPUTE_PGM_RSRC3`` 3630 configuration 3631 register. See 3632 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3633 GFX10 3634 Compute Shader (CS) 3635 program settings used by 3636 CP to set up 3637 ``COMPUTE_PGM_RSRC3`` 3638 configuration 3639 register. See 3640 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3641 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3642 program settings used by 3643 CP to set up 3644 ``COMPUTE_PGM_RSRC1`` 3645 configuration 3646 register. See 3647 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3648 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3649 program settings used by 3650 CP to set up 3651 ``COMPUTE_PGM_RSRC2`` 3652 configuration 3653 register. See 3654 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3655 458:448 7 bits *See separate bits below.* Enable the setup of the 3656 SGPR user data registers 3657 (see 3658 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3659 3660 The total number of SGPR 3661 user data registers 3662 requested must not exceed 3663 16 and match value in 3664 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3665 Any requests beyond 16 3666 will be ignored. 3667 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3668 _BUFFER column of 3669 :ref:`amdgpu-processor-table` 3670 specifies *Architected flat 3671 scratch* then not supported 3672 and must be 0, 3673 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3674 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3675 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3676 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3677 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3678 column of 3679 :ref:`amdgpu-processor-table` 3680 specifies *Architected flat 3681 scratch* then not supported 3682 and must be 0, 3683 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3684 _SIZE 3685 457:455 3 bits Reserved, must be 0. 3686 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3687 Reserved, must be 0. 3688 GFX10 3689 - If 0 execute in 3690 wavefront size 64 mode. 3691 - If 1 execute in 3692 native wavefront size 3693 32 mode. 3694 463:459 1 bit Reserved, must be 0. 3695 464 1 bit RESERVED_464 Deprecated, must be 0. 3696 467:465 3 bits Reserved, must be 0. 3697 468 1 bit RESERVED_468 Deprecated, must be 0. 3698 469:471 3 bits Reserved, must be 0. 3699 511:472 5 bytes Reserved, must be 0. 3700 512 **Total size 64 bytes.** 3701 ======= ==================================================================== 3702 3703.. 3704 3705 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3706 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3707 3708 ======= ======= =============================== =========================================================================== 3709 Bits Size Field Name Description 3710 ======= ======= =============================== =========================================================================== 3711 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3712 blocks used by each work-item; 3713 granularity is device 3714 specific: 3715 3716 GFX6-GFX9 3717 - vgprs_used 0..256 3718 - max(0, ceil(vgprs_used / 4) - 1) 3719 GFX90A 3720 - vgprs_used 0..512 3721 - vgprs_used = align(arch_vgprs, 4) 3722 + acc_vgprs 3723 - max(0, ceil(vgprs_used / 8) - 1) 3724 GFX10 (wavefront size 64) 3725 - max_vgpr 1..256 3726 - max(0, ceil(vgprs_used / 4) - 1) 3727 GFX10 (wavefront size 32) 3728 - max_vgpr 1..256 3729 - max(0, ceil(vgprs_used / 8) - 1) 3730 3731 Where vgprs_used is defined 3732 as the highest VGPR number 3733 explicitly referenced plus 3734 one. 3735 3736 Used by CP to set up 3737 ``COMPUTE_PGM_RSRC1.VGPRS``. 3738 3739 The 3740 :ref:`amdgpu-assembler` 3741 calculates this 3742 automatically for the 3743 selected processor from 3744 values provided to the 3745 `.amdhsa_kernel` directive 3746 by the 3747 `.amdhsa_next_free_vgpr` 3748 nested directive (see 3749 :ref:`amdhsa-kernel-directives-table`). 3750 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3751 blocks used by a wavefront; 3752 granularity is device 3753 specific: 3754 3755 GFX6-GFX8 3756 - sgprs_used 0..112 3757 - max(0, ceil(sgprs_used / 8) - 1) 3758 GFX9 3759 - sgprs_used 0..112 3760 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3761 GFX10 3762 Reserved, must be 0. 3763 (128 SGPRs always 3764 allocated.) 3765 3766 Where sgprs_used is 3767 defined as the highest 3768 SGPR number explicitly 3769 referenced plus one, plus 3770 a target specific number 3771 of additional special 3772 SGPRs for VCC, 3773 FLAT_SCRATCH (GFX7+) and 3774 XNACK_MASK (GFX8+), and 3775 any additional 3776 target specific 3777 limitations. It does not 3778 include the 16 SGPRs added 3779 if a trap handler is 3780 enabled. 3781 3782 The target specific 3783 limitations and special 3784 SGPR layout are defined in 3785 the hardware 3786 documentation, which can 3787 be found in the 3788 :ref:`amdgpu-processors` 3789 table. 3790 3791 Used by CP to set up 3792 ``COMPUTE_PGM_RSRC1.SGPRS``. 3793 3794 The 3795 :ref:`amdgpu-assembler` 3796 calculates this 3797 automatically for the 3798 selected processor from 3799 values provided to the 3800 `.amdhsa_kernel` directive 3801 by the 3802 `.amdhsa_next_free_sgpr` 3803 and `.amdhsa_reserve_*` 3804 nested directives (see 3805 :ref:`amdhsa-kernel-directives-table`). 3806 11:10 2 bits PRIORITY Must be 0. 3807 3808 Start executing wavefront 3809 at the specified priority. 3810 3811 CP is responsible for 3812 filling in 3813 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3814 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3815 with specified rounding 3816 mode for single (32 3817 bit) floating point 3818 precision floating point 3819 operations. 3820 3821 Floating point rounding 3822 mode values are defined in 3823 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3824 3825 Used by CP to set up 3826 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3827 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3828 with specified rounding 3829 denorm mode for half/double (16 3830 and 64-bit) floating point 3831 precision floating point 3832 operations. 3833 3834 Floating point rounding 3835 mode values are defined in 3836 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3837 3838 Used by CP to set up 3839 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3840 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3841 with specified denorm mode 3842 for single (32 3843 bit) floating point 3844 precision floating point 3845 operations. 3846 3847 Floating point denorm mode 3848 values are defined in 3849 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3850 3851 Used by CP to set up 3852 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3853 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3854 with specified denorm mode 3855 for half/double (16 3856 and 64-bit) floating point 3857 precision floating point 3858 operations. 3859 3860 Floating point denorm mode 3861 values are defined in 3862 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3863 3864 Used by CP to set up 3865 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3866 20 1 bit PRIV Must be 0. 3867 3868 Start executing wavefront 3869 in privilege trap handler 3870 mode. 3871 3872 CP is responsible for 3873 filling in 3874 ``COMPUTE_PGM_RSRC1.PRIV``. 3875 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3876 with DX10 clamp mode 3877 enabled. Used by the vector 3878 ALU to force DX10 style 3879 treatment of NaN's (when 3880 set, clamp NaN to zero, 3881 otherwise pass NaN 3882 through). 3883 3884 Used by CP to set up 3885 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3886 22 1 bit DEBUG_MODE Must be 0. 3887 3888 Start executing wavefront 3889 in single step mode. 3890 3891 CP is responsible for 3892 filling in 3893 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3894 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3895 with IEEE mode 3896 enabled. Floating point 3897 opcodes that support 3898 exception flag gathering 3899 will quiet and propagate 3900 signaling-NaN inputs per 3901 IEEE 754-2008. Min_dx10 and 3902 max_dx10 become IEEE 3903 754-2008 compliant due to 3904 signaling-NaN propagation 3905 and quieting. 3906 3907 Used by CP to set up 3908 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3909 24 1 bit BULKY Must be 0. 3910 3911 Only one work-group allowed 3912 to execute on a compute 3913 unit. 3914 3915 CP is responsible for 3916 filling in 3917 ``COMPUTE_PGM_RSRC1.BULKY``. 3918 25 1 bit CDBG_USER Must be 0. 3919 3920 Flag that can be used to 3921 control debugging code. 3922 3923 CP is responsible for 3924 filling in 3925 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3926 26 1 bit FP16_OVFL GFX6-GFX8 3927 Reserved, must be 0. 3928 GFX9-GFX10 3929 Wavefront starts execution 3930 with specified fp16 overflow 3931 mode. 3932 3933 - If 0, fp16 overflow generates 3934 +/-INF values. 3935 - If 1, fp16 overflow that is the 3936 result of an +/-INF input value 3937 or divide by 0 produces a +/-INF, 3938 otherwise clamps computed 3939 overflow to +/-MAX_FP16 as 3940 appropriate. 3941 3942 Used by CP to set up 3943 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3944 28:27 2 bits Reserved, must be 0. 3945 29 1 bit WGP_MODE GFX6-GFX9 3946 Reserved, must be 0. 3947 GFX10 3948 - If 0 execute work-groups in 3949 CU wavefront execution mode. 3950 - If 1 execute work-groups on 3951 in WGP wavefront execution mode. 3952 3953 See :ref:`amdgpu-amdhsa-memory-model`. 3954 3955 Used by CP to set up 3956 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3957 30 1 bit MEM_ORDERED GFX6-GFX9 3958 Reserved, must be 0. 3959 GFX10 3960 Controls the behavior of the 3961 s_waitcnt's vmcnt and vscnt 3962 counters. 3963 3964 - If 0 vmcnt reports completion 3965 of load and atomic with return 3966 out of order with sample 3967 instructions, and the vscnt 3968 reports the completion of 3969 store and atomic without 3970 return in order. 3971 - If 1 vmcnt reports completion 3972 of load, atomic with return 3973 and sample instructions in 3974 order, and the vscnt reports 3975 the completion of store and 3976 atomic without return in order. 3977 3978 Used by CP to set up 3979 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3980 31 1 bit FWD_PROGRESS GFX6-GFX9 3981 Reserved, must be 0. 3982 GFX10 3983 - If 0 execute SIMD wavefronts 3984 using oldest first policy. 3985 - If 1 execute SIMD wavefronts to 3986 ensure wavefronts will make some 3987 forward progress. 3988 3989 Used by CP to set up 3990 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3991 32 **Total size 4 bytes** 3992 ======= =================================================================================================================== 3993 3994.. 3995 3996 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3997 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3998 3999 ======= ======= =============================== =========================================================================== 4000 Bits Size Field Name Description 4001 ======= ======= =============================== =========================================================================== 4002 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4003 private segment. 4004 * If the *Target Properties* 4005 column of 4006 :ref:`amdgpu-processor-table` 4007 does not specify 4008 *Architected flat 4009 scratch* then enable the 4010 setup of the SGPR 4011 wavefront scratch offset 4012 system register (see 4013 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4014 * If the *Target Properties* 4015 column of 4016 :ref:`amdgpu-processor-table` 4017 specifies *Architected 4018 flat scratch* then enable 4019 the setup of the 4020 FLAT_SCRATCH register 4021 pair (see 4022 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4023 4024 Used by CP to set up 4025 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4026 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4027 user data registers 4028 requested. This number must 4029 match the number of user 4030 data registers enabled. 4031 4032 Used by CP to set up 4033 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4034 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4035 4036 This bit represents 4037 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4038 which is set by the CP if 4039 the runtime has installed a 4040 trap handler. 4041 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4042 system SGPR register for 4043 the work-group id in the X 4044 dimension (see 4045 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4046 4047 Used by CP to set up 4048 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4049 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4050 system SGPR register for 4051 the work-group id in the Y 4052 dimension (see 4053 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4054 4055 Used by CP to set up 4056 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4057 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4058 system SGPR register for 4059 the work-group id in the Z 4060 dimension (see 4061 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4062 4063 Used by CP to set up 4064 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4065 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4066 system SGPR register for 4067 work-group information (see 4068 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4069 4070 Used by CP to set up 4071 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4072 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4073 VGPR system registers used 4074 for the work-item ID. 4075 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4076 defines the values. 4077 4078 Used by CP to set up 4079 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4080 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4081 4082 Wavefront starts execution 4083 with address watch 4084 exceptions enabled which 4085 are generated when L1 has 4086 witnessed a thread access 4087 an *address of 4088 interest*. 4089 4090 CP is responsible for 4091 filling in the address 4092 watch bit in 4093 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4094 according to what the 4095 runtime requests. 4096 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4097 4098 Wavefront starts execution 4099 with memory violation 4100 exceptions exceptions 4101 enabled which are generated 4102 when a memory violation has 4103 occurred for this wavefront from 4104 L1 or LDS 4105 (write-to-read-only-memory, 4106 mis-aligned atomic, LDS 4107 address out of range, 4108 illegal address, etc.). 4109 4110 CP sets the memory 4111 violation bit in 4112 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4113 according to what the 4114 runtime requests. 4115 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4116 4117 CP uses the rounded value 4118 from the dispatch packet, 4119 not this value, as the 4120 dispatch may contain 4121 dynamically allocated group 4122 segment memory. CP writes 4123 directly to 4124 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4125 4126 Amount of group segment 4127 (LDS) to allocate for each 4128 work-group. Granularity is 4129 device specific: 4130 4131 GFX6 4132 roundup(lds-size / (64 * 4)) 4133 GFX7-GFX10 4134 roundup(lds-size / (128 * 4)) 4135 4136 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4137 _INVALID_OPERATION with specified exceptions 4138 enabled. 4139 4140 Used by CP to set up 4141 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4142 (set from bits 0..6). 4143 4144 IEEE 754 FP Invalid 4145 Operation 4146 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4147 _SOURCE input operands is a 4148 denormal number 4149 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4150 _DIVISION_BY_ZERO Zero 4151 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4152 _OVERFLOW 4153 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4154 _UNDERFLOW 4155 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4156 _INEXACT 4157 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4158 _ZERO (rcp_iflag_f32 instruction 4159 only) 4160 31 1 bit Reserved, must be 0. 4161 32 **Total size 4 bytes.** 4162 ======= =================================================================================================================== 4163 4164.. 4165 4166 .. table:: compute_pgm_rsrc3 for GFX90A 4167 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4168 4169 ======= ======= =============================== =========================================================================== 4170 Bits Size Field Name Description 4171 ======= ======= =============================== =========================================================================== 4172 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4173 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4174 63 - accum-offset = 256. 4175 6:15 10 Reserved, must be 0. 4176 bits 4177 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4178 launched in the same CU. 4179 - If 1 the waves of a work-group can be 4180 launched in different CUs. The waves 4181 cannot use S_BARRIER or LDS. 4182 17:31 15 Reserved, must be 0. 4183 bits 4184 32 **Total size 4 bytes.** 4185 ======= =================================================================================================================== 4186 4187.. 4188 4189 .. table:: compute_pgm_rsrc3 for GFX10 4190 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4191 4192 ======= ======= =============================== =========================================================================== 4193 Bits Size Field Name Description 4194 ======= ======= =============================== =========================================================================== 4195 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4196 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4197 31:4 28 Reserved, must be 0. 4198 bits 4199 32 **Total size 4 bytes.** 4200 ======= =================================================================================================================== 4201 4202.. 4203 4204 .. table:: Floating Point Rounding Mode Enumeration Values 4205 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4206 4207 ====================================== ===== ============================== 4208 Enumeration Name Value Description 4209 ====================================== ===== ============================== 4210 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4211 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4212 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4213 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4214 ====================================== ===== ============================== 4215 4216.. 4217 4218 .. table:: Floating Point Denorm Mode Enumeration Values 4219 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4220 4221 ====================================== ===== ============================== 4222 Enumeration Name Value Description 4223 ====================================== ===== ============================== 4224 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4225 Denorms 4226 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4227 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4228 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4229 ====================================== ===== ============================== 4230 4231.. 4232 4233 .. table:: System VGPR Work-Item ID Enumeration Values 4234 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4235 4236 ======================================== ===== ============================ 4237 Enumeration Name Value Description 4238 ======================================== ===== ============================ 4239 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4240 ID. 4241 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4242 dimensions ID. 4243 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4244 dimensions ID. 4245 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4246 ======================================== ===== ============================ 4247 4248.. _amdgpu-amdhsa-initial-kernel-execution-state: 4249 4250Initial Kernel Execution State 4251~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4252 4253This section defines the register state that will be set up by the packet 4254processor prior to the start of execution of every wavefront. This is limited by 4255the constraints of the hardware controllers of CP/ADC/SPI. 4256 4257The order of the SGPR registers is defined, but the compiler can specify which 4258ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4259fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4260for enabled registers are dense starting at SGPR0: the first enabled register is 4261SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4262an SGPR number. 4263 4264The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4265all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4266using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4267actually initialized. These are then immediately followed by the System SGPRs 4268that are set up by ADC/SPI and can have different values for each wavefront of 4269the grid dispatch. 4270 4271SGPR register initial state is defined in 4272:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4273 4274 .. table:: SGPR Register Set Up Order 4275 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4276 4277 ========== ========================== ====== ============================== 4278 SGPR Order Name Number Description 4279 (kernel descriptor enable of 4280 field) SGPRs 4281 ========== ========================== ====== ============================== 4282 First Private Segment Buffer 4 See 4283 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4284 _segment_buffer) 4285 then Dispatch Ptr 2 64-bit address of AQL dispatch 4286 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4287 actually executing. 4288 then Queue Ptr 2 64-bit address of amd_queue_t 4289 (enable_sgpr_queue_ptr) object for AQL queue on which 4290 the dispatch packet was 4291 queued. 4292 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4293 (enable_sgpr_kernarg segment. This is directly 4294 _segment_ptr) copied from the 4295 kernarg_address in the kernel 4296 dispatch packet. 4297 4298 Having CP load it once avoids 4299 loading it at the beginning of 4300 every wavefront. 4301 then Dispatch Id 2 64-bit Dispatch ID of the 4302 (enable_sgpr_dispatch_id) dispatch packet being 4303 executed. 4304 then Flat Scratch Init 2 See 4305 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4306 _init) 4307 then Private Segment Size 1 The 32-bit byte size of a 4308 (enable_sgpr_private single work-item's memory 4309 _segment_size) allocation. This is the 4310 value from the kernel 4311 dispatch packet Private 4312 Segment Byte Size rounded up 4313 by CP to a multiple of 4314 DWORD. 4315 4316 Having CP load it once avoids 4317 loading it at the beginning of 4318 every wavefront. 4319 4320 This is not used for 4321 GFX7-GFX8 since it is the same 4322 value as the second SGPR of 4323 Flat Scratch Init. However, it 4324 may be needed for GFX9-GFX10 which 4325 changes the meaning of the 4326 Flat Scratch Init value. 4327 then Work-Group Id X 1 32-bit work-group id in X 4328 (enable_sgpr_workgroup_id dimension of grid for 4329 _X) wavefront. 4330 then Work-Group Id Y 1 32-bit work-group id in Y 4331 (enable_sgpr_workgroup_id dimension of grid for 4332 _Y) wavefront. 4333 then Work-Group Id Z 1 32-bit work-group id in Z 4334 (enable_sgpr_workgroup_id dimension of grid for 4335 _Z) wavefront. 4336 then Work-Group Info 1 {first_wavefront, 14'b0000, 4337 (enable_sgpr_workgroup ordered_append_term[10:0], 4338 _info) threadgroup_size_in_wavefronts[5:0]} 4339 then Scratch Wavefront Offset 1 See 4340 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4341 _segment_wavefront_offset) and 4342 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4343 ========== ========================== ====== ============================== 4344 4345The order of the VGPR registers is defined, but the compiler can specify which 4346ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4347fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4348for enabled registers are dense starting at VGPR0: the first enabled register is 4349VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4350VGPR number. 4351 4352There are different methods used for the VGPR initial state: 4353 4354* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4355 specifies otherwise, a separate VGPR register is used per work-item ID. The 4356 VGPR register initial state for this method is defined in 4357 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4358* If *Target Properties* column of :ref:`amdgpu-processor-table` 4359 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4360 for all work-item IDs. The register layout for this method is defined in 4361 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4362 4363 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4364 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4365 4366 ========== ========================== ====== ============================== 4367 VGPR Order Name Number Description 4368 (kernel descriptor enable of 4369 field) VGPRs 4370 ========== ========================== ====== ============================== 4371 First Work-Item Id X 1 32-bit work-item id in X 4372 (Always initialized) dimension of work-group for 4373 wavefront lane. 4374 then Work-Item Id Y 1 32-bit work-item id in Y 4375 (enable_vgpr_workitem_id dimension of work-group for 4376 > 0) wavefront lane. 4377 then Work-Item Id Z 1 32-bit work-item id in Z 4378 (enable_vgpr_workitem_id dimension of work-group for 4379 > 1) wavefront lane. 4380 ========== ========================== ====== ============================== 4381 4382.. 4383 4384 .. table:: Register Layout for Packed Work-Item ID Method 4385 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4386 4387 ======= ======= ================ ========================================= 4388 Bits Size Field Name Description 4389 ======= ======= ================ ========================================= 4390 0:9 10 bits Work-Item Id X Work-item id in X 4391 dimension of work-group for 4392 wavefront lane. 4393 4394 Always initialized. 4395 4396 10:19 10 bits Work-Item Id Y Work-item id in Y 4397 dimension of work-group for 4398 wavefront lane. 4399 4400 Initialized if enable_vgpr_workitem_id > 4401 0, otherwise set to 0. 4402 20:29 10 bits Work-Item Id Z Work-item id in Z 4403 dimension of work-group for 4404 wavefront lane. 4405 4406 Initialized if enable_vgpr_workitem_id > 4407 1, otherwise set to 0. 4408 30:31 2 bits Reserved, set to 0. 4409 ======= ======= ================ ========================================= 4410 4411The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4412 44131. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4414 registers. 44152. Work-group Id registers X, Y, Z are set by ADC which supports any 4416 combination including none. 44173. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4418 its value cannot be included with the flat scratch init value which is per 4419 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 44204. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4421 or (X, Y, Z). 44225. Flat Scratch register pair initialization is described in 4423 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4424 4425The global segment can be accessed either using buffer instructions (GFX6 which 4426has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4427instructions (GFX9-GFX10). 4428 4429If buffer operations are used, then the compiler can generate a V# with the 4430following properties: 4431 4432* base address of 0 4433* no swizzle 4434* ATC: 1 if IOMMU present (such as APU) 4435* ptr64: 1 4436* MTYPE set to support memory coherence that matches the runtime (such as CC for 4437 APU and NC for dGPU). 4438 4439.. _amdgpu-amdhsa-kernel-prolog: 4440 4441Kernel Prolog 4442~~~~~~~~~~~~~ 4443 4444The compiler performs initialization in the kernel prologue depending on the 4445target and information about things like stack usage in the kernel and called 4446functions. Some of this initialization requires the compiler to request certain 4447User and System SGPRs be present in the 4448:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4449:ref:`amdgpu-amdhsa-kernel-descriptor`. 4450 4451.. _amdgpu-amdhsa-kernel-prolog-cfi: 4452 4453CFI 4454+++ 4455 44561. The CFI return address is undefined. 4457 44582. The CFI CFA is defined using an expression which evaluates to a location 4459 description that comprises one memory location description for the 4460 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4461 4462.. _amdgpu-amdhsa-kernel-prolog-m0: 4463 4464M0 4465++ 4466 4467GFX6-GFX8 4468 The M0 register must be initialized with a value at least the total LDS size 4469 if the kernel may access LDS via DS or flat operations. Total LDS size is 4470 available in dispatch packet. For M0, it is also possible to use maximum 4471 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4472 GFX7-GFX8). 4473GFX9-GFX10 4474 The M0 register is not used for range checking LDS accesses and so does not 4475 need to be initialized in the prolog. 4476 4477.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4478 4479Stack Pointer 4480+++++++++++++ 4481 4482If the kernel has function calls it must set up the ABI stack pointer described 4483in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4484SGPR32 to the unswizzled scratch offset of the address past the last local 4485allocation. 4486 4487.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4488 4489Frame Pointer 4490+++++++++++++ 4491 4492If the kernel needs a frame pointer for the reasons defined in 4493``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4494kernel prolog. If a frame pointer is not required then all uses of the frame 4495pointer are replaced with immediate ``0`` offsets. 4496 4497.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4498 4499Flat Scratch 4500++++++++++++ 4501 4502There are different methods used for initializing flat scratch: 4503 4504* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4505 specifies *Does not support generic address space*: 4506 4507 Flat scratch is not supported and there is no flat scratch register pair. 4508 4509* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4510 specifies *Offset flat scratch*: 4511 4512 If the kernel or any function it calls may use flat operations to access 4513 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4514 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4515 Scratch Wavefront Offset SGPR registers (see 4516 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4517 4518 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4519 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4520 being managed by SPI for the queue executing the kernel dispatch. This is 4521 the same value used in the Scratch Segment Buffer V# base address. 4522 4523 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4524 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4525 4526 The prolog must add the value of Scratch Wavefront Offset to get the 4527 wavefront's byte scratch backing memory offset from 4528 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4529 4530 The Scratch Wavefront Offset must also be used as an offset with Private 4531 segment address when using the Scratch Segment Buffer. 4532 4533 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4534 shifted by 8 before moving into FLAT_SCRATCH_HI. 4535 4536 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4537 SGPRn is the highest numbered SGPR allocated to the wavefront). 4538 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4539 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4540 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4541 aperture. 4542 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4543 work-items scratch memory usage. 4544 4545 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4546 checks that the value in the kernel dispatch packet Private Segment Byte 4547 Size is not larger and requests the runtime to increase the queue's scratch 4548 size if necessary. 4549 4550 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4551 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4552 loading it at the beginning of every wavefront. 4553 4554 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4555 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4556 in flat memory instructions. 4557 4558* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4559 specifies *Absolute flat scratch*: 4560 4561 If the kernel or any function it calls may use flat operations to access 4562 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4563 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4564 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4565 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4566 4567 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4568 memory being managed by SPI for the queue executing the kernel dispatch. 4569 4570 CP obtains this from the runtime. 4571 4572 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4573 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4574 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4575 memory instructions. 4576 4577 The Scratch Wavefront Offset must also be used as an offset with Private 4578 segment address when using the Scratch Segment Buffer (see 4579 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4580 4581* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4582 specifies *Architected flat scratch*: 4583 4584 If ENABLE_PRIVATE_SEGMENT is enabled in 4585 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4586 register pair will be initialized to the 64-bit address of the base of scratch 4587 backing memory being managed by SPI for the queue executing the kernel 4588 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4589 flat scratch base in flat memory instructions. 4590 4591.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4592 4593Private Segment Buffer 4594++++++++++++++++++++++ 4595 4596If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4597*Architected flat scratch* then a Private Segment Buffer is not supported. 4598Instead the flat SCRATCH instructions are used. 4599 4600Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4601that are used as a V# to access scratch. CP uses the value provided by the 4602runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4603access the private memory space using a segment address. See 4604:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4605 4606The scratch V# is a four-aligned SGPR and always selected for the kernel as 4607follows: 4608 4609 - If it is known during instruction selection that there is stack usage, 4610 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4611 optimizations are disabled (``-O0``), if stack objects already exist (for 4612 locals, etc.), or if there are any function calls. 4613 4614 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4615 are reserved for the tentative scratch V#. These will be used if it is 4616 determined that spilling is needed. 4617 4618 - If no use is made of the tentative scratch V#, then it is unreserved, 4619 and the register count is determined ignoring it. 4620 - If use is made of the tentative scratch V#, then its register numbers 4621 are shifted to the first four-aligned SGPR index after the highest one 4622 allocated by the register allocator, and all uses are updated. The 4623 register count includes them in the shifted location. 4624 - In either case, if the processor has the SGPR allocation bug, the 4625 tentative allocation is not shifted or unreserved in order to ensure 4626 the register count is higher to workaround the bug. 4627 4628 .. note:: 4629 4630 This approach of using a tentative scratch V# and shifting the register 4631 numbers if used avoids having to perform register allocation a second 4632 time if the tentative V# is eliminated. This is more efficient and 4633 avoids the problem that the second register allocation may perform 4634 spilling which will fail as there is no longer a scratch V#. 4635 4636When the kernel prolog code is being emitted it is known whether the scratch V# 4637described above is actually used. If it is, the prolog code must set it up by 4638copying the Private Segment Buffer to the scratch V# registers and then adding 4639the Private Segment Wavefront Offset to the queue base address in the V#. The 4640result is a V# with a base address pointing to the beginning of the wavefront 4641scratch backing memory. 4642 4643The Private Segment Buffer is always requested, but the Private Segment 4644Wavefront Offset is only requested if it is used (see 4645:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4646 4647.. _amdgpu-amdhsa-memory-model: 4648 4649Memory Model 4650~~~~~~~~~~~~ 4651 4652This section describes the mapping of the LLVM memory model onto AMDGPU machine 4653code (see :ref:`memmodel`). 4654 4655The AMDGPU backend supports the memory synchronization scopes specified in 4656:ref:`amdgpu-memory-scopes`. 4657 4658The code sequences used to implement the memory model specify the order of 4659instructions that a single thread must execute. The ``s_waitcnt`` and cache 4660management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4661to other memory instructions executed by the same thread. This allows them to be 4662moved earlier or later which can allow them to be combined with other instances 4663of the same instruction, or hoisted/sunk out of loops to improve performance. 4664Only the instructions related to the memory model are given; additional 4665``s_waitcnt`` instructions are required to ensure registers are defined before 4666being used. These may be able to be combined with the memory model ``s_waitcnt`` 4667instructions as described above. 4668 4669The AMDGPU backend supports the following memory models: 4670 4671 HSA Memory Model [HSA]_ 4672 The HSA memory model uses a single happens-before relation for all address 4673 spaces (see :ref:`amdgpu-address-spaces`). 4674 OpenCL Memory Model [OpenCL]_ 4675 The OpenCL memory model which has separate happens-before relations for the 4676 global and local address spaces. Only a fence specifying both global and 4677 local address space, and seq_cst instructions join the relationships. Since 4678 the LLVM ``memfence`` instruction does not allow an address space to be 4679 specified the OpenCL fence has to conservatively assume both local and 4680 global address space was specified. However, optimizations can often be 4681 done to eliminate the additional ``s_waitcnt`` instructions when there are 4682 no intervening memory instructions which access the corresponding address 4683 space. The code sequences in the table indicate what can be omitted for the 4684 OpenCL memory. The target triple environment is used to determine if the 4685 source language is OpenCL (see :ref:`amdgpu-opencl`). 4686 4687``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4688operations. 4689 4690``buffer/global/flat_load/store/atomic`` instructions to global memory are 4691termed vector memory operations. 4692 4693Private address space uses ``buffer_load/store`` using the scratch V# 4694(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4695is accessing the memory, atomic memory orderings are not meaningful, and all 4696accesses are treated as non-atomic. 4697 4698Constant address space uses ``buffer/global_load`` instructions (or equivalent 4699scalar memory instructions). Since the constant address space contents do not 4700change during the execution of a kernel dispatch it is not legal to perform 4701stores, and atomic memory orderings are not meaningful, and all accesses are 4702treated as non-atomic. 4703 4704A memory synchronization scope wider than work-group is not meaningful for the 4705group (LDS) address space and is treated as work-group. 4706 4707The memory model does not support the region address space which is treated as 4708non-atomic. 4709 4710Acquire memory ordering is not meaningful on store atomic instructions and is 4711treated as non-atomic. 4712 4713Release memory ordering is not meaningful on load atomic instructions and is 4714treated a non-atomic. 4715 4716Acquire-release memory ordering is not meaningful on load or store atomic 4717instructions and is treated as acquire and release respectively. 4718 4719The memory order also adds the single thread optimization constraints defined in 4720table 4721:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4722 4723 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4724 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4725 4726 ============ ============================================================== 4727 LLVM Memory Optimization Constraints 4728 Ordering 4729 ============ ============================================================== 4730 unordered *none* 4731 monotonic *none* 4732 acquire - If a load atomic/atomicrmw then no following load/load 4733 atomic/store/store atomic/atomicrmw/fence instruction can be 4734 moved before the acquire. 4735 - If a fence then same as load atomic, plus no preceding 4736 associated fence-paired-atomic can be moved after the fence. 4737 release - If a store atomic/atomicrmw then no preceding load/load 4738 atomic/store/store atomic/atomicrmw/fence instruction can be 4739 moved after the release. 4740 - If a fence then same as store atomic, plus no following 4741 associated fence-paired-atomic can be moved before the 4742 fence. 4743 acq_rel Same constraints as both acquire and release. 4744 seq_cst - If a load atomic then same constraints as acquire, plus no 4745 preceding sequentially consistent load atomic/store 4746 atomic/atomicrmw/fence instruction can be moved after the 4747 seq_cst. 4748 - If a store atomic then the same constraints as release, plus 4749 no following sequentially consistent load atomic/store 4750 atomic/atomicrmw/fence instruction can be moved before the 4751 seq_cst. 4752 - If an atomicrmw/fence then same constraints as acq_rel. 4753 ============ ============================================================== 4754 4755The code sequences used to implement the memory model are defined in the 4756following sections: 4757 4758* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4759* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4760* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4761 4762.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4763 4764Memory Model GFX6-GFX9 4765++++++++++++++++++++++ 4766 4767For GFX6-GFX9: 4768 4769* Each agent has multiple shader arrays (SA). 4770* Each SA has multiple compute units (CU). 4771* Each CU has multiple SIMDs that execute wavefronts. 4772* The wavefronts for a single work-group are executed in the same CU but may be 4773 executed by different SIMDs. 4774* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4775 executing on it. 4776* All LDS operations of a CU are performed as wavefront wide operations in a 4777 global order and involve no caching. Completion is reported to a wavefront in 4778 execution order. 4779* The LDS memory has multiple request queues shared by the SIMDs of a 4780 CU. Therefore, the LDS operations performed by different wavefronts of a 4781 work-group can be reordered relative to each other, which can result in 4782 reordering the visibility of vector memory operations with respect to LDS 4783 operations of other wavefronts in the same work-group. A ``s_waitcnt 4784 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4785 vector memory operations between wavefronts of a work-group, but not between 4786 operations performed by the same wavefront. 4787* The vector memory operations are performed as wavefront wide operations and 4788 completion is reported to a wavefront in execution order. The exception is 4789 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4790 vector memory order if they access LDS memory, and out of LDS operation order 4791 if they access global memory. 4792* The vector memory operations access a single vector L1 cache shared by all 4793 SIMDs a CU. Therefore, no special action is required for coherence between the 4794 lanes of a single wavefront, or for coherence between wavefronts in the same 4795 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4796 wavefronts executing in different work-groups as they may be executing on 4797 different CUs. 4798* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4799 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4800 scalar operations are used in a restricted way so do not impact the memory 4801 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4802* The vector and scalar memory operations use an L2 cache shared by all CUs on 4803 the same agent. 4804* The L2 cache has independent channels to service disjoint ranges of virtual 4805 addresses. 4806* Each CU has a separate request queue per channel. Therefore, the vector and 4807 scalar memory operations performed by wavefronts executing in different 4808 work-groups (which may be executing on different CUs) of an agent can be 4809 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4810 ensure synchronization between vector memory operations of different CUs. It 4811 ensures a previous vector memory operation has completed before executing a 4812 subsequent vector memory or LDS operation and so can be used to meet the 4813 requirements of acquire and release. 4814* The L2 cache can be kept coherent with other agents on some targets, or ranges 4815 of virtual addresses can be set up to bypass it to ensure system coherence. 4816 4817Scalar memory operations are only used to access memory that is proven to not 4818change during the execution of the kernel dispatch. This includes constant 4819address space and global address space for program scope ``const`` variables. 4820Therefore, the kernel machine code does not have to maintain the scalar cache to 4821ensure it is coherent with the vector caches. The scalar and vector caches are 4822invalidated between kernel dispatches by CP since constant address space data 4823may change between kernel dispatch executions. See 4824:ref:`amdgpu-amdhsa-memory-spaces`. 4825 4826The one exception is if scalar writes are used to spill SGPR registers. In this 4827case the AMDGPU backend ensures the memory location used to spill is never 4828accessed by vector memory operations at the same time. If scalar writes are used 4829then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4830return since the locations may be used for vector memory instructions by a 4831future wavefront that uses the same scratch area, or a function call that 4832creates a frame at the same address, respectively. There is no need for a 4833``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4834 4835For kernarg backing memory: 4836 4837* CP invalidates the L1 cache at the start of each kernel dispatch. 4838* On dGPU the kernarg backing memory is allocated in host memory accessed as 4839 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4840 causes it to be treated as non-volatile and so is not invalidated by 4841 ``*_vol``. 4842* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4843 and so the L2 cache will be coherent with the CPU and other agents. 4844 4845Scratch backing memory (which is used for the private address space) is accessed 4846with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4847only accessed by a single thread, and is always write-before-read, there is 4848never a need to invalidate these entries from the L1 cache. Hence all cache 4849invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4850 4851The code sequences used to implement the memory model for GFX6-GFX9 are defined 4852in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4853 4854 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4855 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4856 4857 ============ ============ ============== ========== ================================ 4858 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4859 Ordering Sync Scope Address GFX6-GFX9 4860 Space 4861 ============ ============ ============== ========== ================================ 4862 **Non-Atomic** 4863 ------------------------------------------------------------------------------------ 4864 load *none* *none* - global - !volatile & !nontemporal 4865 - generic 4866 - private 1. buffer/global/flat_load 4867 - constant 4868 - !volatile & nontemporal 4869 4870 1. buffer/global/flat_load 4871 glc=1 slc=1 4872 4873 - volatile 4874 4875 1. buffer/global/flat_load 4876 glc=1 4877 2. s_waitcnt vmcnt(0) 4878 4879 - Must happen before 4880 any following volatile 4881 global/generic 4882 load/store. 4883 - Ensures that 4884 volatile 4885 operations to 4886 different 4887 addresses will not 4888 be reordered by 4889 hardware. 4890 4891 load *none* *none* - local 1. ds_load 4892 store *none* *none* - global - !volatile & !nontemporal 4893 - generic 4894 - private 1. buffer/global/flat_store 4895 - constant 4896 - !volatile & nontemporal 4897 4898 1. buffer/global/flat_store 4899 glc=1 slc=1 4900 4901 - volatile 4902 4903 1. buffer/global/flat_store 4904 2. s_waitcnt vmcnt(0) 4905 4906 - Must happen before 4907 any following volatile 4908 global/generic 4909 load/store. 4910 - Ensures that 4911 volatile 4912 operations to 4913 different 4914 addresses will not 4915 be reordered by 4916 hardware. 4917 4918 store *none* *none* - local 1. ds_store 4919 **Unordered Atomic** 4920 ------------------------------------------------------------------------------------ 4921 load atomic unordered *any* *any* *Same as non-atomic*. 4922 store atomic unordered *any* *any* *Same as non-atomic*. 4923 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4924 **Monotonic Atomic** 4925 ------------------------------------------------------------------------------------ 4926 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4927 - wavefront - local 4928 - workgroup - generic 4929 load atomic monotonic - agent - global 1. buffer/global/flat_load 4930 - system - generic glc=1 4931 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4932 - wavefront - generic 4933 - workgroup 4934 - agent 4935 - system 4936 store atomic monotonic - singlethread - local 1. ds_store 4937 - wavefront 4938 - workgroup 4939 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4940 - wavefront - generic 4941 - workgroup 4942 - agent 4943 - system 4944 atomicrmw monotonic - singlethread - local 1. ds_atomic 4945 - wavefront 4946 - workgroup 4947 **Acquire Atomic** 4948 ------------------------------------------------------------------------------------ 4949 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4950 - wavefront - local 4951 - generic 4952 load atomic acquire - workgroup - global 1. buffer/global_load 4953 load atomic acquire - workgroup - local 1. ds/flat_load 4954 - generic 2. s_waitcnt lgkmcnt(0) 4955 4956 - If OpenCL, omit. 4957 - Must happen before 4958 any following 4959 global/generic 4960 load/load 4961 atomic/store/store 4962 atomic/atomicrmw. 4963 - Ensures any 4964 following global 4965 data read is no 4966 older than a local load 4967 atomic value being 4968 acquired. 4969 4970 load atomic acquire - agent - global 1. buffer/global_load 4971 - system glc=1 4972 2. s_waitcnt vmcnt(0) 4973 4974 - Must happen before 4975 following 4976 buffer_wbinvl1_vol. 4977 - Ensures the load 4978 has completed 4979 before invalidating 4980 the cache. 4981 4982 3. buffer_wbinvl1_vol 4983 4984 - Must happen before 4985 any following 4986 global/generic 4987 load/load 4988 atomic/atomicrmw. 4989 - Ensures that 4990 following 4991 loads will not see 4992 stale global data. 4993 4994 load atomic acquire - agent - generic 1. flat_load glc=1 4995 - system 2. s_waitcnt vmcnt(0) & 4996 lgkmcnt(0) 4997 4998 - If OpenCL omit 4999 lgkmcnt(0). 5000 - Must happen before 5001 following 5002 buffer_wbinvl1_vol. 5003 - Ensures the flat_load 5004 has completed 5005 before invalidating 5006 the cache. 5007 5008 3. buffer_wbinvl1_vol 5009 5010 - Must happen before 5011 any following 5012 global/generic 5013 load/load 5014 atomic/atomicrmw. 5015 - Ensures that 5016 following loads 5017 will not see stale 5018 global data. 5019 5020 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5021 - wavefront - local 5022 - generic 5023 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5024 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5025 - generic 2. s_waitcnt lgkmcnt(0) 5026 5027 - If OpenCL, omit. 5028 - Must happen before 5029 any following 5030 global/generic 5031 load/load 5032 atomic/store/store 5033 atomic/atomicrmw. 5034 - Ensures any 5035 following global 5036 data read is no 5037 older than a local 5038 atomicrmw value 5039 being acquired. 5040 5041 atomicrmw acquire - agent - global 1. buffer/global_atomic 5042 - system 2. s_waitcnt vmcnt(0) 5043 5044 - Must happen before 5045 following 5046 buffer_wbinvl1_vol. 5047 - Ensures the 5048 atomicrmw has 5049 completed before 5050 invalidating the 5051 cache. 5052 5053 3. buffer_wbinvl1_vol 5054 5055 - Must happen before 5056 any following 5057 global/generic 5058 load/load 5059 atomic/atomicrmw. 5060 - Ensures that 5061 following loads 5062 will not see stale 5063 global data. 5064 5065 atomicrmw acquire - agent - generic 1. flat_atomic 5066 - system 2. s_waitcnt vmcnt(0) & 5067 lgkmcnt(0) 5068 5069 - If OpenCL, omit 5070 lgkmcnt(0). 5071 - Must happen before 5072 following 5073 buffer_wbinvl1_vol. 5074 - Ensures the 5075 atomicrmw has 5076 completed before 5077 invalidating the 5078 cache. 5079 5080 3. buffer_wbinvl1_vol 5081 5082 - Must happen before 5083 any following 5084 global/generic 5085 load/load 5086 atomic/atomicrmw. 5087 - Ensures that 5088 following loads 5089 will not see stale 5090 global data. 5091 5092 fence acquire - singlethread *none* *none* 5093 - wavefront 5094 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5095 5096 - If OpenCL and 5097 address space is 5098 not generic, omit. 5099 - However, since LLVM 5100 currently has no 5101 address space on 5102 the fence need to 5103 conservatively 5104 always generate. If 5105 fence had an 5106 address space then 5107 set to address 5108 space of OpenCL 5109 fence flag, or to 5110 generic if both 5111 local and global 5112 flags are 5113 specified. 5114 - Must happen after 5115 any preceding 5116 local/generic load 5117 atomic/atomicrmw 5118 with an equal or 5119 wider sync scope 5120 and memory ordering 5121 stronger than 5122 unordered (this is 5123 termed the 5124 fence-paired-atomic). 5125 - Must happen before 5126 any following 5127 global/generic 5128 load/load 5129 atomic/store/store 5130 atomic/atomicrmw. 5131 - Ensures any 5132 following global 5133 data read is no 5134 older than the 5135 value read by the 5136 fence-paired-atomic. 5137 5138 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5139 - system vmcnt(0) 5140 5141 - If OpenCL and 5142 address space is 5143 not generic, omit 5144 lgkmcnt(0). 5145 - However, since LLVM 5146 currently has no 5147 address space on 5148 the fence need to 5149 conservatively 5150 always generate 5151 (see comment for 5152 previous fence). 5153 - Could be split into 5154 separate s_waitcnt 5155 vmcnt(0) and 5156 s_waitcnt 5157 lgkmcnt(0) to allow 5158 them to be 5159 independently moved 5160 according to the 5161 following rules. 5162 - s_waitcnt vmcnt(0) 5163 must happen after 5164 any preceding 5165 global/generic load 5166 atomic/atomicrmw 5167 with an equal or 5168 wider sync scope 5169 and memory ordering 5170 stronger than 5171 unordered (this is 5172 termed the 5173 fence-paired-atomic). 5174 - s_waitcnt lgkmcnt(0) 5175 must happen after 5176 any preceding 5177 local/generic load 5178 atomic/atomicrmw 5179 with an equal or 5180 wider sync scope 5181 and memory ordering 5182 stronger than 5183 unordered (this is 5184 termed the 5185 fence-paired-atomic). 5186 - Must happen before 5187 the following 5188 buffer_wbinvl1_vol. 5189 - Ensures that the 5190 fence-paired atomic 5191 has completed 5192 before invalidating 5193 the 5194 cache. Therefore 5195 any following 5196 locations read must 5197 be no older than 5198 the value read by 5199 the 5200 fence-paired-atomic. 5201 5202 2. buffer_wbinvl1_vol 5203 5204 - Must happen before any 5205 following global/generic 5206 load/load 5207 atomic/store/store 5208 atomic/atomicrmw. 5209 - Ensures that 5210 following loads 5211 will not see stale 5212 global data. 5213 5214 **Release Atomic** 5215 ------------------------------------------------------------------------------------ 5216 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5217 - wavefront - local 5218 - generic 5219 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5220 - generic 5221 - If OpenCL, omit. 5222 - Must happen after 5223 any preceding 5224 local/generic 5225 load/store/load 5226 atomic/store 5227 atomic/atomicrmw. 5228 - Must happen before 5229 the following 5230 store. 5231 - Ensures that all 5232 memory operations 5233 to local have 5234 completed before 5235 performing the 5236 store that is being 5237 released. 5238 5239 2. buffer/global/flat_store 5240 store atomic release - workgroup - local 1. ds_store 5241 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5242 - system - generic vmcnt(0) 5243 5244 - If OpenCL and 5245 address space is 5246 not generic, omit 5247 lgkmcnt(0). 5248 - Could be split into 5249 separate s_waitcnt 5250 vmcnt(0) and 5251 s_waitcnt 5252 lgkmcnt(0) to allow 5253 them to be 5254 independently moved 5255 according to the 5256 following rules. 5257 - s_waitcnt vmcnt(0) 5258 must happen after 5259 any preceding 5260 global/generic 5261 load/store/load 5262 atomic/store 5263 atomic/atomicrmw. 5264 - s_waitcnt lgkmcnt(0) 5265 must happen after 5266 any preceding 5267 local/generic 5268 load/store/load 5269 atomic/store 5270 atomic/atomicrmw. 5271 - Must happen before 5272 the following 5273 store. 5274 - Ensures that all 5275 memory operations 5276 to memory have 5277 completed before 5278 performing the 5279 store that is being 5280 released. 5281 5282 2. buffer/global/flat_store 5283 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5284 - wavefront - local 5285 - generic 5286 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5287 - generic 5288 - If OpenCL, omit. 5289 - Must happen after 5290 any preceding 5291 local/generic 5292 load/store/load 5293 atomic/store 5294 atomic/atomicrmw. 5295 - Must happen before 5296 the following 5297 atomicrmw. 5298 - Ensures that all 5299 memory operations 5300 to local have 5301 completed before 5302 performing the 5303 atomicrmw that is 5304 being released. 5305 5306 2. buffer/global/flat_atomic 5307 atomicrmw release - workgroup - local 1. ds_atomic 5308 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5309 - system - generic vmcnt(0) 5310 5311 - If OpenCL, omit 5312 lgkmcnt(0). 5313 - Could be split into 5314 separate s_waitcnt 5315 vmcnt(0) and 5316 s_waitcnt 5317 lgkmcnt(0) to allow 5318 them to be 5319 independently moved 5320 according to the 5321 following rules. 5322 - s_waitcnt vmcnt(0) 5323 must happen after 5324 any preceding 5325 global/generic 5326 load/store/load 5327 atomic/store 5328 atomic/atomicrmw. 5329 - s_waitcnt lgkmcnt(0) 5330 must happen after 5331 any preceding 5332 local/generic 5333 load/store/load 5334 atomic/store 5335 atomic/atomicrmw. 5336 - Must happen before 5337 the following 5338 atomicrmw. 5339 - Ensures that all 5340 memory operations 5341 to global and local 5342 have completed 5343 before performing 5344 the atomicrmw that 5345 is being released. 5346 5347 2. buffer/global/flat_atomic 5348 fence release - singlethread *none* *none* 5349 - wavefront 5350 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5351 5352 - If OpenCL and 5353 address space is 5354 not generic, omit. 5355 - However, since LLVM 5356 currently has no 5357 address space on 5358 the fence need to 5359 conservatively 5360 always generate. If 5361 fence had an 5362 address space then 5363 set to address 5364 space of OpenCL 5365 fence flag, or to 5366 generic if both 5367 local and global 5368 flags are 5369 specified. 5370 - Must happen after 5371 any preceding 5372 local/generic 5373 load/load 5374 atomic/store/store 5375 atomic/atomicrmw. 5376 - Must happen before 5377 any following store 5378 atomic/atomicrmw 5379 with an equal or 5380 wider sync scope 5381 and memory ordering 5382 stronger than 5383 unordered (this is 5384 termed the 5385 fence-paired-atomic). 5386 - Ensures that all 5387 memory operations 5388 to local have 5389 completed before 5390 performing the 5391 following 5392 fence-paired-atomic. 5393 5394 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5395 - system vmcnt(0) 5396 5397 - If OpenCL and 5398 address space is 5399 not generic, omit 5400 lgkmcnt(0). 5401 - If OpenCL and 5402 address space is 5403 local, omit 5404 vmcnt(0). 5405 - However, since LLVM 5406 currently has no 5407 address space on 5408 the fence need to 5409 conservatively 5410 always generate. If 5411 fence had an 5412 address space then 5413 set to address 5414 space of OpenCL 5415 fence flag, or to 5416 generic if both 5417 local and global 5418 flags are 5419 specified. 5420 - Could be split into 5421 separate s_waitcnt 5422 vmcnt(0) and 5423 s_waitcnt 5424 lgkmcnt(0) to allow 5425 them to be 5426 independently moved 5427 according to the 5428 following rules. 5429 - s_waitcnt vmcnt(0) 5430 must happen after 5431 any preceding 5432 global/generic 5433 load/store/load 5434 atomic/store 5435 atomic/atomicrmw. 5436 - s_waitcnt lgkmcnt(0) 5437 must happen after 5438 any preceding 5439 local/generic 5440 load/store/load 5441 atomic/store 5442 atomic/atomicrmw. 5443 - Must happen before 5444 any following store 5445 atomic/atomicrmw 5446 with an equal or 5447 wider sync scope 5448 and memory ordering 5449 stronger than 5450 unordered (this is 5451 termed the 5452 fence-paired-atomic). 5453 - Ensures that all 5454 memory operations 5455 have 5456 completed before 5457 performing the 5458 following 5459 fence-paired-atomic. 5460 5461 **Acquire-Release Atomic** 5462 ------------------------------------------------------------------------------------ 5463 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5464 - wavefront - local 5465 - generic 5466 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5467 5468 - If OpenCL, omit. 5469 - Must happen after 5470 any preceding 5471 local/generic 5472 load/store/load 5473 atomic/store 5474 atomic/atomicrmw. 5475 - Must happen before 5476 the following 5477 atomicrmw. 5478 - Ensures that all 5479 memory operations 5480 to local have 5481 completed before 5482 performing the 5483 atomicrmw that is 5484 being released. 5485 5486 2. buffer/global_atomic 5487 5488 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5489 2. s_waitcnt lgkmcnt(0) 5490 5491 - If OpenCL, omit. 5492 - Must happen before 5493 any following 5494 global/generic 5495 load/load 5496 atomic/store/store 5497 atomic/atomicrmw. 5498 - Ensures any 5499 following global 5500 data read is no 5501 older than the local load 5502 atomic value being 5503 acquired. 5504 5505 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5506 5507 - If OpenCL, omit. 5508 - Must happen after 5509 any preceding 5510 local/generic 5511 load/store/load 5512 atomic/store 5513 atomic/atomicrmw. 5514 - Must happen before 5515 the following 5516 atomicrmw. 5517 - Ensures that all 5518 memory operations 5519 to local have 5520 completed before 5521 performing the 5522 atomicrmw that is 5523 being released. 5524 5525 2. flat_atomic 5526 3. s_waitcnt lgkmcnt(0) 5527 5528 - If OpenCL, omit. 5529 - Must happen before 5530 any following 5531 global/generic 5532 load/load 5533 atomic/store/store 5534 atomic/atomicrmw. 5535 - Ensures any 5536 following global 5537 data read is no 5538 older than a local load 5539 atomic value being 5540 acquired. 5541 5542 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5543 - system vmcnt(0) 5544 5545 - If OpenCL, omit 5546 lgkmcnt(0). 5547 - Could be split into 5548 separate s_waitcnt 5549 vmcnt(0) and 5550 s_waitcnt 5551 lgkmcnt(0) to allow 5552 them to be 5553 independently moved 5554 according to the 5555 following rules. 5556 - s_waitcnt vmcnt(0) 5557 must happen after 5558 any preceding 5559 global/generic 5560 load/store/load 5561 atomic/store 5562 atomic/atomicrmw. 5563 - s_waitcnt lgkmcnt(0) 5564 must happen after 5565 any preceding 5566 local/generic 5567 load/store/load 5568 atomic/store 5569 atomic/atomicrmw. 5570 - Must happen before 5571 the following 5572 atomicrmw. 5573 - Ensures that all 5574 memory operations 5575 to global have 5576 completed before 5577 performing the 5578 atomicrmw that is 5579 being released. 5580 5581 2. buffer/global_atomic 5582 3. s_waitcnt vmcnt(0) 5583 5584 - Must happen before 5585 following 5586 buffer_wbinvl1_vol. 5587 - Ensures the 5588 atomicrmw has 5589 completed before 5590 invalidating the 5591 cache. 5592 5593 4. buffer_wbinvl1_vol 5594 5595 - Must happen before 5596 any following 5597 global/generic 5598 load/load 5599 atomic/atomicrmw. 5600 - Ensures that 5601 following loads 5602 will not see stale 5603 global data. 5604 5605 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5606 - system vmcnt(0) 5607 5608 - If OpenCL, omit 5609 lgkmcnt(0). 5610 - Could be split into 5611 separate s_waitcnt 5612 vmcnt(0) and 5613 s_waitcnt 5614 lgkmcnt(0) to allow 5615 them to be 5616 independently moved 5617 according to the 5618 following rules. 5619 - s_waitcnt vmcnt(0) 5620 must happen after 5621 any preceding 5622 global/generic 5623 load/store/load 5624 atomic/store 5625 atomic/atomicrmw. 5626 - s_waitcnt lgkmcnt(0) 5627 must happen after 5628 any preceding 5629 local/generic 5630 load/store/load 5631 atomic/store 5632 atomic/atomicrmw. 5633 - Must happen before 5634 the following 5635 atomicrmw. 5636 - Ensures that all 5637 memory operations 5638 to global have 5639 completed before 5640 performing the 5641 atomicrmw that is 5642 being released. 5643 5644 2. flat_atomic 5645 3. s_waitcnt vmcnt(0) & 5646 lgkmcnt(0) 5647 5648 - If OpenCL, omit 5649 lgkmcnt(0). 5650 - Must happen before 5651 following 5652 buffer_wbinvl1_vol. 5653 - Ensures the 5654 atomicrmw has 5655 completed before 5656 invalidating the 5657 cache. 5658 5659 4. buffer_wbinvl1_vol 5660 5661 - Must happen before 5662 any following 5663 global/generic 5664 load/load 5665 atomic/atomicrmw. 5666 - Ensures that 5667 following loads 5668 will not see stale 5669 global data. 5670 5671 fence acq_rel - singlethread *none* *none* 5672 - wavefront 5673 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5674 5675 - If OpenCL and 5676 address space is 5677 not generic, omit. 5678 - However, 5679 since LLVM 5680 currently has no 5681 address space on 5682 the fence need to 5683 conservatively 5684 always generate 5685 (see comment for 5686 previous fence). 5687 - Must happen after 5688 any preceding 5689 local/generic 5690 load/load 5691 atomic/store/store 5692 atomic/atomicrmw. 5693 - Must happen before 5694 any following 5695 global/generic 5696 load/load 5697 atomic/store/store 5698 atomic/atomicrmw. 5699 - Ensures that all 5700 memory operations 5701 to local have 5702 completed before 5703 performing any 5704 following global 5705 memory operations. 5706 - Ensures that the 5707 preceding 5708 local/generic load 5709 atomic/atomicrmw 5710 with an equal or 5711 wider sync scope 5712 and memory ordering 5713 stronger than 5714 unordered (this is 5715 termed the 5716 acquire-fence-paired-atomic) 5717 has completed 5718 before following 5719 global memory 5720 operations. This 5721 satisfies the 5722 requirements of 5723 acquire. 5724 - Ensures that all 5725 previous memory 5726 operations have 5727 completed before a 5728 following 5729 local/generic store 5730 atomic/atomicrmw 5731 with an equal or 5732 wider sync scope 5733 and memory ordering 5734 stronger than 5735 unordered (this is 5736 termed the 5737 release-fence-paired-atomic). 5738 This satisfies the 5739 requirements of 5740 release. 5741 5742 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5743 - system vmcnt(0) 5744 5745 - If OpenCL and 5746 address space is 5747 not generic, omit 5748 lgkmcnt(0). 5749 - However, since LLVM 5750 currently has no 5751 address space on 5752 the fence need to 5753 conservatively 5754 always generate 5755 (see comment for 5756 previous fence). 5757 - Could be split into 5758 separate s_waitcnt 5759 vmcnt(0) and 5760 s_waitcnt 5761 lgkmcnt(0) to allow 5762 them to be 5763 independently moved 5764 according to the 5765 following rules. 5766 - s_waitcnt vmcnt(0) 5767 must happen after 5768 any preceding 5769 global/generic 5770 load/store/load 5771 atomic/store 5772 atomic/atomicrmw. 5773 - s_waitcnt lgkmcnt(0) 5774 must happen after 5775 any preceding 5776 local/generic 5777 load/store/load 5778 atomic/store 5779 atomic/atomicrmw. 5780 - Must happen before 5781 the following 5782 buffer_wbinvl1_vol. 5783 - Ensures that the 5784 preceding 5785 global/local/generic 5786 load 5787 atomic/atomicrmw 5788 with an equal or 5789 wider sync scope 5790 and memory ordering 5791 stronger than 5792 unordered (this is 5793 termed the 5794 acquire-fence-paired-atomic) 5795 has completed 5796 before invalidating 5797 the cache. This 5798 satisfies the 5799 requirements of 5800 acquire. 5801 - Ensures that all 5802 previous memory 5803 operations have 5804 completed before a 5805 following 5806 global/local/generic 5807 store 5808 atomic/atomicrmw 5809 with an equal or 5810 wider sync scope 5811 and memory ordering 5812 stronger than 5813 unordered (this is 5814 termed the 5815 release-fence-paired-atomic). 5816 This satisfies the 5817 requirements of 5818 release. 5819 5820 2. buffer_wbinvl1_vol 5821 5822 - Must happen before 5823 any following 5824 global/generic 5825 load/load 5826 atomic/store/store 5827 atomic/atomicrmw. 5828 - Ensures that 5829 following loads 5830 will not see stale 5831 global data. This 5832 satisfies the 5833 requirements of 5834 acquire. 5835 5836 **Sequential Consistent Atomic** 5837 ------------------------------------------------------------------------------------ 5838 load atomic seq_cst - singlethread - global *Same as corresponding 5839 - wavefront - local load atomic acquire, 5840 - generic except must generated 5841 all instructions even 5842 for OpenCL.* 5843 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5844 - generic 5845 5846 - Must 5847 happen after 5848 preceding 5849 local/generic load 5850 atomic/store 5851 atomic/atomicrmw 5852 with memory 5853 ordering of seq_cst 5854 and with equal or 5855 wider sync scope. 5856 (Note that seq_cst 5857 fences have their 5858 own s_waitcnt 5859 lgkmcnt(0) and so do 5860 not need to be 5861 considered.) 5862 - Ensures any 5863 preceding 5864 sequential 5865 consistent local 5866 memory instructions 5867 have completed 5868 before executing 5869 this sequentially 5870 consistent 5871 instruction. This 5872 prevents reordering 5873 a seq_cst store 5874 followed by a 5875 seq_cst load. (Note 5876 that seq_cst is 5877 stronger than 5878 acquire/release as 5879 the reordering of 5880 load acquire 5881 followed by a store 5882 release is 5883 prevented by the 5884 s_waitcnt of 5885 the release, but 5886 there is nothing 5887 preventing a store 5888 release followed by 5889 load acquire from 5890 completing out of 5891 order. The s_waitcnt 5892 could be placed after 5893 seq_store or before 5894 the seq_load. We 5895 choose the load to 5896 make the s_waitcnt be 5897 as late as possible 5898 so that the store 5899 may have already 5900 completed.) 5901 5902 2. *Following 5903 instructions same as 5904 corresponding load 5905 atomic acquire, 5906 except must generated 5907 all instructions even 5908 for OpenCL.* 5909 load atomic seq_cst - workgroup - local *Same as corresponding 5910 load atomic acquire, 5911 except must generated 5912 all instructions even 5913 for OpenCL.* 5914 5915 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5916 - system - generic vmcnt(0) 5917 5918 - Could be split into 5919 separate s_waitcnt 5920 vmcnt(0) 5921 and s_waitcnt 5922 lgkmcnt(0) to allow 5923 them to be 5924 independently moved 5925 according to the 5926 following rules. 5927 - s_waitcnt lgkmcnt(0) 5928 must happen after 5929 preceding 5930 global/generic load 5931 atomic/store 5932 atomic/atomicrmw 5933 with memory 5934 ordering of seq_cst 5935 and with equal or 5936 wider sync scope. 5937 (Note that seq_cst 5938 fences have their 5939 own s_waitcnt 5940 lgkmcnt(0) and so do 5941 not need to be 5942 considered.) 5943 - s_waitcnt vmcnt(0) 5944 must happen after 5945 preceding 5946 global/generic load 5947 atomic/store 5948 atomic/atomicrmw 5949 with memory 5950 ordering of seq_cst 5951 and with equal or 5952 wider sync scope. 5953 (Note that seq_cst 5954 fences have their 5955 own s_waitcnt 5956 vmcnt(0) and so do 5957 not need to be 5958 considered.) 5959 - Ensures any 5960 preceding 5961 sequential 5962 consistent global 5963 memory instructions 5964 have completed 5965 before executing 5966 this sequentially 5967 consistent 5968 instruction. This 5969 prevents reordering 5970 a seq_cst store 5971 followed by a 5972 seq_cst load. (Note 5973 that seq_cst is 5974 stronger than 5975 acquire/release as 5976 the reordering of 5977 load acquire 5978 followed by a store 5979 release is 5980 prevented by the 5981 s_waitcnt of 5982 the release, but 5983 there is nothing 5984 preventing a store 5985 release followed by 5986 load acquire from 5987 completing out of 5988 order. The s_waitcnt 5989 could be placed after 5990 seq_store or before 5991 the seq_load. We 5992 choose the load to 5993 make the s_waitcnt be 5994 as late as possible 5995 so that the store 5996 may have already 5997 completed.) 5998 5999 2. *Following 6000 instructions same as 6001 corresponding load 6002 atomic acquire, 6003 except must generated 6004 all instructions even 6005 for OpenCL.* 6006 store atomic seq_cst - singlethread - global *Same as corresponding 6007 - wavefront - local store atomic release, 6008 - workgroup - generic except must generated 6009 - agent all instructions even 6010 - system for OpenCL.* 6011 atomicrmw seq_cst - singlethread - global *Same as corresponding 6012 - wavefront - local atomicrmw acq_rel, 6013 - workgroup - generic except must generated 6014 - agent all instructions even 6015 - system for OpenCL.* 6016 fence seq_cst - singlethread *none* *Same as corresponding 6017 - wavefront fence acq_rel, 6018 - workgroup except must generated 6019 - agent all instructions even 6020 - system for OpenCL.* 6021 ============ ============ ============== ========== ================================ 6022 6023.. _amdgpu-amdhsa-memory-model-gfx90a: 6024 6025Memory Model GFX90A 6026+++++++++++++++++++ 6027 6028For GFX90A: 6029 6030* Each agent has multiple shader arrays (SA). 6031* Each SA has multiple compute units (CU). 6032* Each CU has multiple SIMDs that execute wavefronts. 6033* The wavefronts for a single work-group are executed in the same CU but may be 6034 executed by different SIMDs. The exception is when in tgsplit execution mode 6035 when the wavefronts may be executed by different SIMDs in different CUs. 6036* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6037 executing on it. The exception is when in tgsplit execution mode when no LDS 6038 is allocated as wavefronts of the same work-group can be in different CUs. 6039* All LDS operations of a CU are performed as wavefront wide operations in a 6040 global order and involve no caching. Completion is reported to a wavefront in 6041 execution order. 6042* The LDS memory has multiple request queues shared by the SIMDs of a 6043 CU. Therefore, the LDS operations performed by different wavefronts of a 6044 work-group can be reordered relative to each other, which can result in 6045 reordering the visibility of vector memory operations with respect to LDS 6046 operations of other wavefronts in the same work-group. A ``s_waitcnt 6047 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6048 vector memory operations between wavefronts of a work-group, but not between 6049 operations performed by the same wavefront. 6050* The vector memory operations are performed as wavefront wide operations and 6051 completion is reported to a wavefront in execution order. The exception is 6052 that ``flat_load/store/atomic`` instructions can report out of vector memory 6053 order if they access LDS memory, and out of LDS operation order if they access 6054 global memory. 6055* The vector memory operations access a single vector L1 cache shared by all 6056 SIMDs a CU. Therefore: 6057 6058 * No special action is required for coherence between the lanes of a single 6059 wavefront. 6060 6061 * No special action is required for coherence between wavefronts in the same 6062 work-group since they execute on the same CU. The exception is when in 6063 tgsplit execution mode as wavefronts of the same work-group can be in 6064 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6065 the following item. 6066 6067 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6068 executing in different work-groups as they may be executing on different 6069 CUs. 6070 6071* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6072 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6073 scalar operations are used in a restricted way so do not impact the memory 6074 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6075* The vector and scalar memory operations use an L2 cache shared by all CUs on 6076 the same agent. 6077 6078 * The L2 cache has independent channels to service disjoint ranges of virtual 6079 addresses. 6080 * Each CU has a separate request queue per channel. Therefore, the vector and 6081 scalar memory operations performed by wavefronts executing in different 6082 work-groups (which may be executing on different CUs), or the same 6083 work-group if executing in tgsplit mode, of an agent can be reordered 6084 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6085 synchronization between vector memory operations of different CUs. It 6086 ensures a previous vector memory operation has completed before executing a 6087 subsequent vector memory or LDS operation and so can be used to meet the 6088 requirements of acquire and release. 6089 * The L2 cache of one agent can be kept coherent with other agents by using 6090 the MTYPE CC (cache-coherent) with the PTE C-bit for memory local to the L2, 6091 and MTYPE UC (uncached) with the PTE C-bit set for memory not local to the 6092 L2. 6093 6094 * Any local memory cache lines will be automatically invalidated by writes 6095 from CUs associated with other L2 caches, or writes from the CPU, due to 6096 the cache probe caused by coherent requests. Coherent requests are caused 6097 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6098 XGMI, and by PCIe requests that are configured to be coherent requests. 6099 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6100 Subsequent access from the GPU will automatically invalidate or writeback 6101 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6102 * Since all work-groups on the same agent share the same L2, no L2 6103 invalidation or writeback is required for coherence. 6104 * Since local memory reads and writes of work-groups in different agents 6105 access memory using MTYPE CC, no L2 invalidate or writeback is required 6106 for coherence. MTYPE CC causes write through to DRAM and local reads to be 6107 invalidated by remote writes with with the PTE C-bit. 6108 * Since remote memory reads and writes of work-groups in different agents 6109 access memory using MTYPE UC, no L2 invalidate or writeback is required 6110 for coherence. MTYPE UC causes direct accesses to DRAM. 6111 6112 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6113 MTYPE UC (uncached) which bypasses the L2. 6114 6115Scalar memory operations are only used to access memory that is proven to not 6116change during the execution of the kernel dispatch. This includes constant 6117address space and global address space for program scope ``const`` variables. 6118Therefore, the kernel machine code does not have to maintain the scalar cache to 6119ensure it is coherent with the vector caches. The scalar and vector caches are 6120invalidated between kernel dispatches by CP since constant address space data 6121may change between kernel dispatch executions. See 6122:ref:`amdgpu-amdhsa-memory-spaces`. 6123 6124The one exception is if scalar writes are used to spill SGPR registers. In this 6125case the AMDGPU backend ensures the memory location used to spill is never 6126accessed by vector memory operations at the same time. If scalar writes are used 6127then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6128return since the locations may be used for vector memory instructions by a 6129future wavefront that uses the same scratch area, or a function call that 6130creates a frame at the same address, respectively. There is no need for a 6131``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6132 6133For kernarg backing memory: 6134 6135* CP invalidates the L1 cache at the start of each kernel dispatch. 6136* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6137 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6138 cache. This also causes it to be treated as non-volatile and so is not 6139 invalidated by ``*_vol``. 6140* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6141 so the L2 cache will be coherent with the CPU and other agents. 6142 6143Scratch backing memory (which is used for the private address space) is accessed 6144with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6145only accessed by a single thread, and is always write-before-read, there is 6146never a need to invalidate these entries from the L1 cache. Hence all cache 6147invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6148 6149The code sequences used to implement the memory model for GFX90A are defined 6150in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6151 6152 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6153 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6154 6155 ============ ============ ============== ========== ================================ 6156 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6157 Ordering Sync Scope Address GFX90A 6158 Space 6159 ============ ============ ============== ========== ================================ 6160 **Non-Atomic** 6161 ------------------------------------------------------------------------------------ 6162 load *none* *none* - global - !volatile & !nontemporal 6163 - generic 6164 - private 1. buffer/global/flat_load 6165 - constant 6166 - !volatile & nontemporal 6167 6168 1. buffer/global/flat_load 6169 glc=1 slc=1 6170 6171 - volatile 6172 6173 1. buffer/global/flat_load 6174 glc=1 6175 2. s_waitcnt vmcnt(0) 6176 6177 - Must happen before 6178 any following volatile 6179 global/generic 6180 load/store. 6181 - Ensures that 6182 volatile 6183 operations to 6184 different 6185 addresses will not 6186 be reordered by 6187 hardware. 6188 6189 load *none* *none* - local 1. ds_load 6190 store *none* *none* - global - !volatile & !nontemporal 6191 - generic 6192 - private 1. buffer/global/flat_store 6193 - constant 6194 - !volatile & nontemporal 6195 6196 1. buffer/global/flat_store 6197 glc=1 slc=1 6198 6199 - volatile 6200 6201 1. buffer/global/flat_store 6202 2. s_waitcnt vmcnt(0) 6203 6204 - Must happen before 6205 any following volatile 6206 global/generic 6207 load/store. 6208 - Ensures that 6209 volatile 6210 operations to 6211 different 6212 addresses will not 6213 be reordered by 6214 hardware. 6215 6216 store *none* *none* - local 1. ds_store 6217 **Unordered Atomic** 6218 ------------------------------------------------------------------------------------ 6219 load atomic unordered *any* *any* *Same as non-atomic*. 6220 store atomic unordered *any* *any* *Same as non-atomic*. 6221 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6222 **Monotonic Atomic** 6223 ------------------------------------------------------------------------------------ 6224 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6225 - wavefront - generic 6226 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6227 - generic glc=1 6228 6229 - If not TgSplit execution 6230 mode, omit glc=1. 6231 6232 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6233 - wavefront local address space cannot 6234 - workgroup be used.* 6235 6236 1. ds_load 6237 load atomic monotonic - agent - global 1. buffer/global/flat_load 6238 - generic glc=1 6239 load atomic monotonic - system - global 1. buffer/global/flat_load 6240 - generic glc=1 6241 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6242 - wavefront - generic 6243 - workgroup 6244 - agent 6245 store atomic monotonic - system - global 1. buffer/global/flat_store 6246 - generic 6247 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6248 - wavefront local address space cannot 6249 - workgroup be used.* 6250 6251 1. ds_store 6252 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6253 - wavefront - generic 6254 - workgroup 6255 - agent 6256 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6257 - generic 6258 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6259 - wavefront local address space cannot 6260 - workgroup be used.* 6261 6262 1. ds_atomic 6263 **Acquire Atomic** 6264 ------------------------------------------------------------------------------------ 6265 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6266 - wavefront - local 6267 - generic 6268 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6269 6270 - If not TgSplit execution 6271 mode, omit glc=1. 6272 6273 2. s_waitcnt vmcnt(0) 6274 6275 - If not TgSplit execution 6276 mode, omit. 6277 - Must happen before the 6278 following buffer_wbinvl1_vol. 6279 6280 3. buffer_wbinvl1_vol 6281 6282 - If not TgSplit execution 6283 mode, omit. 6284 - Must happen before 6285 any following 6286 global/generic 6287 load/load 6288 atomic/store/store 6289 atomic/atomicrmw. 6290 - Ensures that 6291 following 6292 loads will not see 6293 stale data. 6294 6295 load atomic acquire - workgroup - local *If TgSplit execution mode, 6296 local address space cannot 6297 be used.* 6298 6299 1. ds_load 6300 2. s_waitcnt lgkmcnt(0) 6301 6302 - If OpenCL, omit. 6303 - Must happen before 6304 any following 6305 global/generic 6306 load/load 6307 atomic/store/store 6308 atomic/atomicrmw. 6309 - Ensures any 6310 following global 6311 data read is no 6312 older than the local load 6313 atomic value being 6314 acquired. 6315 6316 load atomic acquire - workgroup - generic 1. flat_load glc=1 6317 6318 - If not TgSplit execution 6319 mode, omit glc=1. 6320 6321 2. s_waitcnt lgkm/vmcnt(0) 6322 6323 - Use lgkmcnt(0) if not 6324 TgSplit execution mode 6325 and vmcnt(0) if TgSplit 6326 execution mode. 6327 - If OpenCL, omit lgkmcnt(0). 6328 - Must happen before 6329 the following 6330 buffer_wbinvl1_vol and any 6331 following global/generic 6332 load/load 6333 atomic/store/store 6334 atomic/atomicrmw. 6335 - Ensures any 6336 following global 6337 data read is no 6338 older than a local load 6339 atomic value being 6340 acquired. 6341 6342 3. buffer_wbinvl1_vol 6343 6344 - If not TgSplit execution 6345 mode, omit. 6346 - Ensures that 6347 following 6348 loads will not see 6349 stale data. 6350 6351 load atomic acquire - agent - global 1. buffer/global_load 6352 glc=1 6353 2. s_waitcnt vmcnt(0) 6354 6355 - Must happen before 6356 following 6357 buffer_wbinvl1_vol. 6358 - Ensures the load 6359 has completed 6360 before invalidating 6361 the cache. 6362 6363 3. buffer_wbinvl1_vol 6364 6365 - Must happen before 6366 any following 6367 global/generic 6368 load/load 6369 atomic/atomicrmw. 6370 - Ensures that 6371 following 6372 loads will not see 6373 stale global data. 6374 6375 load atomic acquire - system - global 1. buffer/global/flat_load 6376 glc=1 6377 2. s_waitcnt vmcnt(0) 6378 6379 - Must happen before 6380 following 6381 buffer_wbinvl1_vol. 6382 - Ensures the load 6383 has completed 6384 before invalidating 6385 the cache. 6386 6387 3. buffer_wbinvl1_vol 6388 6389 - Must happen before 6390 any following 6391 global/generic 6392 load/load 6393 atomic/atomicrmw. 6394 - Ensures that 6395 following 6396 loads will not see 6397 stale L1 global data. 6398 MTYPE RW and CC memory will 6399 never be stale in L2 due to 6400 the memory probes. 6401 6402 load atomic acquire - agent - generic 1. flat_load glc=1 6403 2. s_waitcnt vmcnt(0) & 6404 lgkmcnt(0) 6405 6406 - If TgSplit execution mode, 6407 omit lgkmcnt(0). 6408 - If OpenCL omit 6409 lgkmcnt(0). 6410 - Must happen before 6411 following 6412 buffer_wbinvl1_vol. 6413 - Ensures the flat_load 6414 has completed 6415 before invalidating 6416 the cache. 6417 6418 3. buffer_wbinvl1_vol 6419 6420 - Must happen before 6421 any following 6422 global/generic 6423 load/load 6424 atomic/atomicrmw. 6425 - Ensures that 6426 following loads 6427 will not see stale 6428 global data. 6429 6430 load atomic acquire - system - generic 1. flat_load glc=1 6431 2. s_waitcnt vmcnt(0) & 6432 lgkmcnt(0) 6433 6434 - If TgSplit execution mode, 6435 omit lgkmcnt(0). 6436 - If OpenCL omit 6437 lgkmcnt(0). 6438 - Must happen before 6439 following 6440 buffer_wbinvl1_vol. 6441 - Ensures the flat_load 6442 has completed 6443 before invalidating 6444 the caches. 6445 6446 3. buffer_wbinvl1_vol 6447 6448 - Must happen before 6449 any following 6450 global/generic 6451 load/load 6452 atomic/atomicrmw. 6453 - Ensures that 6454 following 6455 L1 loads will not see 6456 stale global data. 6457 MTYPE RW and CC memory will 6458 never be stale in L2 due to 6459 the memory probes. 6460 6461 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6462 - wavefront - generic 6463 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6464 - wavefront local address space cannot 6465 be used.* 6466 6467 1. ds_atomic 6468 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6469 2. s_waitcnt vmcnt(0) 6470 6471 - If not TgSplit execution 6472 mode, omit. 6473 - Must happen before the 6474 following buffer_wbinvl1_vol. 6475 - Ensures the atomicrmw 6476 has completed 6477 before invalidating 6478 the cache. 6479 6480 3. buffer_wbinvl1_vol 6481 6482 - If not TgSplit execution 6483 mode, omit. 6484 - Must happen before 6485 any following 6486 global/generic 6487 load/load 6488 atomic/atomicrmw. 6489 - Ensures that 6490 following loads 6491 will not see stale 6492 global data. 6493 6494 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6495 local address space cannot 6496 be used.* 6497 6498 1. ds_atomic 6499 2. s_waitcnt lgkmcnt(0) 6500 6501 - If OpenCL, omit. 6502 - Must happen before 6503 any following 6504 global/generic 6505 load/load 6506 atomic/store/store 6507 atomic/atomicrmw. 6508 - Ensures any 6509 following global 6510 data read is no 6511 older than the local 6512 atomicrmw value 6513 being acquired. 6514 6515 atomicrmw acquire - workgroup - generic 1. flat_atomic 6516 2. s_waitcnt lgkm/vmcnt(0) 6517 6518 - Use lgkmcnt(0) if not 6519 TgSplit execution mode 6520 and vmcnt(0) if TgSplit 6521 execution mode. 6522 - If OpenCL, omit lgkmcnt(0). 6523 - Must happen before 6524 the following 6525 buffer_wbinvl1_vol and 6526 any following 6527 global/generic 6528 load/load 6529 atomic/store/store 6530 atomic/atomicrmw. 6531 - Ensures any 6532 following global 6533 data read is no 6534 older than a local 6535 atomicrmw value 6536 being acquired. 6537 6538 3. buffer_wbinvl1_vol 6539 6540 - If not TgSplit execution 6541 mode, omit. 6542 - Ensures that 6543 following 6544 loads will not see 6545 stale data. 6546 6547 atomicrmw acquire - agent - global 1. buffer/global_atomic 6548 2. s_waitcnt vmcnt(0) 6549 6550 - Must happen before 6551 following 6552 buffer_wbinvl1_vol. 6553 - Ensures the 6554 atomicrmw has 6555 completed before 6556 invalidating the 6557 cache. 6558 6559 3. buffer_wbinvl1_vol 6560 6561 - Must happen before 6562 any following 6563 global/generic 6564 load/load 6565 atomic/atomicrmw. 6566 - Ensures that 6567 following loads 6568 will not see stale 6569 global data. 6570 6571 atomicrmw acquire - system - global 1. buffer/global_atomic 6572 2. s_waitcnt vmcnt(0) 6573 6574 - Must happen before 6575 following 6576 buffer_wbinvl1_vol. 6577 - Ensures the 6578 atomicrmw has 6579 completed before 6580 invalidating the 6581 caches. 6582 6583 3. buffer_wbinvl1_vol 6584 6585 - Must happen before 6586 any following 6587 global/generic 6588 load/load 6589 atomic/atomicrmw. 6590 - Ensures that 6591 following 6592 loads will not see 6593 stale L1 global data. 6594 MTYPE RW and CC L2 memory 6595 never be stale in L2 due to 6596 the memory probes. 6597 6598 atomicrmw acquire - agent - generic 1. flat_atomic 6599 2. s_waitcnt vmcnt(0) & 6600 lgkmcnt(0) 6601 6602 - If TgSplit execution mode, 6603 omit lgkmcnt(0). 6604 - If OpenCL, omit 6605 lgkmcnt(0). 6606 - Must happen before 6607 following 6608 buffer_wbinvl1_vol. 6609 - Ensures the 6610 atomicrmw has 6611 completed before 6612 invalidating the 6613 cache. 6614 6615 3. buffer_wbinvl1_vol 6616 6617 - Must happen before 6618 any following 6619 global/generic 6620 load/load 6621 atomic/atomicrmw. 6622 - Ensures that 6623 following loads 6624 will not see stale 6625 global data. 6626 6627 atomicrmw acquire - system - generic 1. flat_atomic 6628 2. s_waitcnt vmcnt(0) & 6629 lgkmcnt(0) 6630 6631 - If TgSplit execution mode, 6632 omit lgkmcnt(0). 6633 - If OpenCL, omit 6634 lgkmcnt(0). 6635 - Must happen before 6636 following 6637 buffer_wbinvl1_vol. 6638 - Ensures the 6639 atomicrmw has 6640 completed before 6641 invalidating the 6642 caches. 6643 6644 3. buffer_wbinvl1_vol 6645 6646 - Must happen before 6647 any following 6648 global/generic 6649 load/load 6650 atomic/atomicrmw. 6651 - Ensures that 6652 following 6653 loads will not see 6654 stale L1 global data. 6655 MTYPE RW and CC memory will 6656 never be stale in L2 due to 6657 the memory probes. 6658 6659 fence acquire - singlethread *none* *none* 6660 - wavefront 6661 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6662 6663 - Use lgkmcnt(0) if not 6664 TgSplit execution mode 6665 and vmcnt(0) if TgSplit 6666 execution mode. 6667 - If OpenCL and 6668 address space is 6669 not generic, omit 6670 lgkmcnt(0). 6671 - If OpenCL and 6672 address space is 6673 local, omit 6674 vmcnt(0). 6675 - However, since LLVM 6676 currently has no 6677 address space on 6678 the fence need to 6679 conservatively 6680 always generate. If 6681 fence had an 6682 address space then 6683 set to address 6684 space of OpenCL 6685 fence flag, or to 6686 generic if both 6687 local and global 6688 flags are 6689 specified. 6690 - s_waitcnt vmcnt(0) 6691 must happen after 6692 any preceding 6693 global/generic load 6694 atomic/ 6695 atomicrmw 6696 with an equal or 6697 wider sync scope 6698 and memory ordering 6699 stronger than 6700 unordered (this is 6701 termed the 6702 fence-paired-atomic). 6703 - s_waitcnt lgkmcnt(0) 6704 must happen after 6705 any preceding 6706 local/generic load 6707 atomic/atomicrmw 6708 with an equal or 6709 wider sync scope 6710 and memory ordering 6711 stronger than 6712 unordered (this is 6713 termed the 6714 fence-paired-atomic). 6715 - Must happen before 6716 the following 6717 buffer_wbinvl1_vol and 6718 any following 6719 global/generic 6720 load/load 6721 atomic/store/store 6722 atomic/atomicrmw. 6723 - Ensures any 6724 following global 6725 data read is no 6726 older than the 6727 value read by the 6728 fence-paired-atomic. 6729 6730 3. buffer_wbinvl1_vol 6731 6732 - If not TgSplit execution 6733 mode, omit. 6734 - Ensures that 6735 following 6736 loads will not see 6737 stale data. 6738 6739 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6740 vmcnt(0) 6741 6742 - If TgSplit execution mode, 6743 omit lgkmcnt(0). 6744 - If OpenCL and 6745 address space is 6746 not generic, omit 6747 lgkmcnt(0). 6748 - However, since LLVM 6749 currently has no 6750 address space on 6751 the fence need to 6752 conservatively 6753 always generate 6754 (see comment for 6755 previous fence). 6756 - Could be split into 6757 separate s_waitcnt 6758 vmcnt(0) and 6759 s_waitcnt 6760 lgkmcnt(0) to allow 6761 them to be 6762 independently moved 6763 according to the 6764 following rules. 6765 - s_waitcnt vmcnt(0) 6766 must happen after 6767 any preceding 6768 global/generic load 6769 atomic/atomicrmw 6770 with an equal or 6771 wider sync scope 6772 and memory ordering 6773 stronger than 6774 unordered (this is 6775 termed the 6776 fence-paired-atomic). 6777 - s_waitcnt lgkmcnt(0) 6778 must happen after 6779 any preceding 6780 local/generic load 6781 atomic/atomicrmw 6782 with an equal or 6783 wider sync scope 6784 and memory ordering 6785 stronger than 6786 unordered (this is 6787 termed the 6788 fence-paired-atomic). 6789 - Must happen before 6790 the following 6791 buffer_wbinvl1_vol. 6792 - Ensures that the 6793 fence-paired atomic 6794 has completed 6795 before invalidating 6796 the 6797 cache. Therefore 6798 any following 6799 locations read must 6800 be no older than 6801 the value read by 6802 the 6803 fence-paired-atomic. 6804 6805 2. buffer_wbinvl1_vol 6806 6807 - Must happen before any 6808 following global/generic 6809 load/load 6810 atomic/store/store 6811 atomic/atomicrmw. 6812 - Ensures that 6813 following loads 6814 will not see stale 6815 global data. 6816 6817 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6818 vmcnt(0) 6819 6820 - If TgSplit execution mode, 6821 omit lgkmcnt(0). 6822 - If OpenCL and 6823 address space is 6824 not generic, omit 6825 lgkmcnt(0). 6826 - However, since LLVM 6827 currently has no 6828 address space on 6829 the fence need to 6830 conservatively 6831 always generate 6832 (see comment for 6833 previous fence). 6834 - Could be split into 6835 separate s_waitcnt 6836 vmcnt(0) and 6837 s_waitcnt 6838 lgkmcnt(0) to allow 6839 them to be 6840 independently moved 6841 according to the 6842 following rules. 6843 - s_waitcnt vmcnt(0) 6844 must happen after 6845 any preceding 6846 global/generic load 6847 atomic/atomicrmw 6848 with an equal or 6849 wider sync scope 6850 and memory ordering 6851 stronger than 6852 unordered (this is 6853 termed the 6854 fence-paired-atomic). 6855 - s_waitcnt lgkmcnt(0) 6856 must happen after 6857 any preceding 6858 local/generic load 6859 atomic/atomicrmw 6860 with an equal or 6861 wider sync scope 6862 and memory ordering 6863 stronger than 6864 unordered (this is 6865 termed the 6866 fence-paired-atomic). 6867 - Must happen before 6868 the following 6869 buffer_wbinvl1_vol. 6870 - Ensures that the 6871 fence-paired atomic 6872 has completed 6873 before invalidating 6874 the 6875 cache. Therefore 6876 any following 6877 locations read must 6878 be no older than 6879 the value read by 6880 the 6881 fence-paired-atomic. 6882 6883 2. buffer_wbinvl1_vol 6884 6885 - Must happen before any 6886 following global/generic 6887 load/load 6888 atomic/store/store 6889 atomic/atomicrmw. 6890 - Ensures that 6891 following 6892 loads will not see 6893 stale L1 global data. 6894 MTYPE RW and CC memory will 6895 never be stale in L2 due to 6896 the memory probes. 6897 **Release Atomic** 6898 ------------------------------------------------------------------------------------ 6899 store atomic release - singlethread - global 1. buffer/global/flat_store 6900 - wavefront - generic 6901 store atomic release - singlethread - local *If TgSplit execution mode, 6902 - wavefront local address space cannot 6903 be used.* 6904 6905 1. ds_store 6906 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 6907 - generic 6908 - Use lgkmcnt(0) if not 6909 TgSplit execution mode 6910 and vmcnt(0) if TgSplit 6911 execution mode. 6912 - If OpenCL, omit lgkmcnt(0). 6913 - s_waitcnt vmcnt(0) 6914 must happen after 6915 any preceding 6916 global/generic load/store/ 6917 load atomic/store atomic/ 6918 atomicrmw. 6919 - s_waitcnt lgkmcnt(0) 6920 must happen after 6921 any preceding 6922 local/generic 6923 load/store/load 6924 atomic/store 6925 atomic/atomicrmw. 6926 - Must happen before 6927 the following 6928 store. 6929 - Ensures that all 6930 memory operations 6931 have 6932 completed before 6933 performing the 6934 store that is being 6935 released. 6936 6937 2. buffer/global/flat_store 6938 store atomic release - workgroup - local *If TgSplit execution mode, 6939 local address space cannot 6940 be used.* 6941 6942 1. ds_store 6943 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6944 - generic vmcnt(0) 6945 6946 - If TgSplit execution mode, 6947 omit lgkmcnt(0). 6948 - If OpenCL and 6949 address space is 6950 not generic, omit 6951 lgkmcnt(0). 6952 - Could be split into 6953 separate s_waitcnt 6954 vmcnt(0) and 6955 s_waitcnt 6956 lgkmcnt(0) to allow 6957 them to be 6958 independently moved 6959 according to the 6960 following rules. 6961 - s_waitcnt vmcnt(0) 6962 must happen after 6963 any preceding 6964 global/generic 6965 load/store/load 6966 atomic/store 6967 atomic/atomicrmw. 6968 - s_waitcnt lgkmcnt(0) 6969 must happen after 6970 any preceding 6971 local/generic 6972 load/store/load 6973 atomic/store 6974 atomic/atomicrmw. 6975 - Must happen before 6976 the following 6977 store. 6978 - Ensures that all 6979 memory operations 6980 to memory have 6981 completed before 6982 performing the 6983 store that is being 6984 released. 6985 6986 2. buffer/global/flat_store 6987 store atomic release - system - global 1. s_waitcnt lgkmcnt(0) & 6988 - generic vmcnt(0) 6989 6990 - If TgSplit execution mode, 6991 omit lgkmcnt(0). 6992 - If OpenCL and 6993 address space is 6994 not generic, omit 6995 lgkmcnt(0). 6996 - Could be split into 6997 separate s_waitcnt 6998 vmcnt(0) and 6999 s_waitcnt 7000 lgkmcnt(0) to allow 7001 them to be 7002 independently moved 7003 according to the 7004 following rules. 7005 - s_waitcnt vmcnt(0) 7006 must happen after any 7007 preceding 7008 global/generic 7009 load/store/load 7010 atomic/store 7011 atomic/atomicrmw. 7012 - s_waitcnt lgkmcnt(0) 7013 must happen after any 7014 preceding 7015 local/generic 7016 load/store/load 7017 atomic/store 7018 atomic/atomicrmw. 7019 - Must happen before 7020 the following 7021 store. 7022 - Ensures that all 7023 memory operations 7024 to memory and the L2 7025 writeback have 7026 completed before 7027 performing the 7028 store that is being 7029 released. 7030 7031 2. buffer/global/flat_store 7032 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7033 - wavefront - generic 7034 atomicrmw release - singlethread - local *If TgSplit execution mode, 7035 - wavefront local address space cannot 7036 be used.* 7037 7038 1. ds_atomic 7039 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7040 - generic 7041 - Use lgkmcnt(0) if not 7042 TgSplit execution mode 7043 and vmcnt(0) if TgSplit 7044 execution mode. 7045 - If OpenCL, omit 7046 lgkmcnt(0). 7047 - s_waitcnt vmcnt(0) 7048 must happen after 7049 any preceding 7050 global/generic load/store/ 7051 load atomic/store atomic/ 7052 atomicrmw. 7053 - s_waitcnt lgkmcnt(0) 7054 must happen after 7055 any preceding 7056 local/generic 7057 load/store/load 7058 atomic/store 7059 atomic/atomicrmw. 7060 - Must happen before 7061 the following 7062 atomicrmw. 7063 - Ensures that all 7064 memory operations 7065 have 7066 completed before 7067 performing the 7068 atomicrmw that is 7069 being released. 7070 7071 2. buffer/global/flat_atomic 7072 atomicrmw release - workgroup - local *If TgSplit execution mode, 7073 local address space cannot 7074 be used.* 7075 7076 1. ds_atomic 7077 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7078 - generic vmcnt(0) 7079 7080 - If TgSplit execution mode, 7081 omit lgkmcnt(0). 7082 - If OpenCL, omit 7083 lgkmcnt(0). 7084 - Could be split into 7085 separate s_waitcnt 7086 vmcnt(0) and 7087 s_waitcnt 7088 lgkmcnt(0) to allow 7089 them to be 7090 independently moved 7091 according to the 7092 following rules. 7093 - s_waitcnt vmcnt(0) 7094 must happen after 7095 any preceding 7096 global/generic 7097 load/store/load 7098 atomic/store 7099 atomic/atomicrmw. 7100 - s_waitcnt lgkmcnt(0) 7101 must happen after 7102 any preceding 7103 local/generic 7104 load/store/load 7105 atomic/store 7106 atomic/atomicrmw. 7107 - Must happen before 7108 the following 7109 atomicrmw. 7110 - Ensures that all 7111 memory operations 7112 to global and local 7113 have completed 7114 before performing 7115 the atomicrmw that 7116 is being released. 7117 7118 2. buffer/global/flat_atomic 7119 atomicrmw release - system - global 1. s_waitcnt lgkmcnt(0) & 7120 - generic vmcnt(0) 7121 7122 - If TgSplit execution mode, 7123 omit lgkmcnt(0). 7124 - If OpenCL, omit 7125 lgkmcnt(0). 7126 - Could be split into 7127 separate s_waitcnt 7128 vmcnt(0) and 7129 s_waitcnt 7130 lgkmcnt(0) to allow 7131 them to be 7132 independently moved 7133 according to the 7134 following rules. 7135 - s_waitcnt vmcnt(0) 7136 must happen after 7137 any preceding 7138 global/generic 7139 load/store/load 7140 atomic/store 7141 atomic/atomicrmw. 7142 - s_waitcnt lgkmcnt(0) 7143 must happen after 7144 any preceding 7145 local/generic 7146 load/store/load 7147 atomic/store 7148 atomic/atomicrmw. 7149 - Must happen before 7150 the following 7151 atomicrmw. 7152 - Ensures that all 7153 memory operations 7154 to memory and the L2 7155 writeback have 7156 completed before 7157 performing the 7158 store that is being 7159 released. 7160 7161 2. buffer/global/flat_atomic 7162 fence release - singlethread *none* *none* 7163 - wavefront 7164 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7165 7166 - Use lgkmcnt(0) if not 7167 TgSplit execution mode 7168 and vmcnt(0) if TgSplit 7169 execution mode. 7170 - If OpenCL and 7171 address space is 7172 not generic, omit 7173 lgkmcnt(0). 7174 - If OpenCL and 7175 address space is 7176 local, omit 7177 vmcnt(0). 7178 - However, since LLVM 7179 currently has no 7180 address space on 7181 the fence need to 7182 conservatively 7183 always generate. If 7184 fence had an 7185 address space then 7186 set to address 7187 space of OpenCL 7188 fence flag, or to 7189 generic if both 7190 local and global 7191 flags are 7192 specified. 7193 - s_waitcnt vmcnt(0) 7194 must happen after 7195 any preceding 7196 global/generic 7197 load/store/ 7198 load atomic/store atomic/ 7199 atomicrmw. 7200 - s_waitcnt lgkmcnt(0) 7201 must happen after 7202 any preceding 7203 local/generic 7204 load/load 7205 atomic/store/store 7206 atomic/atomicrmw. 7207 - Must happen before 7208 any following store 7209 atomic/atomicrmw 7210 with an equal or 7211 wider sync scope 7212 and memory ordering 7213 stronger than 7214 unordered (this is 7215 termed the 7216 fence-paired-atomic). 7217 - Ensures that all 7218 memory operations 7219 have 7220 completed before 7221 performing the 7222 following 7223 fence-paired-atomic. 7224 7225 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7226 vmcnt(0) 7227 7228 - If TgSplit execution mode, 7229 omit lgkmcnt(0). 7230 - If OpenCL and 7231 address space is 7232 not generic, omit 7233 lgkmcnt(0). 7234 - If OpenCL and 7235 address space is 7236 local, omit 7237 vmcnt(0). 7238 - However, since LLVM 7239 currently has no 7240 address space on 7241 the fence need to 7242 conservatively 7243 always generate. If 7244 fence had an 7245 address space then 7246 set to address 7247 space of OpenCL 7248 fence flag, or to 7249 generic if both 7250 local and global 7251 flags are 7252 specified. 7253 - Could be split into 7254 separate s_waitcnt 7255 vmcnt(0) and 7256 s_waitcnt 7257 lgkmcnt(0) to allow 7258 them to be 7259 independently moved 7260 according to the 7261 following rules. 7262 - s_waitcnt vmcnt(0) 7263 must happen after 7264 any preceding 7265 global/generic 7266 load/store/load 7267 atomic/store 7268 atomic/atomicrmw. 7269 - s_waitcnt lgkmcnt(0) 7270 must happen after 7271 any preceding 7272 local/generic 7273 load/store/load 7274 atomic/store 7275 atomic/atomicrmw. 7276 - Must happen before 7277 any following store 7278 atomic/atomicrmw 7279 with an equal or 7280 wider sync scope 7281 and memory ordering 7282 stronger than 7283 unordered (this is 7284 termed the 7285 fence-paired-atomic). 7286 - Ensures that all 7287 memory operations 7288 have 7289 completed before 7290 performing the 7291 following 7292 fence-paired-atomic. 7293 7294 fence release - system *none* 1. s_waitcnt lgkmcnt(0) & 7295 vmcnt(0) 7296 7297 - If TgSplit execution mode, 7298 omit lgkmcnt(0). 7299 - If OpenCL and 7300 address space is 7301 not generic, omit 7302 lgkmcnt(0). 7303 - If OpenCL and 7304 address space is 7305 local, omit 7306 vmcnt(0). 7307 - However, since LLVM 7308 currently has no 7309 address space on 7310 the fence need to 7311 conservatively 7312 always generate. If 7313 fence had an 7314 address space then 7315 set to address 7316 space of OpenCL 7317 fence flag, or to 7318 generic if both 7319 local and global 7320 flags are 7321 specified. 7322 - Could be split into 7323 separate s_waitcnt 7324 vmcnt(0) and 7325 s_waitcnt 7326 lgkmcnt(0) to allow 7327 them to be 7328 independently moved 7329 according to the 7330 following rules. 7331 - s_waitcnt vmcnt(0) 7332 must happen after 7333 any preceding 7334 global/generic 7335 load/store/load 7336 atomic/store 7337 atomic/atomicrmw. 7338 - s_waitcnt lgkmcnt(0) 7339 must happen after 7340 any preceding 7341 local/generic 7342 load/store/load 7343 atomic/store 7344 atomic/atomicrmw. 7345 - Must happen before 7346 any following store 7347 atomic/atomicrmw 7348 with an equal or 7349 wider sync scope 7350 and memory ordering 7351 stronger than 7352 unordered (this is 7353 termed the 7354 fence-paired-atomic). 7355 - Ensures that all 7356 memory operations 7357 have 7358 completed before 7359 performing the 7360 following 7361 fence-paired-atomic. 7362 7363 **Acquire-Release Atomic** 7364 ------------------------------------------------------------------------------------ 7365 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7366 - wavefront - generic 7367 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7368 - wavefront local address space cannot 7369 be used.* 7370 7371 1. ds_atomic 7372 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7373 7374 - Use lgkmcnt(0) if not 7375 TgSplit execution mode 7376 and vmcnt(0) if TgSplit 7377 execution mode. 7378 - If OpenCL, omit 7379 lgkmcnt(0). 7380 - Must happen after 7381 any preceding 7382 local/generic 7383 load/store/load 7384 atomic/store 7385 atomic/atomicrmw. 7386 - s_waitcnt vmcnt(0) 7387 must happen after 7388 any preceding 7389 global/generic load/store/ 7390 load atomic/store atomic/ 7391 atomicrmw. 7392 - s_waitcnt lgkmcnt(0) 7393 must happen after 7394 any preceding 7395 local/generic 7396 load/store/load 7397 atomic/store 7398 atomic/atomicrmw. 7399 - Must happen before 7400 the following 7401 atomicrmw. 7402 - Ensures that all 7403 memory operations 7404 have 7405 completed before 7406 performing the 7407 atomicrmw that is 7408 being released. 7409 7410 2. buffer/global_atomic 7411 3. s_waitcnt vmcnt(0) 7412 7413 - If not TgSplit execution 7414 mode, omit. 7415 - Must happen before 7416 the following 7417 buffer_wbinvl1_vol. 7418 - Ensures any 7419 following global 7420 data read is no 7421 older than the 7422 atomicrmw value 7423 being acquired. 7424 7425 4. buffer_wbinvl1_vol 7426 7427 - If not TgSplit execution 7428 mode, omit. 7429 - Ensures that 7430 following 7431 loads will not see 7432 stale data. 7433 7434 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7435 local address space cannot 7436 be used.* 7437 7438 1. ds_atomic 7439 2. s_waitcnt lgkmcnt(0) 7440 7441 - If OpenCL, omit. 7442 - Must happen before 7443 any following 7444 global/generic 7445 load/load 7446 atomic/store/store 7447 atomic/atomicrmw. 7448 - Ensures any 7449 following global 7450 data read is no 7451 older than the local load 7452 atomic value being 7453 acquired. 7454 7455 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7456 7457 - Use lgkmcnt(0) if not 7458 TgSplit execution mode 7459 and vmcnt(0) if TgSplit 7460 execution mode. 7461 - If OpenCL, omit 7462 lgkmcnt(0). 7463 - s_waitcnt vmcnt(0) 7464 must happen after 7465 any preceding 7466 global/generic load/store/ 7467 load atomic/store atomic/ 7468 atomicrmw. 7469 - s_waitcnt lgkmcnt(0) 7470 must happen after 7471 any preceding 7472 local/generic 7473 load/store/load 7474 atomic/store 7475 atomic/atomicrmw. 7476 - Must happen before 7477 the following 7478 atomicrmw. 7479 - Ensures that all 7480 memory operations 7481 have 7482 completed before 7483 performing the 7484 atomicrmw that is 7485 being released. 7486 7487 2. flat_atomic 7488 3. s_waitcnt lgkmcnt(0) & 7489 vmcnt(0) 7490 7491 - If not TgSplit execution 7492 mode, omit vmcnt(0). 7493 - If OpenCL, omit 7494 lgkmcnt(0). 7495 - Must happen before 7496 the following 7497 buffer_wbinvl1_vol and 7498 any following 7499 global/generic 7500 load/load 7501 atomic/store/store 7502 atomic/atomicrmw. 7503 - Ensures any 7504 following global 7505 data read is no 7506 older than a local load 7507 atomic value being 7508 acquired. 7509 7510 3. buffer_wbinvl1_vol 7511 7512 - If not TgSplit execution 7513 mode, omit. 7514 - Ensures that 7515 following 7516 loads will not see 7517 stale data. 7518 7519 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7520 vmcnt(0) 7521 7522 - If TgSplit execution mode, 7523 omit lgkmcnt(0). 7524 - If OpenCL, omit 7525 lgkmcnt(0). 7526 - Could be split into 7527 separate s_waitcnt 7528 vmcnt(0) and 7529 s_waitcnt 7530 lgkmcnt(0) to allow 7531 them to be 7532 independently moved 7533 according to the 7534 following rules. 7535 - s_waitcnt vmcnt(0) 7536 must happen after 7537 any preceding 7538 global/generic 7539 load/store/load 7540 atomic/store 7541 atomic/atomicrmw. 7542 - s_waitcnt lgkmcnt(0) 7543 must happen after 7544 any preceding 7545 local/generic 7546 load/store/load 7547 atomic/store 7548 atomic/atomicrmw. 7549 - Must happen before 7550 the following 7551 atomicrmw. 7552 - Ensures that all 7553 memory operations 7554 to global have 7555 completed before 7556 performing the 7557 atomicrmw that is 7558 being released. 7559 7560 2. buffer/global_atomic 7561 3. s_waitcnt vmcnt(0) 7562 7563 - Must happen before 7564 following 7565 buffer_wbinvl1_vol. 7566 - Ensures the 7567 atomicrmw has 7568 completed before 7569 invalidating the 7570 cache. 7571 7572 4. buffer_wbinvl1_vol 7573 7574 - Must happen before 7575 any following 7576 global/generic 7577 load/load 7578 atomic/atomicrmw. 7579 - Ensures that 7580 following loads 7581 will not see stale 7582 global data. 7583 7584 atomicrmw acq_rel - system - global 1. s_waitcnt lgkmcnt(0) & 7585 vmcnt(0) 7586 7587 - If TgSplit execution mode, 7588 omit lgkmcnt(0). 7589 - If OpenCL, omit 7590 lgkmcnt(0). 7591 - Could be split into 7592 separate s_waitcnt 7593 vmcnt(0) and 7594 s_waitcnt 7595 lgkmcnt(0) to allow 7596 them to be 7597 independently moved 7598 according to the 7599 following rules. 7600 - s_waitcnt vmcnt(0) 7601 must happen after 7602 any preceding 7603 global/generic 7604 load/store/load 7605 atomic/store 7606 atomic/atomicrmw. 7607 - s_waitcnt lgkmcnt(0) 7608 must happen after 7609 any preceding 7610 local/generic 7611 load/store/load 7612 atomic/store 7613 atomic/atomicrmw. 7614 - Must happen before 7615 the following 7616 atomicrmw. 7617 - Ensures that all 7618 memory operations 7619 to global and L2 writeback 7620 have completed before 7621 performing the 7622 atomicrmw that is 7623 being released. 7624 7625 2. buffer/global_atomic 7626 3. s_waitcnt vmcnt(0) 7627 7628 - Must happen before 7629 following 7630 buffer_wbinvl1_vol. 7631 - Ensures the 7632 atomicrmw has 7633 completed before 7634 invalidating the 7635 caches. 7636 7637 4. buffer_wbinvl1_vol 7638 7639 - Must happen before 7640 any following 7641 global/generic 7642 load/load 7643 atomic/atomicrmw. 7644 - Ensures that 7645 following 7646 loads will not see 7647 stale L1 global data. 7648 MTYPE RW and CC memory will 7649 never be stale in L2 due to 7650 the memory probes. 7651 7652 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7653 vmcnt(0) 7654 7655 - If TgSplit execution mode, 7656 omit lgkmcnt(0). 7657 - If OpenCL, omit 7658 lgkmcnt(0). 7659 - Could be split into 7660 separate s_waitcnt 7661 vmcnt(0) and 7662 s_waitcnt 7663 lgkmcnt(0) to allow 7664 them to be 7665 independently moved 7666 according to the 7667 following rules. 7668 - s_waitcnt vmcnt(0) 7669 must happen after 7670 any preceding 7671 global/generic 7672 load/store/load 7673 atomic/store 7674 atomic/atomicrmw. 7675 - s_waitcnt lgkmcnt(0) 7676 must happen after 7677 any preceding 7678 local/generic 7679 load/store/load 7680 atomic/store 7681 atomic/atomicrmw. 7682 - Must happen before 7683 the following 7684 atomicrmw. 7685 - Ensures that all 7686 memory operations 7687 to global have 7688 completed before 7689 performing the 7690 atomicrmw that is 7691 being released. 7692 7693 2. flat_atomic 7694 3. s_waitcnt vmcnt(0) & 7695 lgkmcnt(0) 7696 7697 - If TgSplit execution mode, 7698 omit lgkmcnt(0). 7699 - If OpenCL, omit 7700 lgkmcnt(0). 7701 - Must happen before 7702 following 7703 buffer_wbinvl1_vol. 7704 - Ensures the 7705 atomicrmw has 7706 completed before 7707 invalidating the 7708 cache. 7709 7710 4. buffer_wbinvl1_vol 7711 7712 - Must happen before 7713 any following 7714 global/generic 7715 load/load 7716 atomic/atomicrmw. 7717 - Ensures that 7718 following loads 7719 will not see stale 7720 global data. 7721 7722 atomicrmw acq_rel - system - generic 1. s_waitcnt lgkmcnt(0) & 7723 vmcnt(0) 7724 7725 - If TgSplit execution mode, 7726 omit lgkmcnt(0). 7727 - If OpenCL, omit 7728 lgkmcnt(0). 7729 - Could be split into 7730 separate s_waitcnt 7731 vmcnt(0) and 7732 s_waitcnt 7733 lgkmcnt(0) to allow 7734 them to be 7735 independently moved 7736 according to the 7737 following rules. 7738 - s_waitcnt vmcnt(0) 7739 must happen after 7740 any preceding 7741 global/generic 7742 load/store/load 7743 atomic/store 7744 atomic/atomicrmw. 7745 - s_waitcnt lgkmcnt(0) 7746 must happen after 7747 any preceding 7748 local/generic 7749 load/store/load 7750 atomic/store 7751 atomic/atomicrmw. 7752 - Must happen before 7753 the following 7754 atomicrmw. 7755 - Ensures that all 7756 memory operations 7757 to global and L2 writeback 7758 have completed before 7759 performing the 7760 atomicrmw that is 7761 being released. 7762 7763 2. flat_atomic 7764 3. s_waitcnt vmcnt(0) & 7765 lgkmcnt(0) 7766 7767 - If TgSplit execution mode, 7768 omit lgkmcnt(0). 7769 - If OpenCL, omit 7770 lgkmcnt(0). 7771 - Must happen before 7772 following 7773 buffer_wbinvl1_vol. 7774 - Ensures the 7775 atomicrmw has 7776 completed before 7777 invalidating the 7778 caches. 7779 7780 4. buffer_wbinvl1_vol 7781 7782 - Must happen before 7783 any following 7784 global/generic 7785 load/load 7786 atomic/atomicrmw. 7787 - Ensures that 7788 following 7789 loads will not see 7790 stale L1 global data. 7791 MTYPE RW and CC memory will 7792 never be stale in L2 due to 7793 the memory probes. 7794 7795 fence acq_rel - singlethread *none* *none* 7796 - wavefront 7797 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7798 7799 - Use lgkmcnt(0) if not 7800 TgSplit execution mode 7801 and vmcnt(0) if TgSplit 7802 execution mode. 7803 - If OpenCL and 7804 address space is 7805 not generic, omit 7806 lgkmcnt(0). 7807 - If OpenCL and 7808 address space is 7809 local, omit 7810 vmcnt(0). 7811 - However, 7812 since LLVM 7813 currently has no 7814 address space on 7815 the fence need to 7816 conservatively 7817 always generate 7818 (see comment for 7819 previous fence). 7820 - s_waitcnt vmcnt(0) 7821 must happen after 7822 any preceding 7823 global/generic 7824 load/store/ 7825 load atomic/store atomic/ 7826 atomicrmw. 7827 - s_waitcnt lgkmcnt(0) 7828 must happen after 7829 any preceding 7830 local/generic 7831 load/load 7832 atomic/store/store 7833 atomic/atomicrmw. 7834 - Must happen before 7835 any following 7836 global/generic 7837 load/load 7838 atomic/store/store 7839 atomic/atomicrmw. 7840 - Ensures that all 7841 memory operations 7842 have 7843 completed before 7844 performing any 7845 following global 7846 memory operations. 7847 - Ensures that the 7848 preceding 7849 local/generic load 7850 atomic/atomicrmw 7851 with an equal or 7852 wider sync scope 7853 and memory ordering 7854 stronger than 7855 unordered (this is 7856 termed the 7857 acquire-fence-paired-atomic) 7858 has completed 7859 before following 7860 global memory 7861 operations. This 7862 satisfies the 7863 requirements of 7864 acquire. 7865 - Ensures that all 7866 previous memory 7867 operations have 7868 completed before a 7869 following 7870 local/generic store 7871 atomic/atomicrmw 7872 with an equal or 7873 wider sync scope 7874 and memory ordering 7875 stronger than 7876 unordered (this is 7877 termed the 7878 release-fence-paired-atomic). 7879 This satisfies the 7880 requirements of 7881 release. 7882 - Must happen before 7883 the following 7884 buffer_wbinvl1_vol. 7885 - Ensures that the 7886 acquire-fence-paired 7887 atomic has completed 7888 before invalidating 7889 the 7890 cache. Therefore 7891 any following 7892 locations read must 7893 be no older than 7894 the value read by 7895 the 7896 acquire-fence-paired-atomic. 7897 7898 3. buffer_wbinvl1_vol 7899 7900 - If not TgSplit execution 7901 mode, omit. 7902 - Ensures that 7903 following 7904 loads will not see 7905 stale data. 7906 7907 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7908 vmcnt(0) 7909 7910 - If TgSplit execution mode, 7911 omit lgkmcnt(0). 7912 - If OpenCL and 7913 address space is 7914 not generic, omit 7915 lgkmcnt(0). 7916 - However, since LLVM 7917 currently has no 7918 address space on 7919 the fence need to 7920 conservatively 7921 always generate 7922 (see comment for 7923 previous fence). 7924 - Could be split into 7925 separate s_waitcnt 7926 vmcnt(0) and 7927 s_waitcnt 7928 lgkmcnt(0) to allow 7929 them to be 7930 independently moved 7931 according to the 7932 following rules. 7933 - s_waitcnt vmcnt(0) 7934 must happen after 7935 any preceding 7936 global/generic 7937 load/store/load 7938 atomic/store 7939 atomic/atomicrmw. 7940 - s_waitcnt lgkmcnt(0) 7941 must happen after 7942 any preceding 7943 local/generic 7944 load/store/load 7945 atomic/store 7946 atomic/atomicrmw. 7947 - Must happen before 7948 the following 7949 buffer_wbinvl1_vol. 7950 - Ensures that the 7951 preceding 7952 global/local/generic 7953 load 7954 atomic/atomicrmw 7955 with an equal or 7956 wider sync scope 7957 and memory ordering 7958 stronger than 7959 unordered (this is 7960 termed the 7961 acquire-fence-paired-atomic) 7962 has completed 7963 before invalidating 7964 the cache. This 7965 satisfies the 7966 requirements of 7967 acquire. 7968 - Ensures that all 7969 previous memory 7970 operations have 7971 completed before a 7972 following 7973 global/local/generic 7974 store 7975 atomic/atomicrmw 7976 with an equal or 7977 wider sync scope 7978 and memory ordering 7979 stronger than 7980 unordered (this is 7981 termed the 7982 release-fence-paired-atomic). 7983 This satisfies the 7984 requirements of 7985 release. 7986 7987 2. buffer_wbinvl1_vol 7988 7989 - Must happen before 7990 any following 7991 global/generic 7992 load/load 7993 atomic/store/store 7994 atomic/atomicrmw. 7995 - Ensures that 7996 following loads 7997 will not see stale 7998 global data. This 7999 satisfies the 8000 requirements of 8001 acquire. 8002 8003 fence acq_rel - system *none* 1. s_waitcnt lgkmcnt(0) & 8004 vmcnt(0) 8005 8006 - If TgSplit execution mode, 8007 omit lgkmcnt(0). 8008 - If OpenCL and 8009 address space is 8010 not generic, omit 8011 lgkmcnt(0). 8012 - However, since LLVM 8013 currently has no 8014 address space on 8015 the fence need to 8016 conservatively 8017 always generate 8018 (see comment for 8019 previous fence). 8020 - Could be split into 8021 separate s_waitcnt 8022 vmcnt(0) and 8023 s_waitcnt 8024 lgkmcnt(0) to allow 8025 them to be 8026 independently moved 8027 according to the 8028 following rules. 8029 - s_waitcnt vmcnt(0) 8030 must happen after 8031 any preceding 8032 global/generic 8033 load/store/load 8034 atomic/store 8035 atomic/atomicrmw. 8036 - s_waitcnt lgkmcnt(0) 8037 must happen after 8038 any preceding 8039 local/generic 8040 load/store/load 8041 atomic/store 8042 atomic/atomicrmw. 8043 - Must happen before 8044 the following 8045 buffer_wbinvl1_vol. 8046 - Ensures that the 8047 preceding 8048 global/local/generic 8049 load 8050 atomic/atomicrmw 8051 with an equal or 8052 wider sync scope 8053 and memory ordering 8054 stronger than 8055 unordered (this is 8056 termed the 8057 acquire-fence-paired-atomic) 8058 has completed 8059 before invalidating 8060 the cache. This 8061 satisfies the 8062 requirements of 8063 acquire. 8064 - Ensures that all 8065 previous memory 8066 operations have 8067 completed before a 8068 following 8069 global/local/generic 8070 store 8071 atomic/atomicrmw 8072 with an equal or 8073 wider sync scope 8074 and memory ordering 8075 stronger than 8076 unordered (this is 8077 termed the 8078 release-fence-paired-atomic). 8079 This satisfies the 8080 requirements of 8081 release. 8082 8083 2. buffer_wbinvl1_vol 8084 8085 - Must happen before 8086 any following 8087 global/generic 8088 load/load 8089 atomic/store/store 8090 atomic/atomicrmw. 8091 - Ensures that 8092 following 8093 loads will not see 8094 stale L1 global data. 8095 MTYPE RW and CC memory will 8096 never be stale in L2 due to 8097 the memory probes. 8098 8099 **Sequential Consistent Atomic** 8100 ------------------------------------------------------------------------------------ 8101 load atomic seq_cst - singlethread - global *Same as corresponding 8102 - wavefront - local load atomic acquire, 8103 - generic except must generated 8104 all instructions even 8105 for OpenCL.* 8106 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8107 - generic 8108 - Use lgkmcnt(0) if not 8109 TgSplit execution mode 8110 and vmcnt(0) if TgSplit 8111 execution mode. 8112 - s_waitcnt lgkmcnt(0) must 8113 happen after 8114 preceding 8115 local/generic load 8116 atomic/store 8117 atomic/atomicrmw 8118 with memory 8119 ordering of seq_cst 8120 and with equal or 8121 wider sync scope. 8122 (Note that seq_cst 8123 fences have their 8124 own s_waitcnt 8125 lgkmcnt(0) and so do 8126 not need to be 8127 considered.) 8128 - s_waitcnt vmcnt(0) 8129 must happen after 8130 preceding 8131 global/generic load 8132 atomic/store 8133 atomic/atomicrmw 8134 with memory 8135 ordering of seq_cst 8136 and with equal or 8137 wider sync scope. 8138 (Note that seq_cst 8139 fences have their 8140 own s_waitcnt 8141 vmcnt(0) and so do 8142 not need to be 8143 considered.) 8144 - Ensures any 8145 preceding 8146 sequential 8147 consistent global/local 8148 memory instructions 8149 have completed 8150 before executing 8151 this sequentially 8152 consistent 8153 instruction. This 8154 prevents reordering 8155 a seq_cst store 8156 followed by a 8157 seq_cst load. (Note 8158 that seq_cst is 8159 stronger than 8160 acquire/release as 8161 the reordering of 8162 load acquire 8163 followed by a store 8164 release is 8165 prevented by the 8166 s_waitcnt of 8167 the release, but 8168 there is nothing 8169 preventing a store 8170 release followed by 8171 load acquire from 8172 completing out of 8173 order. The s_waitcnt 8174 could be placed after 8175 seq_store or before 8176 the seq_load. We 8177 choose the load to 8178 make the s_waitcnt be 8179 as late as possible 8180 so that the store 8181 may have already 8182 completed.) 8183 8184 2. *Following 8185 instructions same as 8186 corresponding load 8187 atomic acquire, 8188 except must generated 8189 all instructions even 8190 for OpenCL.* 8191 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8192 local address space cannot 8193 be used.* 8194 8195 *Same as corresponding 8196 load atomic acquire, 8197 except must generated 8198 all instructions even 8199 for OpenCL.* 8200 8201 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8202 - system - generic vmcnt(0) 8203 8204 - If TgSplit execution mode, 8205 omit lgkmcnt(0). 8206 - Could be split into 8207 separate s_waitcnt 8208 vmcnt(0) 8209 and s_waitcnt 8210 lgkmcnt(0) to allow 8211 them to be 8212 independently moved 8213 according to the 8214 following rules. 8215 - s_waitcnt lgkmcnt(0) 8216 must happen after 8217 preceding 8218 global/generic load 8219 atomic/store 8220 atomic/atomicrmw 8221 with memory 8222 ordering of seq_cst 8223 and with equal or 8224 wider sync scope. 8225 (Note that seq_cst 8226 fences have their 8227 own s_waitcnt 8228 lgkmcnt(0) and so do 8229 not need to be 8230 considered.) 8231 - s_waitcnt vmcnt(0) 8232 must happen after 8233 preceding 8234 global/generic load 8235 atomic/store 8236 atomic/atomicrmw 8237 with memory 8238 ordering of seq_cst 8239 and with equal or 8240 wider sync scope. 8241 (Note that seq_cst 8242 fences have their 8243 own s_waitcnt 8244 vmcnt(0) and so do 8245 not need to be 8246 considered.) 8247 - Ensures any 8248 preceding 8249 sequential 8250 consistent global 8251 memory instructions 8252 have completed 8253 before executing 8254 this sequentially 8255 consistent 8256 instruction. This 8257 prevents reordering 8258 a seq_cst store 8259 followed by a 8260 seq_cst load. (Note 8261 that seq_cst is 8262 stronger than 8263 acquire/release as 8264 the reordering of 8265 load acquire 8266 followed by a store 8267 release is 8268 prevented by the 8269 s_waitcnt of 8270 the release, but 8271 there is nothing 8272 preventing a store 8273 release followed by 8274 load acquire from 8275 completing out of 8276 order. The s_waitcnt 8277 could be placed after 8278 seq_store or before 8279 the seq_load. We 8280 choose the load to 8281 make the s_waitcnt be 8282 as late as possible 8283 so that the store 8284 may have already 8285 completed.) 8286 8287 2. *Following 8288 instructions same as 8289 corresponding load 8290 atomic acquire, 8291 except must generated 8292 all instructions even 8293 for OpenCL.* 8294 store atomic seq_cst - singlethread - global *Same as corresponding 8295 - wavefront - local store atomic release, 8296 - workgroup - generic except must generated 8297 - agent all instructions even 8298 - system for OpenCL.* 8299 atomicrmw seq_cst - singlethread - global *Same as corresponding 8300 - wavefront - local atomicrmw acq_rel, 8301 - workgroup - generic except must generated 8302 - agent all instructions even 8303 - system for OpenCL.* 8304 fence seq_cst - singlethread *none* *Same as corresponding 8305 - wavefront fence acq_rel, 8306 - workgroup except must generated 8307 - agent all instructions even 8308 - system for OpenCL.* 8309 ============ ============ ============== ========== ================================ 8310 8311.. _amdgpu-amdhsa-memory-model-gfx10: 8312 8313Memory Model GFX10 8314++++++++++++++++++ 8315 8316For GFX10: 8317 8318* Each agent has multiple shader arrays (SA). 8319* Each SA has multiple work-group processors (WGP). 8320* Each WGP has multiple compute units (CU). 8321* Each CU has multiple SIMDs that execute wavefronts. 8322* The wavefronts for a single work-group are executed in the same 8323 WGP. In CU wavefront execution mode the wavefronts may be executed by 8324 different SIMDs in the same CU. In WGP wavefront execution mode the 8325 wavefronts may be executed by different SIMDs in different CUs in the same 8326 WGP. 8327* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8328 executing on it. 8329* All LDS operations of a WGP are performed as wavefront wide operations in a 8330 global order and involve no caching. Completion is reported to a wavefront in 8331 execution order. 8332* The LDS memory has multiple request queues shared by the SIMDs of a 8333 WGP. Therefore, the LDS operations performed by different wavefronts of a 8334 work-group can be reordered relative to each other, which can result in 8335 reordering the visibility of vector memory operations with respect to LDS 8336 operations of other wavefronts in the same work-group. A ``s_waitcnt 8337 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8338 vector memory operations between wavefronts of a work-group, but not between 8339 operations performed by the same wavefront. 8340* The vector memory operations are performed as wavefront wide operations. 8341 Completion of load/store/sample operations are reported to a wavefront in 8342 execution order of other load/store/sample operations performed by that 8343 wavefront. 8344* The vector memory operations access a vector L0 cache. There is a single L0 8345 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8346 special action is required for coherence between the lanes of a single 8347 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8348 wavefronts executing in the same work-group as they may be executing on SIMDs 8349 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8350 required for coherence between wavefronts executing in different work-groups 8351 as they may be executing on different WGPs. 8352* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8353 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8354 operations are used in a restricted way so do not impact the memory model. See 8355 :ref:`amdgpu-amdhsa-memory-spaces`. 8356* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8357 the same SA. Therefore, no special action is required for coherence between 8358 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8359 required for coherence between wavefronts executing in different work-groups 8360 as they may be executing on different SAs that access different L1s. 8361* The L1 caches have independent quadrants to service disjoint ranges of virtual 8362 addresses. 8363* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8364 vector and scalar memory operations performed by different wavefronts, whether 8365 executing in the same or different work-groups (which may be executing on 8366 different CUs accessing different L0s), can be reordered relative to each 8367 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8368 synchronization between vector memory operations of different wavefronts. It 8369 ensures a previous vector memory operation has completed before executing a 8370 subsequent vector memory or LDS operation and so can be used to meet the 8371 requirements of acquire, release and sequential consistency. 8372* The L1 caches use an L2 cache shared by all SAs on the same agent. 8373* The L2 cache has independent channels to service disjoint ranges of virtual 8374 addresses. 8375* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8376 quadrant has a separate request queue per L2 channel. Therefore, the vector 8377 and scalar memory operations performed by wavefronts executing in different 8378 work-groups (which may be executing on different SAs) of an agent can be 8379 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8380 required to ensure synchronization between vector memory operations of 8381 different SAs. It ensures a previous vector memory operation has completed 8382 before executing a subsequent vector memory and so can be used to meet the 8383 requirements of acquire, release and sequential consistency. 8384* The L2 cache can be kept coherent with other agents on some targets, or ranges 8385 of virtual addresses can be set up to bypass it to ensure system coherence. 8386 8387Scalar memory operations are only used to access memory that is proven to not 8388change during the execution of the kernel dispatch. This includes constant 8389address space and global address space for program scope ``const`` variables. 8390Therefore, the kernel machine code does not have to maintain the scalar cache to 8391ensure it is coherent with the vector caches. The scalar and vector caches are 8392invalidated between kernel dispatches by CP since constant address space data 8393may change between kernel dispatch executions. See 8394:ref:`amdgpu-amdhsa-memory-spaces`. 8395 8396The one exception is if scalar writes are used to spill SGPR registers. In this 8397case the AMDGPU backend ensures the memory location used to spill is never 8398accessed by vector memory operations at the same time. If scalar writes are used 8399then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8400return since the locations may be used for vector memory instructions by a 8401future wavefront that uses the same scratch area, or a function call that 8402creates a frame at the same address, respectively. There is no need for a 8403``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8404 8405For kernarg backing memory: 8406 8407* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8408* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8409 needing to invalidate the L2 cache. 8410* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8411 so the L2 cache will be coherent with the CPU and other agents. 8412 8413Scratch backing memory (which is used for the private address space) is accessed 8414with MTYPE NC (non-coherent). Since the private address space is only accessed 8415by a single thread, and is always write-before-read, there is never a need to 8416invalidate these entries from the L0 or L1 caches. 8417 8418Wavefronts are executed in native mode with in-order reporting of loads and 8419sample instructions. In this mode vmcnt reports completion of load, atomic with 8420return and sample instructions in order, and the vscnt reports the completion of 8421store and atomic without return in order. See ``MEM_ORDERED`` field in 8422:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8423 8424Wavefronts can be executed in WGP or CU wavefront execution mode: 8425 8426* In WGP wavefront execution mode the wavefronts of a work-group are executed 8427 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8428 CU L0 caches is required for work-group synchronization. Also accesses to L1 8429 at work-group scope need to be explicitly ordered as the accesses from 8430 different CUs are not ordered. 8431* In CU wavefront execution mode the wavefronts of a work-group are executed on 8432 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8433 the work-group access the same L0 which in turn ensures L1 accesses are 8434 ordered and so do not require explicit management of the caches for 8435 work-group synchronization. 8436 8437See ``WGP_MODE`` field in 8438:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8439:ref:`amdgpu-target-features`. 8440 8441The code sequences used to implement the memory model for GFX10 are defined in 8442table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8443 8444 .. table:: AMDHSA Memory Model Code Sequences GFX10 8445 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8446 8447 ============ ============ ============== ========== ================================ 8448 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8449 Ordering Sync Scope Address GFX10 8450 Space 8451 ============ ============ ============== ========== ================================ 8452 **Non-Atomic** 8453 ------------------------------------------------------------------------------------ 8454 load *none* *none* - global - !volatile & !nontemporal 8455 - generic 8456 - private 1. buffer/global/flat_load 8457 - constant 8458 - !volatile & nontemporal 8459 8460 1. buffer/global/flat_load 8461 slc=1 8462 8463 - volatile 8464 8465 1. buffer/global/flat_load 8466 glc=1 dlc=1 8467 2. s_waitcnt vmcnt(0) 8468 8469 - Must happen before 8470 any following volatile 8471 global/generic 8472 load/store. 8473 - Ensures that 8474 volatile 8475 operations to 8476 different 8477 addresses will not 8478 be reordered by 8479 hardware. 8480 8481 load *none* *none* - local 1. ds_load 8482 store *none* *none* - global - !volatile & !nontemporal 8483 - generic 8484 - private 1. buffer/global/flat_store 8485 - constant 8486 - !volatile & nontemporal 8487 8488 1. buffer/global/flat_store 8489 slc=1 8490 8491 - volatile 8492 8493 1. buffer/global/flat_store 8494 2. s_waitcnt vscnt(0) 8495 8496 - Must happen before 8497 any following volatile 8498 global/generic 8499 load/store. 8500 - Ensures that 8501 volatile 8502 operations to 8503 different 8504 addresses will not 8505 be reordered by 8506 hardware. 8507 8508 store *none* *none* - local 1. ds_store 8509 **Unordered Atomic** 8510 ------------------------------------------------------------------------------------ 8511 load atomic unordered *any* *any* *Same as non-atomic*. 8512 store atomic unordered *any* *any* *Same as non-atomic*. 8513 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8514 **Monotonic Atomic** 8515 ------------------------------------------------------------------------------------ 8516 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8517 - wavefront - generic 8518 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8519 - generic glc=1 8520 8521 - If CU wavefront execution 8522 mode, omit glc=1. 8523 8524 load atomic monotonic - singlethread - local 1. ds_load 8525 - wavefront 8526 - workgroup 8527 load atomic monotonic - agent - global 1. buffer/global/flat_load 8528 - system - generic glc=1 dlc=1 8529 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8530 - wavefront - generic 8531 - workgroup 8532 - agent 8533 - system 8534 store atomic monotonic - singlethread - local 1. ds_store 8535 - wavefront 8536 - workgroup 8537 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8538 - wavefront - generic 8539 - workgroup 8540 - agent 8541 - system 8542 atomicrmw monotonic - singlethread - local 1. ds_atomic 8543 - wavefront 8544 - workgroup 8545 **Acquire Atomic** 8546 ------------------------------------------------------------------------------------ 8547 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8548 - wavefront - local 8549 - generic 8550 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8551 8552 - If CU wavefront execution 8553 mode, omit glc=1. 8554 8555 2. s_waitcnt vmcnt(0) 8556 8557 - If CU wavefront execution 8558 mode, omit. 8559 - Must happen before 8560 the following buffer_gl0_inv 8561 and before any following 8562 global/generic 8563 load/load 8564 atomic/store/store 8565 atomic/atomicrmw. 8566 8567 3. buffer_gl0_inv 8568 8569 - If CU wavefront execution 8570 mode, omit. 8571 - Ensures that 8572 following 8573 loads will not see 8574 stale data. 8575 8576 load atomic acquire - workgroup - local 1. ds_load 8577 2. s_waitcnt lgkmcnt(0) 8578 8579 - If OpenCL, omit. 8580 - Must happen before 8581 the following buffer_gl0_inv 8582 and before any following 8583 global/generic load/load 8584 atomic/store/store 8585 atomic/atomicrmw. 8586 - Ensures any 8587 following global 8588 data read is no 8589 older than the local load 8590 atomic value being 8591 acquired. 8592 8593 3. buffer_gl0_inv 8594 8595 - If CU wavefront execution 8596 mode, omit. 8597 - If OpenCL, omit. 8598 - Ensures that 8599 following 8600 loads will not see 8601 stale data. 8602 8603 load atomic acquire - workgroup - generic 1. flat_load glc=1 8604 8605 - If CU wavefront execution 8606 mode, omit glc=1. 8607 8608 2. s_waitcnt lgkmcnt(0) & 8609 vmcnt(0) 8610 8611 - If CU wavefront execution 8612 mode, omit vmcnt(0). 8613 - If OpenCL, omit 8614 lgkmcnt(0). 8615 - Must happen before 8616 the following 8617 buffer_gl0_inv and any 8618 following global/generic 8619 load/load 8620 atomic/store/store 8621 atomic/atomicrmw. 8622 - Ensures any 8623 following global 8624 data read is no 8625 older than a local load 8626 atomic value being 8627 acquired. 8628 8629 3. buffer_gl0_inv 8630 8631 - If CU wavefront execution 8632 mode, omit. 8633 - Ensures that 8634 following 8635 loads will not see 8636 stale data. 8637 8638 load atomic acquire - agent - global 1. buffer/global_load 8639 - system glc=1 dlc=1 8640 2. s_waitcnt vmcnt(0) 8641 8642 - Must happen before 8643 following 8644 buffer_gl*_inv. 8645 - Ensures the load 8646 has completed 8647 before invalidating 8648 the caches. 8649 8650 3. buffer_gl0_inv; 8651 buffer_gl1_inv 8652 8653 - Must happen before 8654 any following 8655 global/generic 8656 load/load 8657 atomic/atomicrmw. 8658 - Ensures that 8659 following 8660 loads will not see 8661 stale global data. 8662 8663 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8664 - system 2. s_waitcnt vmcnt(0) & 8665 lgkmcnt(0) 8666 8667 - If OpenCL omit 8668 lgkmcnt(0). 8669 - Must happen before 8670 following 8671 buffer_gl*_invl. 8672 - Ensures the flat_load 8673 has completed 8674 before invalidating 8675 the caches. 8676 8677 3. buffer_gl0_inv; 8678 buffer_gl1_inv 8679 8680 - Must happen before 8681 any following 8682 global/generic 8683 load/load 8684 atomic/atomicrmw. 8685 - Ensures that 8686 following loads 8687 will not see stale 8688 global data. 8689 8690 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8691 - wavefront - local 8692 - generic 8693 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8694 2. s_waitcnt vm/vscnt(0) 8695 8696 - If CU wavefront execution 8697 mode, omit. 8698 - Use vmcnt(0) if atomic with 8699 return and vscnt(0) if 8700 atomic with no-return. 8701 - Must happen before 8702 the following buffer_gl0_inv 8703 and before any following 8704 global/generic 8705 load/load 8706 atomic/store/store 8707 atomic/atomicrmw. 8708 8709 3. buffer_gl0_inv 8710 8711 - If CU wavefront execution 8712 mode, omit. 8713 - Ensures that 8714 following 8715 loads will not see 8716 stale data. 8717 8718 atomicrmw acquire - workgroup - local 1. ds_atomic 8719 2. s_waitcnt lgkmcnt(0) 8720 8721 - If OpenCL, omit. 8722 - Must happen before 8723 the following 8724 buffer_gl0_inv. 8725 - Ensures any 8726 following global 8727 data read is no 8728 older than the local 8729 atomicrmw value 8730 being acquired. 8731 8732 3. buffer_gl0_inv 8733 8734 - If OpenCL omit. 8735 - Ensures that 8736 following 8737 loads will not see 8738 stale data. 8739 8740 atomicrmw acquire - workgroup - generic 1. flat_atomic 8741 2. s_waitcnt lgkmcnt(0) & 8742 vm/vscnt(0) 8743 8744 - If CU wavefront execution 8745 mode, omit vm/vscnt(0). 8746 - If OpenCL, omit lgkmcnt(0). 8747 - Use vmcnt(0) if atomic with 8748 return and vscnt(0) if 8749 atomic with no-return. 8750 - Must happen before 8751 the following 8752 buffer_gl0_inv. 8753 - Ensures any 8754 following global 8755 data read is no 8756 older than a local 8757 atomicrmw value 8758 being acquired. 8759 8760 3. buffer_gl0_inv 8761 8762 - If CU wavefront execution 8763 mode, omit. 8764 - Ensures that 8765 following 8766 loads will not see 8767 stale data. 8768 8769 atomicrmw acquire - agent - global 1. buffer/global_atomic 8770 - system 2. s_waitcnt vm/vscnt(0) 8771 8772 - Use vmcnt(0) if atomic with 8773 return and vscnt(0) if 8774 atomic with no-return. 8775 - Must happen before 8776 following 8777 buffer_gl*_inv. 8778 - Ensures the 8779 atomicrmw has 8780 completed before 8781 invalidating the 8782 caches. 8783 8784 3. buffer_gl0_inv; 8785 buffer_gl1_inv 8786 8787 - Must happen before 8788 any following 8789 global/generic 8790 load/load 8791 atomic/atomicrmw. 8792 - Ensures that 8793 following loads 8794 will not see stale 8795 global data. 8796 8797 atomicrmw acquire - agent - generic 1. flat_atomic 8798 - system 2. s_waitcnt vm/vscnt(0) & 8799 lgkmcnt(0) 8800 8801 - If OpenCL, omit 8802 lgkmcnt(0). 8803 - Use vmcnt(0) if atomic with 8804 return and vscnt(0) if 8805 atomic with no-return. 8806 - Must happen before 8807 following 8808 buffer_gl*_inv. 8809 - Ensures the 8810 atomicrmw has 8811 completed before 8812 invalidating the 8813 caches. 8814 8815 3. buffer_gl0_inv; 8816 buffer_gl1_inv 8817 8818 - Must happen before 8819 any following 8820 global/generic 8821 load/load 8822 atomic/atomicrmw. 8823 - Ensures that 8824 following loads 8825 will not see stale 8826 global data. 8827 8828 fence acquire - singlethread *none* *none* 8829 - wavefront 8830 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 8831 vmcnt(0) & vscnt(0) 8832 8833 - If CU wavefront execution 8834 mode, omit vmcnt(0) and 8835 vscnt(0). 8836 - If OpenCL and 8837 address space is 8838 not generic, omit 8839 lgkmcnt(0). 8840 - If OpenCL and 8841 address space is 8842 local, omit 8843 vmcnt(0) and vscnt(0). 8844 - However, since LLVM 8845 currently has no 8846 address space on 8847 the fence need to 8848 conservatively 8849 always generate. If 8850 fence had an 8851 address space then 8852 set to address 8853 space of OpenCL 8854 fence flag, or to 8855 generic if both 8856 local and global 8857 flags are 8858 specified. 8859 - Could be split into 8860 separate s_waitcnt 8861 vmcnt(0), s_waitcnt 8862 vscnt(0) and s_waitcnt 8863 lgkmcnt(0) to allow 8864 them to be 8865 independently moved 8866 according to the 8867 following rules. 8868 - s_waitcnt vmcnt(0) 8869 must happen after 8870 any preceding 8871 global/generic load 8872 atomic/ 8873 atomicrmw-with-return-value 8874 with an equal or 8875 wider sync scope 8876 and memory ordering 8877 stronger than 8878 unordered (this is 8879 termed the 8880 fence-paired-atomic). 8881 - s_waitcnt vscnt(0) 8882 must happen after 8883 any preceding 8884 global/generic 8885 atomicrmw-no-return-value 8886 with an equal or 8887 wider sync scope 8888 and memory ordering 8889 stronger than 8890 unordered (this is 8891 termed the 8892 fence-paired-atomic). 8893 - s_waitcnt lgkmcnt(0) 8894 must happen after 8895 any preceding 8896 local/generic load 8897 atomic/atomicrmw 8898 with an equal or 8899 wider sync scope 8900 and memory ordering 8901 stronger than 8902 unordered (this is 8903 termed the 8904 fence-paired-atomic). 8905 - Must happen before 8906 the following 8907 buffer_gl0_inv. 8908 - Ensures that the 8909 fence-paired atomic 8910 has completed 8911 before invalidating 8912 the 8913 cache. Therefore 8914 any following 8915 locations read must 8916 be no older than 8917 the value read by 8918 the 8919 fence-paired-atomic. 8920 8921 3. buffer_gl0_inv 8922 8923 - If CU wavefront execution 8924 mode, omit. 8925 - Ensures that 8926 following 8927 loads will not see 8928 stale data. 8929 8930 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 8931 - system vmcnt(0) & vscnt(0) 8932 8933 - If OpenCL and 8934 address space is 8935 not generic, omit 8936 lgkmcnt(0). 8937 - If OpenCL and 8938 address space is 8939 local, omit 8940 vmcnt(0) and vscnt(0). 8941 - However, since LLVM 8942 currently has no 8943 address space on 8944 the fence need to 8945 conservatively 8946 always generate 8947 (see comment for 8948 previous fence). 8949 - Could be split into 8950 separate s_waitcnt 8951 vmcnt(0), s_waitcnt 8952 vscnt(0) and s_waitcnt 8953 lgkmcnt(0) to allow 8954 them to be 8955 independently moved 8956 according to the 8957 following rules. 8958 - s_waitcnt vmcnt(0) 8959 must happen after 8960 any preceding 8961 global/generic load 8962 atomic/ 8963 atomicrmw-with-return-value 8964 with an equal or 8965 wider sync scope 8966 and memory ordering 8967 stronger than 8968 unordered (this is 8969 termed the 8970 fence-paired-atomic). 8971 - s_waitcnt vscnt(0) 8972 must happen after 8973 any preceding 8974 global/generic 8975 atomicrmw-no-return-value 8976 with an equal or 8977 wider sync scope 8978 and memory ordering 8979 stronger than 8980 unordered (this is 8981 termed the 8982 fence-paired-atomic). 8983 - s_waitcnt lgkmcnt(0) 8984 must happen after 8985 any preceding 8986 local/generic load 8987 atomic/atomicrmw 8988 with an equal or 8989 wider sync scope 8990 and memory ordering 8991 stronger than 8992 unordered (this is 8993 termed the 8994 fence-paired-atomic). 8995 - Must happen before 8996 the following 8997 buffer_gl*_inv. 8998 - Ensures that the 8999 fence-paired atomic 9000 has completed 9001 before invalidating 9002 the 9003 caches. Therefore 9004 any following 9005 locations read must 9006 be no older than 9007 the value read by 9008 the 9009 fence-paired-atomic. 9010 9011 2. buffer_gl0_inv; 9012 buffer_gl1_inv 9013 9014 - Must happen before any 9015 following global/generic 9016 load/load 9017 atomic/store/store 9018 atomic/atomicrmw. 9019 - Ensures that 9020 following loads 9021 will not see stale 9022 global data. 9023 9024 **Release Atomic** 9025 ------------------------------------------------------------------------------------ 9026 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9027 - wavefront - local 9028 - generic 9029 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9030 - generic vmcnt(0) & vscnt(0) 9031 9032 - If CU wavefront execution 9033 mode, omit vmcnt(0) and 9034 vscnt(0). 9035 - If OpenCL, omit 9036 lgkmcnt(0). 9037 - Could be split into 9038 separate s_waitcnt 9039 vmcnt(0), s_waitcnt 9040 vscnt(0) and s_waitcnt 9041 lgkmcnt(0) to allow 9042 them to be 9043 independently moved 9044 according to the 9045 following rules. 9046 - s_waitcnt vmcnt(0) 9047 must happen after 9048 any preceding 9049 global/generic load/load 9050 atomic/ 9051 atomicrmw-with-return-value. 9052 - s_waitcnt vscnt(0) 9053 must happen after 9054 any preceding 9055 global/generic 9056 store/store 9057 atomic/ 9058 atomicrmw-no-return-value. 9059 - s_waitcnt lgkmcnt(0) 9060 must happen after 9061 any preceding 9062 local/generic 9063 load/store/load 9064 atomic/store 9065 atomic/atomicrmw. 9066 - Must happen before 9067 the following 9068 store. 9069 - Ensures that all 9070 memory operations 9071 have 9072 completed before 9073 performing the 9074 store that is being 9075 released. 9076 9077 2. buffer/global/flat_store 9078 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9079 9080 - If CU wavefront execution 9081 mode, omit. 9082 - If OpenCL, omit. 9083 - Could be split into 9084 separate s_waitcnt 9085 vmcnt(0) and s_waitcnt 9086 vscnt(0) to allow 9087 them to be 9088 independently moved 9089 according to the 9090 following rules. 9091 - s_waitcnt vmcnt(0) 9092 must happen after 9093 any preceding 9094 global/generic load/load 9095 atomic/ 9096 atomicrmw-with-return-value. 9097 - s_waitcnt vscnt(0) 9098 must happen after 9099 any preceding 9100 global/generic 9101 store/store atomic/ 9102 atomicrmw-no-return-value. 9103 - Must happen before 9104 the following 9105 store. 9106 - Ensures that all 9107 global memory 9108 operations have 9109 completed before 9110 performing the 9111 store that is being 9112 released. 9113 9114 2. ds_store 9115 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9116 - system - generic vmcnt(0) & vscnt(0) 9117 9118 - If OpenCL and 9119 address space is 9120 not generic, omit 9121 lgkmcnt(0). 9122 - Could be split into 9123 separate s_waitcnt 9124 vmcnt(0), s_waitcnt vscnt(0) 9125 and s_waitcnt 9126 lgkmcnt(0) to allow 9127 them to be 9128 independently moved 9129 according to the 9130 following rules. 9131 - s_waitcnt vmcnt(0) 9132 must happen after 9133 any preceding 9134 global/generic 9135 load/load 9136 atomic/ 9137 atomicrmw-with-return-value. 9138 - s_waitcnt vscnt(0) 9139 must happen after 9140 any preceding 9141 global/generic 9142 store/store atomic/ 9143 atomicrmw-no-return-value. 9144 - s_waitcnt lgkmcnt(0) 9145 must happen after 9146 any preceding 9147 local/generic 9148 load/store/load 9149 atomic/store 9150 atomic/atomicrmw. 9151 - Must happen before 9152 the following 9153 store. 9154 - Ensures that all 9155 memory operations 9156 have 9157 completed before 9158 performing the 9159 store that is being 9160 released. 9161 9162 2. buffer/global/flat_store 9163 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9164 - wavefront - local 9165 - generic 9166 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9167 - generic vmcnt(0) & vscnt(0) 9168 9169 - If CU wavefront execution 9170 mode, omit vmcnt(0) and 9171 vscnt(0). 9172 - If OpenCL, omit lgkmcnt(0). 9173 - Could be split into 9174 separate s_waitcnt 9175 vmcnt(0), s_waitcnt 9176 vscnt(0) and s_waitcnt 9177 lgkmcnt(0) to allow 9178 them to be 9179 independently moved 9180 according to the 9181 following rules. 9182 - s_waitcnt vmcnt(0) 9183 must happen after 9184 any preceding 9185 global/generic load/load 9186 atomic/ 9187 atomicrmw-with-return-value. 9188 - s_waitcnt vscnt(0) 9189 must happen after 9190 any preceding 9191 global/generic 9192 store/store 9193 atomic/ 9194 atomicrmw-no-return-value. 9195 - s_waitcnt lgkmcnt(0) 9196 must happen after 9197 any preceding 9198 local/generic 9199 load/store/load 9200 atomic/store 9201 atomic/atomicrmw. 9202 - Must happen before 9203 the following 9204 atomicrmw. 9205 - Ensures that all 9206 memory operations 9207 have 9208 completed before 9209 performing the 9210 atomicrmw that is 9211 being released. 9212 9213 2. buffer/global/flat_atomic 9214 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9215 9216 - If CU wavefront execution 9217 mode, omit. 9218 - If OpenCL, omit. 9219 - Could be split into 9220 separate s_waitcnt 9221 vmcnt(0) and s_waitcnt 9222 vscnt(0) to allow 9223 them to be 9224 independently moved 9225 according to the 9226 following rules. 9227 - s_waitcnt vmcnt(0) 9228 must happen after 9229 any preceding 9230 global/generic load/load 9231 atomic/ 9232 atomicrmw-with-return-value. 9233 - s_waitcnt vscnt(0) 9234 must happen after 9235 any preceding 9236 global/generic 9237 store/store atomic/ 9238 atomicrmw-no-return-value. 9239 - Must happen before 9240 the following 9241 store. 9242 - Ensures that all 9243 global memory 9244 operations have 9245 completed before 9246 performing the 9247 store that is being 9248 released. 9249 9250 2. ds_atomic 9251 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9252 - system - generic vmcnt(0) & vscnt(0) 9253 9254 - If OpenCL, omit 9255 lgkmcnt(0). 9256 - Could be split into 9257 separate s_waitcnt 9258 vmcnt(0), s_waitcnt 9259 vscnt(0) and s_waitcnt 9260 lgkmcnt(0) to allow 9261 them to be 9262 independently moved 9263 according to the 9264 following rules. 9265 - s_waitcnt vmcnt(0) 9266 must happen after 9267 any preceding 9268 global/generic 9269 load/load atomic/ 9270 atomicrmw-with-return-value. 9271 - s_waitcnt vscnt(0) 9272 must happen after 9273 any preceding 9274 global/generic 9275 store/store atomic/ 9276 atomicrmw-no-return-value. 9277 - s_waitcnt lgkmcnt(0) 9278 must happen after 9279 any preceding 9280 local/generic 9281 load/store/load 9282 atomic/store 9283 atomic/atomicrmw. 9284 - Must happen before 9285 the following 9286 atomicrmw. 9287 - Ensures that all 9288 memory operations 9289 to global and local 9290 have completed 9291 before performing 9292 the atomicrmw that 9293 is being released. 9294 9295 2. buffer/global/flat_atomic 9296 fence release - singlethread *none* *none* 9297 - wavefront 9298 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9299 vmcnt(0) & vscnt(0) 9300 9301 - If CU wavefront execution 9302 mode, omit vmcnt(0) and 9303 vscnt(0). 9304 - If OpenCL and 9305 address space is 9306 not generic, omit 9307 lgkmcnt(0). 9308 - If OpenCL and 9309 address space is 9310 local, omit 9311 vmcnt(0) and vscnt(0). 9312 - However, since LLVM 9313 currently has no 9314 address space on 9315 the fence need to 9316 conservatively 9317 always generate. If 9318 fence had an 9319 address space then 9320 set to address 9321 space of OpenCL 9322 fence flag, or to 9323 generic if both 9324 local and global 9325 flags are 9326 specified. 9327 - Could be split into 9328 separate s_waitcnt 9329 vmcnt(0), s_waitcnt 9330 vscnt(0) and s_waitcnt 9331 lgkmcnt(0) to allow 9332 them to be 9333 independently moved 9334 according to the 9335 following rules. 9336 - s_waitcnt vmcnt(0) 9337 must happen after 9338 any preceding 9339 global/generic 9340 load/load 9341 atomic/ 9342 atomicrmw-with-return-value. 9343 - s_waitcnt vscnt(0) 9344 must happen after 9345 any preceding 9346 global/generic 9347 store/store atomic/ 9348 atomicrmw-no-return-value. 9349 - s_waitcnt lgkmcnt(0) 9350 must happen after 9351 any preceding 9352 local/generic 9353 load/store/load 9354 atomic/store atomic/ 9355 atomicrmw. 9356 - Must happen before 9357 any following store 9358 atomic/atomicrmw 9359 with an equal or 9360 wider sync scope 9361 and memory ordering 9362 stronger than 9363 unordered (this is 9364 termed the 9365 fence-paired-atomic). 9366 - Ensures that all 9367 memory operations 9368 have 9369 completed before 9370 performing the 9371 following 9372 fence-paired-atomic. 9373 9374 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9375 - system vmcnt(0) & vscnt(0) 9376 9377 - If OpenCL and 9378 address space is 9379 not generic, omit 9380 lgkmcnt(0). 9381 - If OpenCL and 9382 address space is 9383 local, omit 9384 vmcnt(0) and vscnt(0). 9385 - However, since LLVM 9386 currently has no 9387 address space on 9388 the fence need to 9389 conservatively 9390 always generate. If 9391 fence had an 9392 address space then 9393 set to address 9394 space of OpenCL 9395 fence flag, or to 9396 generic if both 9397 local and global 9398 flags are 9399 specified. 9400 - Could be split into 9401 separate s_waitcnt 9402 vmcnt(0), s_waitcnt 9403 vscnt(0) and s_waitcnt 9404 lgkmcnt(0) to allow 9405 them to be 9406 independently moved 9407 according to the 9408 following rules. 9409 - s_waitcnt vmcnt(0) 9410 must happen after 9411 any preceding 9412 global/generic 9413 load/load atomic/ 9414 atomicrmw-with-return-value. 9415 - s_waitcnt vscnt(0) 9416 must happen after 9417 any preceding 9418 global/generic 9419 store/store atomic/ 9420 atomicrmw-no-return-value. 9421 - s_waitcnt lgkmcnt(0) 9422 must happen after 9423 any preceding 9424 local/generic 9425 load/store/load 9426 atomic/store 9427 atomic/atomicrmw. 9428 - Must happen before 9429 any following store 9430 atomic/atomicrmw 9431 with an equal or 9432 wider sync scope 9433 and memory ordering 9434 stronger than 9435 unordered (this is 9436 termed the 9437 fence-paired-atomic). 9438 - Ensures that all 9439 memory operations 9440 have 9441 completed before 9442 performing the 9443 following 9444 fence-paired-atomic. 9445 9446 **Acquire-Release Atomic** 9447 ------------------------------------------------------------------------------------ 9448 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9449 - wavefront - local 9450 - generic 9451 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9452 vmcnt(0) & vscnt(0) 9453 9454 - If CU wavefront execution 9455 mode, omit vmcnt(0) and 9456 vscnt(0). 9457 - If OpenCL, omit 9458 lgkmcnt(0). 9459 - Must happen after 9460 any preceding 9461 local/generic 9462 load/store/load 9463 atomic/store 9464 atomic/atomicrmw. 9465 - Could be split into 9466 separate s_waitcnt 9467 vmcnt(0), s_waitcnt 9468 vscnt(0), and s_waitcnt 9469 lgkmcnt(0) to allow 9470 them to be 9471 independently moved 9472 according to the 9473 following rules. 9474 - s_waitcnt vmcnt(0) 9475 must happen after 9476 any preceding 9477 global/generic load/load 9478 atomic/ 9479 atomicrmw-with-return-value. 9480 - s_waitcnt vscnt(0) 9481 must happen after 9482 any preceding 9483 global/generic 9484 store/store 9485 atomic/ 9486 atomicrmw-no-return-value. 9487 - s_waitcnt lgkmcnt(0) 9488 must happen after 9489 any preceding 9490 local/generic 9491 load/store/load 9492 atomic/store 9493 atomic/atomicrmw. 9494 - Must happen before 9495 the following 9496 atomicrmw. 9497 - Ensures that all 9498 memory operations 9499 have 9500 completed before 9501 performing the 9502 atomicrmw that is 9503 being released. 9504 9505 2. buffer/global_atomic 9506 3. s_waitcnt vm/vscnt(0) 9507 9508 - If CU wavefront execution 9509 mode, omit. 9510 - Use vmcnt(0) if atomic with 9511 return and vscnt(0) if 9512 atomic with no-return. 9513 - Must happen before 9514 the following 9515 buffer_gl0_inv. 9516 - Ensures any 9517 following global 9518 data read is no 9519 older than the 9520 atomicrmw value 9521 being acquired. 9522 9523 4. buffer_gl0_inv 9524 9525 - If CU wavefront execution 9526 mode, omit. 9527 - Ensures that 9528 following 9529 loads will not see 9530 stale data. 9531 9532 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9533 9534 - If CU wavefront execution 9535 mode, omit. 9536 - If OpenCL, omit. 9537 - Could be split into 9538 separate s_waitcnt 9539 vmcnt(0) and s_waitcnt 9540 vscnt(0) to allow 9541 them to be 9542 independently moved 9543 according to the 9544 following rules. 9545 - s_waitcnt vmcnt(0) 9546 must happen after 9547 any preceding 9548 global/generic load/load 9549 atomic/ 9550 atomicrmw-with-return-value. 9551 - s_waitcnt vscnt(0) 9552 must happen after 9553 any preceding 9554 global/generic 9555 store/store atomic/ 9556 atomicrmw-no-return-value. 9557 - Must happen before 9558 the following 9559 store. 9560 - Ensures that all 9561 global memory 9562 operations have 9563 completed before 9564 performing the 9565 store that is being 9566 released. 9567 9568 2. ds_atomic 9569 3. s_waitcnt lgkmcnt(0) 9570 9571 - If OpenCL, omit. 9572 - Must happen before 9573 the following 9574 buffer_gl0_inv. 9575 - Ensures any 9576 following global 9577 data read is no 9578 older than the local load 9579 atomic value being 9580 acquired. 9581 9582 4. buffer_gl0_inv 9583 9584 - If CU wavefront execution 9585 mode, omit. 9586 - If OpenCL omit. 9587 - Ensures that 9588 following 9589 loads will not see 9590 stale data. 9591 9592 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9593 vmcnt(0) & vscnt(0) 9594 9595 - If CU wavefront execution 9596 mode, omit vmcnt(0) and 9597 vscnt(0). 9598 - If OpenCL, omit lgkmcnt(0). 9599 - Could be split into 9600 separate s_waitcnt 9601 vmcnt(0), s_waitcnt 9602 vscnt(0) and s_waitcnt 9603 lgkmcnt(0) to allow 9604 them to be 9605 independently moved 9606 according to the 9607 following rules. 9608 - s_waitcnt vmcnt(0) 9609 must happen after 9610 any preceding 9611 global/generic load/load 9612 atomic/ 9613 atomicrmw-with-return-value. 9614 - s_waitcnt vscnt(0) 9615 must happen after 9616 any preceding 9617 global/generic 9618 store/store 9619 atomic/ 9620 atomicrmw-no-return-value. 9621 - s_waitcnt lgkmcnt(0) 9622 must happen after 9623 any preceding 9624 local/generic 9625 load/store/load 9626 atomic/store 9627 atomic/atomicrmw. 9628 - Must happen before 9629 the following 9630 atomicrmw. 9631 - Ensures that all 9632 memory operations 9633 have 9634 completed before 9635 performing the 9636 atomicrmw that is 9637 being released. 9638 9639 2. flat_atomic 9640 3. s_waitcnt lgkmcnt(0) & 9641 vmcnt(0) & vscnt(0) 9642 9643 - If CU wavefront execution 9644 mode, omit vmcnt(0) and 9645 vscnt(0). 9646 - If OpenCL, omit lgkmcnt(0). 9647 - Must happen before 9648 the following 9649 buffer_gl0_inv. 9650 - Ensures any 9651 following global 9652 data read is no 9653 older than the load 9654 atomic value being 9655 acquired. 9656 9657 3. buffer_gl0_inv 9658 9659 - If CU wavefront execution 9660 mode, omit. 9661 - Ensures that 9662 following 9663 loads will not see 9664 stale data. 9665 9666 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9667 - system vmcnt(0) & vscnt(0) 9668 9669 - If OpenCL, omit 9670 lgkmcnt(0). 9671 - Could be split into 9672 separate s_waitcnt 9673 vmcnt(0), s_waitcnt 9674 vscnt(0) and s_waitcnt 9675 lgkmcnt(0) to allow 9676 them to be 9677 independently moved 9678 according to the 9679 following rules. 9680 - s_waitcnt vmcnt(0) 9681 must happen after 9682 any preceding 9683 global/generic 9684 load/load atomic/ 9685 atomicrmw-with-return-value. 9686 - s_waitcnt vscnt(0) 9687 must happen after 9688 any preceding 9689 global/generic 9690 store/store atomic/ 9691 atomicrmw-no-return-value. 9692 - s_waitcnt lgkmcnt(0) 9693 must happen after 9694 any preceding 9695 local/generic 9696 load/store/load 9697 atomic/store 9698 atomic/atomicrmw. 9699 - Must happen before 9700 the following 9701 atomicrmw. 9702 - Ensures that all 9703 memory operations 9704 to global have 9705 completed before 9706 performing the 9707 atomicrmw that is 9708 being released. 9709 9710 2. buffer/global_atomic 9711 3. s_waitcnt vm/vscnt(0) 9712 9713 - Use vmcnt(0) if atomic with 9714 return and vscnt(0) if 9715 atomic with no-return. 9716 - Must happen before 9717 following 9718 buffer_gl*_inv. 9719 - Ensures the 9720 atomicrmw has 9721 completed before 9722 invalidating the 9723 caches. 9724 9725 4. buffer_gl0_inv; 9726 buffer_gl1_inv 9727 9728 - Must happen before 9729 any following 9730 global/generic 9731 load/load 9732 atomic/atomicrmw. 9733 - Ensures that 9734 following loads 9735 will not see stale 9736 global data. 9737 9738 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9739 - system vmcnt(0) & vscnt(0) 9740 9741 - If OpenCL, omit 9742 lgkmcnt(0). 9743 - Could be split into 9744 separate s_waitcnt 9745 vmcnt(0), s_waitcnt 9746 vscnt(0), and s_waitcnt 9747 lgkmcnt(0) to allow 9748 them to be 9749 independently moved 9750 according to the 9751 following rules. 9752 - s_waitcnt vmcnt(0) 9753 must happen after 9754 any preceding 9755 global/generic 9756 load/load atomic 9757 atomicrmw-with-return-value. 9758 - s_waitcnt vscnt(0) 9759 must happen after 9760 any preceding 9761 global/generic 9762 store/store atomic/ 9763 atomicrmw-no-return-value. 9764 - s_waitcnt lgkmcnt(0) 9765 must happen after 9766 any preceding 9767 local/generic 9768 load/store/load 9769 atomic/store 9770 atomic/atomicrmw. 9771 - Must happen before 9772 the following 9773 atomicrmw. 9774 - Ensures that all 9775 memory operations 9776 have 9777 completed before 9778 performing the 9779 atomicrmw that is 9780 being released. 9781 9782 2. flat_atomic 9783 3. s_waitcnt vm/vscnt(0) & 9784 lgkmcnt(0) 9785 9786 - If OpenCL, omit 9787 lgkmcnt(0). 9788 - Use vmcnt(0) if atomic with 9789 return and vscnt(0) if 9790 atomic with no-return. 9791 - Must happen before 9792 following 9793 buffer_gl*_inv. 9794 - Ensures the 9795 atomicrmw has 9796 completed before 9797 invalidating the 9798 caches. 9799 9800 4. buffer_gl0_inv; 9801 buffer_gl1_inv 9802 9803 - Must happen before 9804 any following 9805 global/generic 9806 load/load 9807 atomic/atomicrmw. 9808 - Ensures that 9809 following loads 9810 will not see stale 9811 global data. 9812 9813 fence acq_rel - singlethread *none* *none* 9814 - wavefront 9815 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9816 vmcnt(0) & vscnt(0) 9817 9818 - If CU wavefront execution 9819 mode, omit vmcnt(0) and 9820 vscnt(0). 9821 - If OpenCL and 9822 address space is 9823 not generic, omit 9824 lgkmcnt(0). 9825 - If OpenCL and 9826 address space is 9827 local, omit 9828 vmcnt(0) and vscnt(0). 9829 - However, 9830 since LLVM 9831 currently has no 9832 address space on 9833 the fence need to 9834 conservatively 9835 always generate 9836 (see comment for 9837 previous fence). 9838 - Could be split into 9839 separate s_waitcnt 9840 vmcnt(0), s_waitcnt 9841 vscnt(0) and s_waitcnt 9842 lgkmcnt(0) to allow 9843 them to be 9844 independently moved 9845 according to the 9846 following rules. 9847 - s_waitcnt vmcnt(0) 9848 must happen after 9849 any preceding 9850 global/generic 9851 load/load 9852 atomic/ 9853 atomicrmw-with-return-value. 9854 - s_waitcnt vscnt(0) 9855 must happen after 9856 any preceding 9857 global/generic 9858 store/store atomic/ 9859 atomicrmw-no-return-value. 9860 - s_waitcnt lgkmcnt(0) 9861 must happen after 9862 any preceding 9863 local/generic 9864 load/store/load 9865 atomic/store atomic/ 9866 atomicrmw. 9867 - Must happen before 9868 any following 9869 global/generic 9870 load/load 9871 atomic/store/store 9872 atomic/atomicrmw. 9873 - Ensures that all 9874 memory operations 9875 have 9876 completed before 9877 performing any 9878 following global 9879 memory operations. 9880 - Ensures that the 9881 preceding 9882 local/generic load 9883 atomic/atomicrmw 9884 with an equal or 9885 wider sync scope 9886 and memory ordering 9887 stronger than 9888 unordered (this is 9889 termed the 9890 acquire-fence-paired-atomic) 9891 has completed 9892 before following 9893 global memory 9894 operations. This 9895 satisfies the 9896 requirements of 9897 acquire. 9898 - Ensures that all 9899 previous memory 9900 operations have 9901 completed before a 9902 following 9903 local/generic store 9904 atomic/atomicrmw 9905 with an equal or 9906 wider sync scope 9907 and memory ordering 9908 stronger than 9909 unordered (this is 9910 termed the 9911 release-fence-paired-atomic). 9912 This satisfies the 9913 requirements of 9914 release. 9915 - Must happen before 9916 the following 9917 buffer_gl0_inv. 9918 - Ensures that the 9919 acquire-fence-paired 9920 atomic has completed 9921 before invalidating 9922 the 9923 cache. Therefore 9924 any following 9925 locations read must 9926 be no older than 9927 the value read by 9928 the 9929 acquire-fence-paired-atomic. 9930 9931 3. buffer_gl0_inv 9932 9933 - If CU wavefront execution 9934 mode, omit. 9935 - Ensures that 9936 following 9937 loads will not see 9938 stale data. 9939 9940 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 9941 - system vmcnt(0) & vscnt(0) 9942 9943 - If OpenCL and 9944 address space is 9945 not generic, omit 9946 lgkmcnt(0). 9947 - If OpenCL and 9948 address space is 9949 local, omit 9950 vmcnt(0) and vscnt(0). 9951 - However, since LLVM 9952 currently has no 9953 address space on 9954 the fence need to 9955 conservatively 9956 always generate 9957 (see comment for 9958 previous fence). 9959 - Could be split into 9960 separate s_waitcnt 9961 vmcnt(0), s_waitcnt 9962 vscnt(0) and s_waitcnt 9963 lgkmcnt(0) to allow 9964 them to be 9965 independently moved 9966 according to the 9967 following rules. 9968 - s_waitcnt vmcnt(0) 9969 must happen after 9970 any preceding 9971 global/generic 9972 load/load 9973 atomic/ 9974 atomicrmw-with-return-value. 9975 - s_waitcnt vscnt(0) 9976 must happen after 9977 any preceding 9978 global/generic 9979 store/store atomic/ 9980 atomicrmw-no-return-value. 9981 - s_waitcnt lgkmcnt(0) 9982 must happen after 9983 any preceding 9984 local/generic 9985 load/store/load 9986 atomic/store 9987 atomic/atomicrmw. 9988 - Must happen before 9989 the following 9990 buffer_gl*_inv. 9991 - Ensures that the 9992 preceding 9993 global/local/generic 9994 load 9995 atomic/atomicrmw 9996 with an equal or 9997 wider sync scope 9998 and memory ordering 9999 stronger than 10000 unordered (this is 10001 termed the 10002 acquire-fence-paired-atomic) 10003 has completed 10004 before invalidating 10005 the caches. This 10006 satisfies the 10007 requirements of 10008 acquire. 10009 - Ensures that all 10010 previous memory 10011 operations have 10012 completed before a 10013 following 10014 global/local/generic 10015 store 10016 atomic/atomicrmw 10017 with an equal or 10018 wider sync scope 10019 and memory ordering 10020 stronger than 10021 unordered (this is 10022 termed the 10023 release-fence-paired-atomic). 10024 This satisfies the 10025 requirements of 10026 release. 10027 10028 2. buffer_gl0_inv; 10029 buffer_gl1_inv 10030 10031 - Must happen before 10032 any following 10033 global/generic 10034 load/load 10035 atomic/store/store 10036 atomic/atomicrmw. 10037 - Ensures that 10038 following loads 10039 will not see stale 10040 global data. This 10041 satisfies the 10042 requirements of 10043 acquire. 10044 10045 **Sequential Consistent Atomic** 10046 ------------------------------------------------------------------------------------ 10047 load atomic seq_cst - singlethread - global *Same as corresponding 10048 - wavefront - local load atomic acquire, 10049 - generic except must generated 10050 all instructions even 10051 for OpenCL.* 10052 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10053 - generic vmcnt(0) & vscnt(0) 10054 10055 - If CU wavefront execution 10056 mode, omit vmcnt(0) and 10057 vscnt(0). 10058 - Could be split into 10059 separate s_waitcnt 10060 vmcnt(0), s_waitcnt 10061 vscnt(0), and s_waitcnt 10062 lgkmcnt(0) to allow 10063 them to be 10064 independently moved 10065 according to the 10066 following rules. 10067 - s_waitcnt lgkmcnt(0) must 10068 happen after 10069 preceding 10070 local/generic load 10071 atomic/store 10072 atomic/atomicrmw 10073 with memory 10074 ordering of seq_cst 10075 and with equal or 10076 wider sync scope. 10077 (Note that seq_cst 10078 fences have their 10079 own s_waitcnt 10080 lgkmcnt(0) and so do 10081 not need to be 10082 considered.) 10083 - s_waitcnt vmcnt(0) 10084 must happen after 10085 preceding 10086 global/generic load 10087 atomic/ 10088 atomicrmw-with-return-value 10089 with memory 10090 ordering of seq_cst 10091 and with equal or 10092 wider sync scope. 10093 (Note that seq_cst 10094 fences have their 10095 own s_waitcnt 10096 vmcnt(0) and so do 10097 not need to be 10098 considered.) 10099 - s_waitcnt vscnt(0) 10100 Must happen after 10101 preceding 10102 global/generic store 10103 atomic/ 10104 atomicrmw-no-return-value 10105 with memory 10106 ordering of seq_cst 10107 and with equal or 10108 wider sync scope. 10109 (Note that seq_cst 10110 fences have their 10111 own s_waitcnt 10112 vscnt(0) and so do 10113 not need to be 10114 considered.) 10115 - Ensures any 10116 preceding 10117 sequential 10118 consistent global/local 10119 memory instructions 10120 have completed 10121 before executing 10122 this sequentially 10123 consistent 10124 instruction. This 10125 prevents reordering 10126 a seq_cst store 10127 followed by a 10128 seq_cst load. (Note 10129 that seq_cst is 10130 stronger than 10131 acquire/release as 10132 the reordering of 10133 load acquire 10134 followed by a store 10135 release is 10136 prevented by the 10137 s_waitcnt of 10138 the release, but 10139 there is nothing 10140 preventing a store 10141 release followed by 10142 load acquire from 10143 completing out of 10144 order. The s_waitcnt 10145 could be placed after 10146 seq_store or before 10147 the seq_load. We 10148 choose the load to 10149 make the s_waitcnt be 10150 as late as possible 10151 so that the store 10152 may have already 10153 completed.) 10154 10155 2. *Following 10156 instructions same as 10157 corresponding load 10158 atomic acquire, 10159 except must generated 10160 all instructions even 10161 for OpenCL.* 10162 load atomic seq_cst - workgroup - local 10163 10164 1. s_waitcnt vmcnt(0) & vscnt(0) 10165 10166 - If CU wavefront execution 10167 mode, omit. 10168 - Could be split into 10169 separate s_waitcnt 10170 vmcnt(0) and s_waitcnt 10171 vscnt(0) to allow 10172 them to be 10173 independently moved 10174 according to the 10175 following rules. 10176 - s_waitcnt vmcnt(0) 10177 Must happen after 10178 preceding 10179 global/generic load 10180 atomic/ 10181 atomicrmw-with-return-value 10182 with memory 10183 ordering of seq_cst 10184 and with equal or 10185 wider sync scope. 10186 (Note that seq_cst 10187 fences have their 10188 own s_waitcnt 10189 vmcnt(0) and so do 10190 not need to be 10191 considered.) 10192 - s_waitcnt vscnt(0) 10193 Must happen after 10194 preceding 10195 global/generic store 10196 atomic/ 10197 atomicrmw-no-return-value 10198 with memory 10199 ordering of seq_cst 10200 and with equal or 10201 wider sync scope. 10202 (Note that seq_cst 10203 fences have their 10204 own s_waitcnt 10205 vscnt(0) and so do 10206 not need to be 10207 considered.) 10208 - Ensures any 10209 preceding 10210 sequential 10211 consistent global 10212 memory instructions 10213 have completed 10214 before executing 10215 this sequentially 10216 consistent 10217 instruction. This 10218 prevents reordering 10219 a seq_cst store 10220 followed by a 10221 seq_cst load. (Note 10222 that seq_cst is 10223 stronger than 10224 acquire/release as 10225 the reordering of 10226 load acquire 10227 followed by a store 10228 release is 10229 prevented by the 10230 s_waitcnt of 10231 the release, but 10232 there is nothing 10233 preventing a store 10234 release followed by 10235 load acquire from 10236 completing out of 10237 order. The s_waitcnt 10238 could be placed after 10239 seq_store or before 10240 the seq_load. We 10241 choose the load to 10242 make the s_waitcnt be 10243 as late as possible 10244 so that the store 10245 may have already 10246 completed.) 10247 10248 2. *Following 10249 instructions same as 10250 corresponding load 10251 atomic acquire, 10252 except must generated 10253 all instructions even 10254 for OpenCL.* 10255 10256 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10257 - system - generic vmcnt(0) & vscnt(0) 10258 10259 - Could be split into 10260 separate s_waitcnt 10261 vmcnt(0), s_waitcnt 10262 vscnt(0) and s_waitcnt 10263 lgkmcnt(0) to allow 10264 them to be 10265 independently moved 10266 according to the 10267 following rules. 10268 - s_waitcnt lgkmcnt(0) 10269 must happen after 10270 preceding 10271 local load 10272 atomic/store 10273 atomic/atomicrmw 10274 with memory 10275 ordering of seq_cst 10276 and with equal or 10277 wider sync scope. 10278 (Note that seq_cst 10279 fences have their 10280 own s_waitcnt 10281 lgkmcnt(0) and so do 10282 not need to be 10283 considered.) 10284 - s_waitcnt vmcnt(0) 10285 must happen after 10286 preceding 10287 global/generic load 10288 atomic/ 10289 atomicrmw-with-return-value 10290 with memory 10291 ordering of seq_cst 10292 and with equal or 10293 wider sync scope. 10294 (Note that seq_cst 10295 fences have their 10296 own s_waitcnt 10297 vmcnt(0) and so do 10298 not need to be 10299 considered.) 10300 - s_waitcnt vscnt(0) 10301 Must happen after 10302 preceding 10303 global/generic store 10304 atomic/ 10305 atomicrmw-no-return-value 10306 with memory 10307 ordering of seq_cst 10308 and with equal or 10309 wider sync scope. 10310 (Note that seq_cst 10311 fences have their 10312 own s_waitcnt 10313 vscnt(0) and so do 10314 not need to be 10315 considered.) 10316 - Ensures any 10317 preceding 10318 sequential 10319 consistent global 10320 memory instructions 10321 have completed 10322 before executing 10323 this sequentially 10324 consistent 10325 instruction. This 10326 prevents reordering 10327 a seq_cst store 10328 followed by a 10329 seq_cst load. (Note 10330 that seq_cst is 10331 stronger than 10332 acquire/release as 10333 the reordering of 10334 load acquire 10335 followed by a store 10336 release is 10337 prevented by the 10338 s_waitcnt of 10339 the release, but 10340 there is nothing 10341 preventing a store 10342 release followed by 10343 load acquire from 10344 completing out of 10345 order. The s_waitcnt 10346 could be placed after 10347 seq_store or before 10348 the seq_load. We 10349 choose the load to 10350 make the s_waitcnt be 10351 as late as possible 10352 so that the store 10353 may have already 10354 completed.) 10355 10356 2. *Following 10357 instructions same as 10358 corresponding load 10359 atomic acquire, 10360 except must generated 10361 all instructions even 10362 for OpenCL.* 10363 store atomic seq_cst - singlethread - global *Same as corresponding 10364 - wavefront - local store atomic release, 10365 - workgroup - generic except must generated 10366 - agent all instructions even 10367 - system for OpenCL.* 10368 atomicrmw seq_cst - singlethread - global *Same as corresponding 10369 - wavefront - local atomicrmw acq_rel, 10370 - workgroup - generic except must generated 10371 - agent all instructions even 10372 - system for OpenCL.* 10373 fence seq_cst - singlethread *none* *Same as corresponding 10374 - wavefront fence acq_rel, 10375 - workgroup except must generated 10376 - agent all instructions even 10377 - system for OpenCL.* 10378 ============ ============ ============== ========== ================================ 10379 10380Trap Handler ABI 10381~~~~~~~~~~~~~~~~ 10382 10383For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10384runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10385supports the ``s_trap`` instruction. For usage see: 10386 10387- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10388- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10389- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10390 10391 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10392 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10393 10394 =================== =============== =============== ======================================= 10395 Usage Code Sequence Trap Handler Description 10396 Inputs 10397 =================== =============== =============== ======================================= 10398 reserved ``s_trap 0x00`` Reserved by hardware. 10399 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10400 ``queue_ptr`` intrinsic (not implemented). 10401 ``VGPR0``: 10402 ``arg`` 10403 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10404 ``queue_ptr`` the trap instruction. The associated 10405 queue is signalled to put it into the 10406 error state. When the queue is put in 10407 the error state, the waves executing 10408 dispatches on the queue will be 10409 terminated. 10410 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10411 as a no-operation. The trap handler 10412 is entered and immediately returns to 10413 continue execution of the wavefront. 10414 - If the debugger is enabled, causes 10415 the debug trap to be reported by the 10416 debugger and the wavefront is put in 10417 the halt state with the PC at the 10418 instruction. The debugger must 10419 increment the PC and resume the wave. 10420 reserved ``s_trap 0x04`` Reserved. 10421 reserved ``s_trap 0x05`` Reserved. 10422 reserved ``s_trap 0x06`` Reserved. 10423 reserved ``s_trap 0x07`` Reserved. 10424 reserved ``s_trap 0x08`` Reserved. 10425 reserved ``s_trap 0xfe`` Reserved. 10426 reserved ``s_trap 0xff`` Reserved. 10427 =================== =============== =============== ======================================= 10428 10429.. 10430 10431 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10432 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10433 10434 =================== =============== =============== ======================================= 10435 Usage Code Sequence Trap Handler Description 10436 Inputs 10437 =================== =============== =============== ======================================= 10438 reserved ``s_trap 0x00`` Reserved by hardware. 10439 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10440 breakpoints. Causes wave to be halted 10441 with the PC at the trap instruction. 10442 The debugger is responsible to resume 10443 the wave, including the instruction 10444 that the breakpoint overwrote. 10445 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10446 ``queue_ptr`` the trap instruction. The associated 10447 queue is signalled to put it into the 10448 error state. When the queue is put in 10449 the error state, the waves executing 10450 dispatches on the queue will be 10451 terminated. 10452 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10453 as a no-operation. The trap handler 10454 is entered and immediately returns to 10455 continue execution of the wavefront. 10456 - If the debugger is enabled, causes 10457 the debug trap to be reported by the 10458 debugger and the wavefront is put in 10459 the halt state with the PC at the 10460 instruction. The debugger must 10461 increment the PC and resume the wave. 10462 reserved ``s_trap 0x04`` Reserved. 10463 reserved ``s_trap 0x05`` Reserved. 10464 reserved ``s_trap 0x06`` Reserved. 10465 reserved ``s_trap 0x07`` Reserved. 10466 reserved ``s_trap 0x08`` Reserved. 10467 reserved ``s_trap 0xfe`` Reserved. 10468 reserved ``s_trap 0xff`` Reserved. 10469 =================== =============== =============== ======================================= 10470 10471.. 10472 10473 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10474 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10475 10476 =================== =============== ================ ================= ======================================= 10477 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10478 =================== =============== ================ ================= ======================================= 10479 reserved ``s_trap 0x00`` Reserved by hardware. 10480 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10481 breakpoints. Causes wave to be halted 10482 with the PC at the trap instruction. 10483 The debugger is responsible to resume 10484 the wave, including the instruction 10485 that the breakpoint overwrote. 10486 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10487 ``queue_ptr`` the trap instruction. The associated 10488 queue is signalled to put it into the 10489 error state. When the queue is put in 10490 the error state, the waves executing 10491 dispatches on the queue will be 10492 terminated. 10493 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10494 as a no-operation. The trap handler 10495 is entered and immediately returns to 10496 continue execution of the wavefront. 10497 - If the debugger is enabled, causes 10498 the debug trap to be reported by the 10499 debugger and the wavefront is put in 10500 the halt state with the PC at the 10501 instruction. The debugger must 10502 increment the PC and resume the wave. 10503 reserved ``s_trap 0x04`` Reserved. 10504 reserved ``s_trap 0x05`` Reserved. 10505 reserved ``s_trap 0x06`` Reserved. 10506 reserved ``s_trap 0x07`` Reserved. 10507 reserved ``s_trap 0x08`` Reserved. 10508 reserved ``s_trap 0xfe`` Reserved. 10509 reserved ``s_trap 0xff`` Reserved. 10510 =================== =============== ================ ================= ======================================= 10511 10512.. _amdgpu-amdhsa-function-call-convention: 10513 10514Call Convention 10515~~~~~~~~~~~~~~~ 10516 10517.. note:: 10518 10519 This section is currently incomplete and has inaccuracies. It is WIP that will 10520 be updated as information is determined. 10521 10522See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10523addresses. Unswizzled addresses are normal linear addresses. 10524 10525.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10526 10527Kernel Functions 10528++++++++++++++++ 10529 10530This section describes the call convention ABI for the outer kernel function. 10531 10532See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10533convention. 10534 10535The following is not part of the AMDGPU kernel calling convention but describes 10536how the AMDGPU implements function calls: 10537 105381. Clang decides the kernarg layout to match the *HSA Programmer's Language 10539 Reference* [HSA]_. 10540 10541 - All structs are passed directly. 10542 - Lambda values are passed *TBA*. 10543 10544 .. TODO:: 10545 10546 - Does this really follow HSA rules? Or are structs >16 bytes passed 10547 by-value struct? 10548 - What is ABI for lambda values? 10549 105504. The kernel performs certain setup in its prolog, as described in 10551 :ref:`amdgpu-amdhsa-kernel-prolog`. 10552 10553.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10554 10555Non-Kernel Functions 10556++++++++++++++++++++ 10557 10558This section describes the call convention ABI for functions other than the 10559outer kernel function. 10560 10561If a kernel has function calls then scratch is always allocated and used for 10562the call stack which grows from low address to high address using the swizzled 10563scratch address space. 10564 10565On entry to a function: 10566 105671. SGPR0-3 contain a V# with the following properties (see 10568 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10569 10570 * Base address pointing to the beginning of the wavefront scratch backing 10571 memory. 10572 * Swizzled with dword element size and stride of wavefront size elements. 10573 105742. The FLAT_SCRATCH register pair is setup. See 10575 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 105763. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10577 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 105784. The EXEC register is set to the lanes active on entry to the function. 105795. MODE register: *TBD* 105806. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10581 below. 105827. SGPR30-31 return address (RA). The code address that the function must 10583 return to when it completes. The value is undefined if the function is *no 10584 return*. 105858. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10586 offset relative to the beginning of the wavefront scratch backing memory. 10587 10588 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10589 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10590 manner. 10591 10592 The unswizzled SP value can be converted into the swizzled SP value by: 10593 10594 | swizzled SP = unswizzled SP / wavefront size 10595 10596 This may be used to obtain the private address space address of stack 10597 objects and to convert this address to a flat address by adding the flat 10598 scratch aperture base address. 10599 10600 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10601 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10602 10603 .. note:: 10604 10605 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10606 OpenCL language which has the largest base type defined as 16 bytes. 10607 10608 On entry, the swizzled SP value is the address of the first function 10609 argument passed on the stack. Other stack passed arguments are positive 10610 offsets from the entry swizzled SP value. 10611 10612 The function may use positive offsets beyond the last stack passed argument 10613 for stack allocated local variables and register spill slots. If necessary, 10614 the function may align these to greater alignment than 16 bytes. After these 10615 the function may dynamically allocate space for such things as runtime sized 10616 ``alloca`` local allocations. 10617 10618 If the function calls another function, it will place any stack allocated 10619 arguments after the last local allocation and adjust SGPR32 to the address 10620 after the last local allocation. 10621 106229. All other registers are unspecified. 1062310. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10624 to the function. 10625 10626On exit from a function: 10627 106281. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10629 described below. Any registers used are considered clobbered registers. 106302. The following registers are preserved and have the same value as on entry: 10631 10632 * FLAT_SCRATCH 10633 * EXEC 10634 * GFX6-GFX8: M0 10635 * All SGPR registers except the clobbered registers of SGPR4-31. 10636 * VGPR40-47 10637 * VGPR56-63 10638 * VGPR72-79 10639 * VGPR88-95 10640 * VGPR104-111 10641 * VGPR120-127 10642 * VGPR136-143 10643 * VGPR152-159 10644 * VGPR168-175 10645 * VGPR184-191 10646 * VGPR200-207 10647 * VGPR216-223 10648 * VGPR232-239 10649 * VGPR248-255 10650 10651 .. note:: 10652 10653 Except the argument registers, the VGPRs clobbered and the preserved 10654 registers are intermixed at regular intervals in order to keep a 10655 similar ratio independent of the number of allocated VGPRs. 10656 10657 * Lanes of all VGPRs that are inactive at the call site. 10658 10659 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10660 optimization may mark some of clobbered SGPR and VGPR registers as 10661 preserved if it can be determined that the called function does not change 10662 their value. 10663 106642. The PC is set to the RA provided on entry. 106653. MODE register: *TBD*. 106664. All other registers are clobbered. 106675. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10668 function is available to the caller. 10669 10670.. TODO:: 10671 10672 - On gfx908 are all ACC registers clobbered? 10673 10674 - How are function results returned? The address of structured types is passed 10675 by reference, but what about other types? 10676 10677The function input arguments are made up of the formal arguments explicitly 10678declared by the source language function plus the implicit input arguments used 10679by the implementation. 10680 10681The source language input arguments are: 10682 106831. Any source language implicit ``this`` or ``self`` argument comes first as a 10684 pointer type. 106852. Followed by the function formal arguments in left to right source order. 10686 10687The source language result arguments are: 10688 106891. The function result argument. 10690 10691The source language input or result struct type arguments that are less than or 10692equal to 16 bytes, are decomposed recursively into their base type fields, and 10693each field is passed as if a separate argument. For input arguments, if the 10694called function requires the struct to be in memory, for example because its 10695address is taken, then the function body is responsible for allocating a stack 10696location and copying the field arguments into it. Clang terms this *direct 10697struct*. 10698 10699The source language input struct type arguments that are greater than 16 bytes, 10700are passed by reference. The caller is responsible for allocating a stack 10701location to make a copy of the struct value and pass the address as the input 10702argument. The called function is responsible to perform the dereference when 10703accessing the input argument. Clang terms this *by-value struct*. 10704 10705A source language result struct type argument that is greater than 16 bytes, is 10706returned by reference. The caller is responsible for allocating a stack location 10707to hold the result value and passes the address as the last input argument 10708(before the implicit input arguments). In this case there are no result 10709arguments. The called function is responsible to perform the dereference when 10710storing the result value. Clang terms this *structured return (sret)*. 10711 10712*TODO: correct the ``sret`` definition.* 10713 10714.. TODO:: 10715 10716 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10717 pass as non-decomposed struct as stack argument? Or something else? Is the 10718 memory location in the caller stack frame, or a stack memory argument and so 10719 no address is passed as the caller can directly write to the argument stack 10720 location? But then the stack location is still live after return. If an 10721 argument stack location is it the first stack argument or the last one? 10722 10723Lambda argument types are treated as struct types with an implementation defined 10724set of fields. 10725 10726.. TODO:: 10727 10728 Need to specify the ABI for lambda types for AMDGPU. 10729 10730For AMDGPU backend all source language arguments (including the decomposed 10731struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10732they are passed in SGPRs. 10733 10734The AMDGPU backend walks the function call graph from the leaves to determine 10735which implicit input arguments are used, propagating to each caller of the 10736function. The used implicit arguments are appended to the function arguments 10737after the source language arguments in the following order: 10738 10739.. TODO:: 10740 10741 Is recursion or external functions supported? 10742 107431. Work-Item ID (1 VGPR) 10744 10745 The X, Y and Z work-item ID are packed into a single VGRP with the following 10746 layout. Only fields actually used by the function are set. The other bits 10747 are undefined. 10748 10749 The values come from the initial kernel execution state. See 10750 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10751 10752 .. table:: Work-item implicit argument layout 10753 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10754 10755 ======= ======= ============== 10756 Bits Size Field Name 10757 ======= ======= ============== 10758 9:0 10 bits X Work-Item ID 10759 19:10 10 bits Y Work-Item ID 10760 29:20 10 bits Z Work-Item ID 10761 31:30 2 bits Unused 10762 ======= ======= ============== 10763 107642. Dispatch Ptr (2 SGPRs) 10765 10766 The value comes from the initial kernel execution state. See 10767 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10768 107693. Queue Ptr (2 SGPRs) 10770 10771 The value comes from the initial kernel execution state. See 10772 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10773 107744. Kernarg Segment Ptr (2 SGPRs) 10775 10776 The value comes from the initial kernel execution state. See 10777 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10778 107795. Dispatch id (2 SGPRs) 10780 10781 The value comes from the initial kernel execution state. See 10782 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10783 107846. Work-Group ID X (1 SGPR) 10785 10786 The value comes from the initial kernel execution state. See 10787 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10788 107897. Work-Group ID Y (1 SGPR) 10790 10791 The value comes from the initial kernel execution state. See 10792 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10793 107948. Work-Group ID Z (1 SGPR) 10795 10796 The value comes from the initial kernel execution state. See 10797 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10798 107999. Implicit Argument Ptr (2 SGPRs) 10800 10801 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10802 global address space pointer to the first kernarg implicit argument. 10803 10804The input and result arguments are assigned in order in the following manner: 10805 10806.. note:: 10807 10808 There are likely some errors and omissions in the following description that 10809 need correction. 10810 10811 .. TODO:: 10812 10813 Check the Clang source code to decipher how function arguments and return 10814 results are handled. Also see the AMDGPU specific values used. 10815 10816* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 10817 VGPR31. 10818 10819 If there are more arguments than will fit in these registers, the remaining 10820 arguments are allocated on the stack in order on naturally aligned 10821 addresses. 10822 10823 .. TODO:: 10824 10825 How are overly aligned structures allocated on the stack? 10826 10827* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 10828 SGPR29. 10829 10830 If there are more arguments than will fit in these registers, the remaining 10831 arguments are allocated on the stack in order on naturally aligned 10832 addresses. 10833 10834Note that decomposed struct type arguments may have some fields passed in 10835registers and some in memory. 10836 10837.. TODO:: 10838 10839 So, a struct which can pass some fields as decomposed register arguments, will 10840 pass the rest as decomposed stack elements? But an argument that will not start 10841 in registers will not be decomposed and will be passed as a non-decomposed 10842 stack value? 10843 10844The following is not part of the AMDGPU function calling convention but 10845describes how the AMDGPU implements function calls: 10846 108471. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 10848 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 10849 are used, or for the reasons defined in ``SIFrameLowering``. 108502. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 10851 to access the incoming stack arguments in the function. The BP is needed 10852 only when the function requires the runtime stack alignment. 10853 108543. Allocating SGPR arguments on the stack are not supported. 10855 108564. No CFI is currently generated. See 10857 :ref:`amdgpu-dwarf-call-frame-information`. 10858 10859 .. note:: 10860 10861 CFI will be generated that defines the CFA as the unswizzled address 10862 relative to the wave scratch base in the unswizzled private address space 10863 of the lowest address stack allocated local variable. 10864 10865 ``DW_AT_frame_base`` will be defined as the swizzled address in the 10866 swizzled private address space by dividing the CFA by the wavefront size 10867 (since CFA is always at least dword aligned which matches the scratch 10868 swizzle element size). 10869 10870 If no dynamic stack alignment was performed, the stack allocated arguments 10871 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 10872 local variables and register spill slots are accessed as positive offsets 10873 relative to ``DW_AT_frame_base``. 10874 108755. Function argument passing is implemented by copying the input physical 10876 registers to virtual registers on entry. The register allocator can spill if 10877 necessary. These are copied back to physical registers at call sites. The 10878 net effect is that each function call can have these values in entirely 10879 distinct locations. The IPRA can help avoid shuffling argument registers. 108806. Call sites are implemented by setting up the arguments at positive offsets 10881 from SP. Then SP is incremented to account for the known frame size before 10882 the call and decremented after the call. 10883 10884 .. note:: 10885 10886 The CFI will reflect the changed calculation needed to compute the CFA 10887 from SP. 10888 108897. 4 byte spill slots are used in the stack frame. One slot is allocated for an 10890 emergency spill slot. Buffer instructions are used for stack accesses and 10891 not the ``flat_scratch`` instruction. 10892 10893 .. TODO:: 10894 10895 Explain when the emergency spill slot is used. 10896 10897.. TODO:: 10898 10899 Possible broken issues: 10900 10901 - Stack arguments must be aligned to required alignment. 10902 - Stack is aligned to max(16, max formal argument alignment) 10903 - Direct argument < 64 bits should check register budget. 10904 - Register budget calculation should respect ``inreg`` for SGPR. 10905 - SGPR overflow is not handled. 10906 - struct with 1 member unpeeling is not checking size of member. 10907 - ``sret`` is after ``this`` pointer. 10908 - Caller is not implementing stack realignment: need an extra pointer. 10909 - Should say AMDGPU passes FP rather than SP. 10910 - Should CFI define CFA as address of locals or arguments. Difference is 10911 apparent when have implemented dynamic alignment. 10912 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 10913 highest address of stack frame and use negative offset for locals. Would 10914 allow SP to be the same as FP and could support signal-handler-like as now 10915 have a real SP for the top of the stack. 10916 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 10917 arguments? 10918 10919AMDPAL 10920------ 10921 10922This section provides code conventions used when the target triple OS is 10923``amdpal`` (see :ref:`amdgpu-target-triples`). 10924 10925.. _amdgpu-amdpal-code-object-metadata-section: 10926 10927Code Object Metadata 10928~~~~~~~~~~~~~~~~~~~~ 10929 10930.. note:: 10931 10932 The metadata is currently in development and is subject to major 10933 changes. Only the current version is supported. *When this document 10934 was generated the version was 2.6.* 10935 10936Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 10937record (see :ref:`amdgpu-note-records-v3-v4`). 10938 10939The metadata is represented as Message Pack formatted binary data (see 10940[MsgPack]_). The top level is a Message Pack map that includes the keys 10941defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 10942and referenced tables. 10943 10944Additional information can be added to the maps. To avoid conflicts, any 10945key names should be prefixed by "*vendor-name*." where ``vendor-name`` 10946can be the name of the vendor and specific vendor tool that generates the 10947information. The prefix is abbreviated to simply "." when it appears 10948within a map that has been added by the same *vendor-name*. 10949 10950 .. table:: AMDPAL Code Object Metadata Map 10951 :name: amdgpu-amdpal-code-object-metadata-map-table 10952 10953 =================== ============== ========= ====================================================================== 10954 String Key Value Type Required? Description 10955 =================== ============== ========= ====================================================================== 10956 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 10957 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 10958 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 10959 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 10960 definition of the keys included in that map. 10961 =================== ============== ========= ====================================================================== 10962 10963.. 10964 10965 .. table:: AMDPAL Code Object Pipeline Metadata Map 10966 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 10967 10968 ====================================== ============== ========= =================================================== 10969 String Key Value Type Required? Description 10970 ====================================== ============== ========= =================================================== 10971 ".name" string Source name of the pipeline. 10972 ".type" string Pipeline type, e.g. VsPs. Values include: 10973 10974 - "VsPs" 10975 - "Gs" 10976 - "Cs" 10977 - "Ngg" 10978 - "Tess" 10979 - "GsTess" 10980 - "NggTess" 10981 10982 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 10983 2 integers 64 bits is the "stable" portion of the hash, used 10984 for e.g. shader replacement lookup. Upper 64 bits 10985 is the "unique" portion of the hash, used for 10986 e.g. pipeline cache lookup. The value is 10987 implementation defined, and can not be relied on 10988 between different builds of the compiler. 10989 ".shaders" map Per-API shader metadata. See 10990 :ref:`amdgpu-amdpal-code-object-shader-map-table` 10991 for the definition of the keys included in that 10992 map. 10993 ".hardware_stages" map Per-hardware stage metadata. See 10994 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 10995 for the definition of the keys included in that 10996 map. 10997 ".shader_functions" map Per-shader function metadata. See 10998 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 10999 for the definition of the keys included in that 11000 map. 11001 ".registers" map Required Hardware register configuration. See 11002 :ref:`amdgpu-amdpal-code-object-register-map-table` 11003 for the definition of the keys included in that 11004 map. 11005 ".user_data_limit" integer Number of user data entries accessed by this 11006 pipeline. 11007 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 11008 NoUserDataSpilling. 11009 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 11010 viewport array index feature. Pipelines which use 11011 this feature can render into all 16 viewports, 11012 whereas pipelines which do not use it are 11013 restricted to viewport #0. 11014 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 11015 handling data-passing between the ES and GS 11016 shader stages. This can be zero if the data is 11017 passed using off-chip buffers. This value should 11018 be used to program all user-SGPRs which have been 11019 marked with "UserDataMapping::EsGsLdsSize" 11020 (typically only the GS and VS HW stages will ever 11021 have a user-SGPR so marked). 11022 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 11023 (maximum number of threads in a subgroup). 11024 ".num_interpolants" integer Graphics only. Number of PS interpolants. 11025 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 11026 ".api" string Name of the client graphics API. 11027 ".api_create_info" binary Graphics API shader create info binary blob. Can 11028 be defined by the driver using the compiler if 11029 they want to be able to correlate API-specific 11030 information used during creation at a later time. 11031 ====================================== ============== ========= =================================================== 11032 11033.. 11034 11035 .. table:: AMDPAL Code Object Shader Map 11036 :name: amdgpu-amdpal-code-object-shader-map-table 11037 11038 11039 +-------------+--------------+-------------------------------------------------------------------+ 11040 |String Key |Value Type |Description | 11041 +=============+==============+===================================================================+ 11042 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 11043 |- ".vertex" | |for the definition of the keys included in that map. | 11044 |- ".hull" | | | 11045 |- ".domain" | | | 11046 |- ".geometry"| | | 11047 |- ".pixel" | | | 11048 +-------------+--------------+-------------------------------------------------------------------+ 11049 11050.. 11051 11052 .. table:: AMDPAL Code Object API Shader Metadata Map 11053 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 11054 11055 ==================== ============== ========= ===================================================================== 11056 String Key Value Type Required? Description 11057 ==================== ============== ========= ===================================================================== 11058 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 11059 2 integers is implementation defined, and can not be relied on between 11060 different builds of the compiler. 11061 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 11062 string include: 11063 11064 - ".ls" 11065 - ".hs" 11066 - ".es" 11067 - ".gs" 11068 - ".vs" 11069 - ".ps" 11070 - ".cs" 11071 11072 ==================== ============== ========= ===================================================================== 11073 11074.. 11075 11076 .. table:: AMDPAL Code Object Hardware Stage Map 11077 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 11078 11079 +-------------+--------------+-----------------------------------------------------------------------+ 11080 |String Key |Value Type |Description | 11081 +=============+==============+=======================================================================+ 11082 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 11083 |- ".hs" | |for the definition of the keys included in that map. | 11084 |- ".es" | | | 11085 |- ".gs" | | | 11086 |- ".vs" | | | 11087 |- ".ps" | | | 11088 |- ".cs" | | | 11089 +-------------+--------------+-----------------------------------------------------------------------+ 11090 11091.. 11092 11093 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 11094 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 11095 11096 ========================== ============== ========= =============================================================== 11097 String Key Value Type Required? Description 11098 ========================== ============== ========= =============================================================== 11099 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 11100 ".scratch_memory_size" integer Scratch memory size in bytes. 11101 ".lds_size" integer Local Data Share size in bytes. 11102 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 11103 ".vgpr_count" integer Number of VGPRs used. 11104 ".sgpr_count" integer Number of SGPRs used. 11105 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 11106 directive to instruct the compiler to limit the VGPR usage to 11107 be less than or equal to the specified value (only set if 11108 different from HW default). 11109 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 11110 default). 11111 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 11112 3 integers 11113 ".wavefront_size" integer Wavefront size (only set if different from HW default). 11114 ".uses_uavs" boolean The shader reads or writes UAVs. 11115 ".uses_rovs" boolean The shader reads or writes ROVs. 11116 ".writes_uavs" boolean The shader writes to one or more UAVs. 11117 ".writes_depth" boolean The shader writes out a depth value. 11118 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 11119 memory or GDS. 11120 ".uses_prim_id" boolean The shader uses PrimID. 11121 ========================== ============== ========= =============================================================== 11122 11123.. 11124 11125 .. table:: AMDPAL Code Object Shader Function Map 11126 :name: amdgpu-amdpal-code-object-shader-function-map-table 11127 11128 =============== ============== ==================================================================== 11129 String Key Value Type Description 11130 =============== ============== ==================================================================== 11131 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 11132 entry address. The value is the function's metadata. See 11133 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 11134 =============== ============== ==================================================================== 11135 11136.. 11137 11138 .. table:: AMDPAL Code Object Shader Function Metadata Map 11139 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 11140 11141 ============================= ============== ================================================================= 11142 String Key Value Type Description 11143 ============================= ============== ================================================================= 11144 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 11145 2 integers is implementation defined, and can not be relied on between 11146 different builds of the compiler. 11147 ".scratch_memory_size" sequence of Size in bytes of scratch memory used by the shader. 11148 2 integers 11149 ".lds_size" sequence of Size in bytes of LDS memory. 11150 2 integers 11151 ".vgpr_count" integer Number of VGPRs used by the shader. 11152 ".sgpr_count" integer Number of SGPRs used by the shader. 11153 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 11154 ".shader_subtype" string Shader subtype/kind. Values include: 11155 11156 - "Unknown" 11157 11158 ============================= ============== ================================================================= 11159 11160.. 11161 11162 .. table:: AMDPAL Code Object Register Map 11163 :name: amdgpu-amdpal-code-object-register-map-table 11164 11165 ========================== ============== ==================================================================== 11166 32-bit Integer Key Value Type Description 11167 ========================== ============== ==================================================================== 11168 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 11169 a GRBM register (i.e., driver accessible GPU register number, not 11170 shader GPR register number). The driver is required to program each 11171 specified register to the corresponding specified value when 11172 executing this pipeline. Typically, the ``reg offsets`` are the 11173 ``uint16_t`` offsets to each register as defined by the hardware 11174 chip headers. The register is set to the provided value. However, a 11175 ``reg offset`` that specifies a user data register (e.g., 11176 COMPUTE_USER_DATA_0) needs special treatment. See 11177 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 11178 information. 11179 ========================== ============== ==================================================================== 11180 11181.. _amdgpu-amdpal-code-object-user-data-section: 11182 11183User Data 11184+++++++++ 11185 11186Each hardware stage has a set of 32-bit physical SPI *user data registers* 11187(either 16 or 32 based on graphics IP and the stage) which can be 11188written from a command buffer and then loaded into SGPRs when waves are 11189launched via a subsequent dispatch or draw operation. This is the way 11190most arguments are passed from the application/runtime to a hardware 11191shader. 11192 11193PAL abstracts this functionality by exposing a set of 128 *user data 11194entries* per pipeline a client can use to pass arguments from a command 11195buffer to one or more shaders in that pipeline. The ELF code object must 11196specify a mapping from virtualized *user data entries* to physical *user 11197data registers*, and PAL is responsible for implementing that mapping, 11198including spilling overflow *user data entries* to memory if needed. 11199 11200Since the *user data registers* are GRBM-accessible SPI registers, this 11201mapping is actually embedded in the ``.registers`` metadata entry. For 11202most registers, the value in that map is a literal 32-bit value that 11203should be written to the register by the driver. However, when the 11204register is a *user data register* (any USER_DATA register e.g., 11205SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 11206the driver to write either a *user data entry* value or one of several 11207driver-internal values to the register. This encoding is described in 11208the following table: 11209 11210.. note:: 11211 11212 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 11213 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 11214 always be programmed to the address of the GlobalTable, and *user data 11215 register* 1 must always be programmed to the address of the PerShaderTable. 11216 11217.. 11218 11219 .. table:: AMDPAL User Data Mapping 11220 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 11221 11222 ========== ================= =============================================================================== 11223 Value Name Description 11224 ========== ================= =============================================================================== 11225 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 11226 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 11227 always point to *user data register* 0). 11228 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 11229 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 11230 for more detail (should always point to *user data register* 1). 11231 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 11232 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 11233 more detail. 11234 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 11235 reference the draw index in the vertex shader. Only supported by the first 11236 stage in a graphics pipeline. 11237 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 11238 a graphics pipeline. 11239 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 11240 graphics pipeline. 11241 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 11242 a buffer containing the grid dimensions for a Compute dispatch operation. The 11243 high half of the address is stored in the next sequential user-SGPR. Only 11244 supported by compute pipelines. 11245 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 11246 space used for the ES/GS pseudo-ring-buffer for passing data between shader 11247 stages. 11248 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 11249 pipeline instancing. 11250 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 11251 can only appear for one shader stage per pipeline. 11252 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 11253 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 11254 only appear for one shader stage per pipeline. 11255 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 11256 only appear for one shader stage per pipeline (PS). These replace color targets 11257 and are completely separate from any UAVs used by the shader. This is optional, 11258 and only used by the PS when UAV exports are used to replace color-target 11259 exports to optimize specific shaders. 11260 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 11261 some NGG pipelines to perform culling. This value contains the address of the 11262 first of two consecutive registers which provide the full GPU address. 11263 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 11264 ========== ================= =============================================================================== 11265 11266.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 11267 11268Per-Shader Table 11269################ 11270 11271Low 32 bits of the GPU address for an optional buffer in the ``.data`` 11272section of the ELF. The high 32 bits of the address match the high 32 bits 11273of the shader's program counter. 11274 11275The buffer can be anything the shader compiler needs it for, and 11276allows each shader to have its own region of the ``.data`` section. 11277Typically, this could be a table of buffer SRD's and the data pointed to 11278by the buffer SRD's, but it could be a flat-address region of memory as 11279well. Its layout and usage are defined by the shader compiler. 11280 11281Each shader's table in the ``.data`` section is referenced by the symbol 11282``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 11283hardware shader stage the data is for. E.g., 11284``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 11285 11286.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 11287 11288Spill Table 11289########### 11290 11291It is possible for a hardware shader to need access to more *user data 11292entries* than there are slots available in user data registers for one 11293or more hardware shader stages. In that case, the PAL runtime expects 11294the necessary *user data entries* to be spilled to GPU memory and use 11295one user data register to point to the spilled user data memory. The 11296value of the *user data entry* must then represent the location where 11297a shader expects to read the low 32-bits of the table's GPU virtual 11298address. The *spill table* itself represents a set of 32-bit values 11299managed by the PAL runtime in GPU-accessible memory that can be made 11300indirectly accessible to a hardware shader. 11301 11302Unspecified OS 11303-------------- 11304 11305This section provides code conventions used when the target triple OS is 11306empty (see :ref:`amdgpu-target-triples`). 11307 11308Trap Handler ABI 11309~~~~~~~~~~~~~~~~ 11310 11311For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11312not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11313instructions are handled as follows: 11314 11315 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11316 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11317 11318 =============== =============== =========================================== 11319 Usage Code Sequence Description 11320 =============== =============== =========================================== 11321 llvm.trap s_endpgm Causes wavefront to be terminated. 11322 llvm.debugtrap *none* Compiler warning given that there is no 11323 trap handler installed. 11324 =============== =============== =========================================== 11325 11326Source Languages 11327================ 11328 11329.. _amdgpu-opencl: 11330 11331OpenCL 11332------ 11333 11334When the language is OpenCL the following differences occur: 11335 113361. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 113372. The AMDGPU backend appends additional arguments to the kernel's explicit 11338 arguments for the AMDHSA OS (see 11339 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 113403. Additional metadata is generated 11341 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11342 11343 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11344 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11345 11346 ======== ==== ========= =========================================== 11347 Position Byte Byte Description 11348 Size Alignment 11349 ======== ==== ========= =========================================== 11350 1 8 8 OpenCL Global Offset X 11351 2 8 8 OpenCL Global Offset Y 11352 3 8 8 OpenCL Global Offset Z 11353 4 8 8 OpenCL address of printf buffer 11354 5 8 8 OpenCL address of virtual queue used by 11355 enqueue_kernel. 11356 6 8 8 OpenCL address of AqlWrap struct used by 11357 enqueue_kernel. 11358 7 8 8 Pointer argument used for Multi-gird 11359 synchronization. 11360 ======== ==== ========= =========================================== 11361 11362.. _amdgpu-hcc: 11363 11364HCC 11365--- 11366 11367When the language is HCC the following differences occur: 11368 113691. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11370 11371.. _amdgpu-assembler: 11372 11373Assembler 11374--------- 11375 11376AMDGPU backend has LLVM-MC based assembler which is currently in development. 11377It supports AMDGCN GFX6-GFX10. 11378 11379This section describes general syntax for instructions and operands. 11380 11381Instructions 11382~~~~~~~~~~~~ 11383 11384An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11385 11386 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11387 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11388 11389:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11390:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11391 11392The order of operands and modifiers is fixed. 11393Most modifiers are optional and may be omitted. 11394 11395Links to detailed instruction syntax description may be found in the following 11396table. Note that features under development are not included 11397in this description. 11398 11399 =================================== ======================================= 11400 Core ISA ISA Extensions 11401 =================================== ======================================= 11402 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11403 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11404 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11405 11406 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11407 11408 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11409 11410 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11411 11412 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11413 11414 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11415 11416 :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 11417 11418 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11419 11420 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11421 =================================== ======================================= 11422 11423For more information about instructions, their semantics and supported 11424combinations of operands, refer to one of instruction set architecture manuals 11425[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_, 11426[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11427 11428Operands 11429~~~~~~~~ 11430 11431Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11432 11433Modifiers 11434~~~~~~~~~ 11435 11436Detailed description of modifiers may be found 11437:doc:`here<AMDGPUModifierSyntax>`. 11438 11439Instruction Examples 11440~~~~~~~~~~~~~~~~~~~~ 11441 11442DS 11443++ 11444 11445.. code-block:: nasm 11446 11447 ds_add_u32 v2, v4 offset:16 11448 ds_write_src2_b64 v2 offset0:4 offset1:8 11449 ds_cmpst_f32 v2, v4, v6 11450 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11451 11452For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11453Manual. 11454 11455FLAT 11456++++ 11457 11458.. code-block:: nasm 11459 11460 flat_load_dword v1, v[3:4] 11461 flat_store_dwordx3 v[3:4], v[5:7] 11462 flat_atomic_swap v1, v[3:4], v5 glc 11463 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11464 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11465 11466For full list of supported instructions, refer to "FLAT instructions" in ISA 11467Manual. 11468 11469MUBUF 11470+++++ 11471 11472.. code-block:: nasm 11473 11474 buffer_load_dword v1, off, s[4:7], s1 11475 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11476 buffer_store_format_xy v[1:2], off, s[4:7], s1 11477 buffer_wbinvl1 11478 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11479 11480For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11481Manual. 11482 11483SMRD/SMEM 11484+++++++++ 11485 11486.. code-block:: nasm 11487 11488 s_load_dword s1, s[2:3], 0xfc 11489 s_load_dwordx8 s[8:15], s[2:3], s4 11490 s_load_dwordx16 s[88:103], s[2:3], s4 11491 s_dcache_inv_vol 11492 s_memtime s[4:5] 11493 11494For full list of supported instructions, refer to "Scalar Memory Operations" in 11495ISA Manual. 11496 11497SOP1 11498++++ 11499 11500.. code-block:: nasm 11501 11502 s_mov_b32 s1, s2 11503 s_mov_b64 s[0:1], 0x80000000 11504 s_cmov_b32 s1, 200 11505 s_wqm_b64 s[2:3], s[4:5] 11506 s_bcnt0_i32_b64 s1, s[2:3] 11507 s_swappc_b64 s[2:3], s[4:5] 11508 s_cbranch_join s[4:5] 11509 11510For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11511Manual. 11512 11513SOP2 11514++++ 11515 11516.. code-block:: nasm 11517 11518 s_add_u32 s1, s2, s3 11519 s_and_b64 s[2:3], s[4:5], s[6:7] 11520 s_cselect_b32 s1, s2, s3 11521 s_andn2_b32 s2, s4, s6 11522 s_lshr_b64 s[2:3], s[4:5], s6 11523 s_ashr_i32 s2, s4, s6 11524 s_bfm_b64 s[2:3], s4, s6 11525 s_bfe_i64 s[2:3], s[4:5], s6 11526 s_cbranch_g_fork s[4:5], s[6:7] 11527 11528For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11529Manual. 11530 11531SOPC 11532++++ 11533 11534.. code-block:: nasm 11535 11536 s_cmp_eq_i32 s1, s2 11537 s_bitcmp1_b32 s1, s2 11538 s_bitcmp0_b64 s[2:3], s4 11539 s_setvskip s3, s5 11540 11541For full list of supported instructions, refer to "SOPC Instructions" in ISA 11542Manual. 11543 11544SOPP 11545++++ 11546 11547.. code-block:: nasm 11548 11549 s_barrier 11550 s_nop 2 11551 s_endpgm 11552 s_waitcnt 0 ; Wait for all counters to be 0 11553 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11554 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11555 s_sethalt 9 11556 s_sleep 10 11557 s_sendmsg 0x1 11558 s_sendmsg sendmsg(MSG_INTERRUPT) 11559 s_trap 1 11560 11561For full list of supported instructions, refer to "SOPP Instructions" in ISA 11562Manual. 11563 11564Unless otherwise mentioned, little verification is performed on the operands 11565of SOPP Instructions, so it is up to the programmer to be familiar with the 11566range or acceptable values. 11567 11568VALU 11569++++ 11570 11571For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11572the assembler will automatically use optimal encoding based on its operands. To 11573force specific encoding, one can add a suffix to the opcode of the instruction: 11574 11575* _e32 for 32-bit VOP1/VOP2/VOPC 11576* _e64 for 64-bit VOP3 11577* _dpp for VOP_DPP 11578* _sdwa for VOP_SDWA 11579 11580VOP1/VOP2/VOP3/VOPC examples: 11581 11582.. code-block:: nasm 11583 11584 v_mov_b32 v1, v2 11585 v_mov_b32_e32 v1, v2 11586 v_nop 11587 v_cvt_f64_i32_e32 v[1:2], v2 11588 v_floor_f32_e32 v1, v2 11589 v_bfrev_b32_e32 v1, v2 11590 v_add_f32_e32 v1, v2, v3 11591 v_mul_i32_i24_e64 v1, v2, 3 11592 v_mul_i32_i24_e32 v1, -3, v3 11593 v_mul_i32_i24_e32 v1, -100, v3 11594 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11595 v_max_f16_e32 v1, v2, v3 11596 11597VOP_DPP examples: 11598 11599.. code-block:: nasm 11600 11601 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11602 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11603 v_mov_b32 v0, v0 wave_shl:1 11604 v_mov_b32 v0, v0 row_mirror 11605 v_mov_b32 v0, v0 row_bcast:31 11606 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11607 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11608 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11609 11610VOP_SDWA examples: 11611 11612.. code-block:: nasm 11613 11614 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11615 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11616 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11617 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11618 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11619 11620For full list of supported instructions, refer to "Vector ALU instructions". 11621 11622.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11623 11624Code Object V2 Predefined Symbols 11625~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11626 11627.. warning:: 11628 Code object V2 is not the default code object version emitted by 11629 this version of LLVM. 11630 11631The AMDGPU assembler defines and updates some symbols automatically. These 11632symbols do not affect code generation. 11633 11634.option.machine_version_major 11635+++++++++++++++++++++++++++++ 11636 11637Set to the GFX major generation number of the target being assembled for. For 11638example, when assembling for a "GFX9" target this will be set to the integer 11639value "9". The possible GFX major generation numbers are presented in 11640:ref:`amdgpu-processors`. 11641 11642.option.machine_version_minor 11643+++++++++++++++++++++++++++++ 11644 11645Set to the GFX minor generation number of the target being assembled for. For 11646example, when assembling for a "GFX810" target this will be set to the integer 11647value "1". The possible GFX minor generation numbers are presented in 11648:ref:`amdgpu-processors`. 11649 11650.option.machine_version_stepping 11651++++++++++++++++++++++++++++++++ 11652 11653Set to the GFX stepping generation number of the target being assembled for. 11654For example, when assembling for a "GFX704" target this will be set to the 11655integer value "4". The possible GFX stepping generation numbers are presented 11656in :ref:`amdgpu-processors`. 11657 11658.kernel.vgpr_count 11659++++++++++++++++++ 11660 11661Set to zero each time a 11662:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11663encountered. At each instruction, if the current value of this symbol is less 11664than or equal to the maximum VGPR number explicitly referenced within that 11665instruction then the symbol value is updated to equal that VGPR number plus 11666one. 11667 11668.kernel.sgpr_count 11669++++++++++++++++++ 11670 11671Set to zero each time a 11672:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11673encountered. At each instruction, if the current value of this symbol is less 11674than or equal to the maximum VGPR number explicitly referenced within that 11675instruction then the symbol value is updated to equal that SGPR number plus 11676one. 11677 11678.. _amdgpu-amdhsa-assembler-directives-v2: 11679 11680Code Object V2 Directives 11681~~~~~~~~~~~~~~~~~~~~~~~~~ 11682 11683.. warning:: 11684 Code object V2 is not the default code object version emitted by 11685 this version of LLVM. 11686 11687AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11688one can specify them with assembler directives. 11689 11690.hsa_code_object_version major, minor 11691+++++++++++++++++++++++++++++++++++++ 11692 11693*major* and *minor* are integers that specify the version of the HSA code 11694object that will be generated by the assembler. 11695 11696.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11697+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11698 11699 11700*major*, *minor*, and *stepping* are all integers that describe the instruction 11701set architecture (ISA) version of the assembly program. 11702 11703*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11704"AMD" and *arch* should always be equal to "AMDGPU". 11705 11706By default, the assembler will derive the ISA version, *vendor*, and *arch* 11707from the value of the -mcpu option that is passed to the assembler. 11708 11709.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11710 11711.amdgpu_hsa_kernel (name) 11712+++++++++++++++++++++++++ 11713 11714This directives specifies that the symbol with given name is a kernel entry 11715point (label) and the object should contain corresponding symbol of type 11716STT_AMDGPU_HSA_KERNEL. 11717 11718.amd_kernel_code_t 11719++++++++++++++++++ 11720 11721This directive marks the beginning of a list of key / value pairs that are used 11722to specify the amd_kernel_code_t object that will be emitted by the assembler. 11723The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11724amd_kernel_code_t values that are unspecified a default value will be used. The 11725default value for all keys is 0, with the following exceptions: 11726 11727- *amd_code_version_major* defaults to 1. 11728- *amd_kernel_code_version_minor* defaults to 2. 11729- *amd_machine_kind* defaults to 1. 11730- *amd_machine_version_major*, *machine_version_minor*, and 11731 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11732 that is passed to the assembler. 11733- *kernel_code_entry_byte_offset* defaults to 256. 11734- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11735 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11736 Note that wavefront size is specified as a power of two, so a value of **n** 11737 means a size of 2^ **n**. 11738- *call_convention* defaults to -1. 11739- *kernarg_segment_alignment*, *group_segment_alignment*, and 11740 *private_segment_alignment* default to 4. Note that alignments are specified 11741 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11742- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11743 GFX90A onwards. 11744- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11745 GFX10 onwards. 11746- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11747 11748The *.amd_kernel_code_t* directive must be placed immediately after the 11749function label and before any instructions. 11750 11751For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11752comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11753 11754.. _amdgpu-amdhsa-assembler-example-v2: 11755 11756Code Object V2 Example Source Code 11757~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11758 11759.. warning:: 11760 Code Object V2 is not the default code object version emitted by 11761 this version of LLVM. 11762 11763Here is an example of a minimal assembly source file, defining one HSA kernel: 11764 11765.. code:: 11766 :number-lines: 11767 11768 .hsa_code_object_version 1,0 11769 .hsa_code_object_isa 11770 11771 .hsatext 11772 .globl hello_world 11773 .p2align 8 11774 .amdgpu_hsa_kernel hello_world 11775 11776 hello_world: 11777 11778 .amd_kernel_code_t 11779 enable_sgpr_kernarg_segment_ptr = 1 11780 is_ptr64 = 1 11781 compute_pgm_rsrc1_vgprs = 0 11782 compute_pgm_rsrc1_sgprs = 0 11783 compute_pgm_rsrc2_user_sgpr = 2 11784 compute_pgm_rsrc1_wgp_mode = 0 11785 compute_pgm_rsrc1_mem_ordered = 0 11786 compute_pgm_rsrc1_fwd_progress = 1 11787 .end_amd_kernel_code_t 11788 11789 s_load_dwordx2 s[0:1], s[0:1] 0x0 11790 v_mov_b32 v0, 3.14159 11791 s_waitcnt lgkmcnt(0) 11792 v_mov_b32 v1, s0 11793 v_mov_b32 v2, s1 11794 flat_store_dword v[1:2], v0 11795 s_endpgm 11796 .Lfunc_end0: 11797 .size hello_world, .Lfunc_end0-hello_world 11798 11799.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11800 11801Code Object V3 to V4 Predefined Symbols 11802~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11803 11804The AMDGPU assembler defines and updates some symbols automatically. These 11805symbols do not affect code generation. 11806 11807.amdgcn.gfx_generation_number 11808+++++++++++++++++++++++++++++ 11809 11810Set to the GFX major generation number of the target being assembled for. For 11811example, when assembling for a "GFX9" target this will be set to the integer 11812value "9". The possible GFX major generation numbers are presented in 11813:ref:`amdgpu-processors`. 11814 11815.amdgcn.gfx_generation_minor 11816++++++++++++++++++++++++++++ 11817 11818Set to the GFX minor generation number of the target being assembled for. For 11819example, when assembling for a "GFX810" target this will be set to the integer 11820value "1". The possible GFX minor generation numbers are presented in 11821:ref:`amdgpu-processors`. 11822 11823.amdgcn.gfx_generation_stepping 11824+++++++++++++++++++++++++++++++ 11825 11826Set to the GFX stepping generation number of the target being assembled for. 11827For example, when assembling for a "GFX704" target this will be set to the 11828integer value "4". The possible GFX stepping generation numbers are presented 11829in :ref:`amdgpu-processors`. 11830 11831.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 11832 11833.amdgcn.next_free_vgpr 11834++++++++++++++++++++++ 11835 11836Set to zero before assembly begins. At each instruction, if the current value 11837of this symbol is less than or equal to the maximum VGPR number explicitly 11838referenced within that instruction then the symbol value is updated to equal 11839that VGPR number plus one. 11840 11841May be used to set the `.amdhsa_next_free_vgpr` directive in 11842:ref:`amdhsa-kernel-directives-table`. 11843 11844May be set at any time, e.g. manually set to zero at the start of each kernel. 11845 11846.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 11847 11848.amdgcn.next_free_sgpr 11849++++++++++++++++++++++ 11850 11851Set to zero before assembly begins. At each instruction, if the current value 11852of this symbol is less than or equal the maximum SGPR number explicitly 11853referenced within that instruction then the symbol value is updated to equal 11854that SGPR number plus one. 11855 11856May be used to set the `.amdhsa_next_free_spgr` directive in 11857:ref:`amdhsa-kernel-directives-table`. 11858 11859May be set at any time, e.g. manually set to zero at the start of each kernel. 11860 11861.. _amdgpu-amdhsa-assembler-directives-v3-v4: 11862 11863Code Object V3 to V4 Directives 11864~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11865 11866Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 11867architecture processors, and are not OS-specific. Directives which begin with 11868``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 11869``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 11870:ref:`amdgpu-processors`. 11871 11872.. _amdgpu-assembler-directive-amdgcn-target: 11873 11874.amdgcn_target <target-triple> "-" <target-id> 11875++++++++++++++++++++++++++++++++++++++++++++++ 11876 11877Optional directive which declares the ``<target-triple>-<target-id>`` supported 11878by the containing assembler source file. Used by the assembler to validate 11879command-line options such as ``-triple``, ``-mcpu``, and 11880``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 11881:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 11882 11883.. note:: 11884 11885 The target ID syntax used for code object V2 to V3 for this directive differs 11886 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 11887 11888.amdhsa_kernel <name> 11889+++++++++++++++++++++ 11890 11891Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 11892``<name>.kd``, in the current location of the current section. Only valid when 11893the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 11894instruction to execute, and does not need to be previously defined. 11895 11896Marks the beginning of a list of directives used to generate the bytes of a 11897kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 11898Directives which may appear in this list are described in 11899:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 11900be valid for the target being assembled for, and cannot be repeated. Directives 11901support the range of values specified by the field they reference in 11902:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 11903assumed to have its default value, unless it is marked as "Required", in which 11904case it is an error to omit the directive. This list of directives is 11905terminated by an ``.end_amdhsa_kernel`` directive. 11906 11907 .. table:: AMDHSA Kernel Assembler Directives 11908 :name: amdhsa-kernel-directives-table 11909 11910 ======================================================== =================== ============ =================== 11911 Directive Default Supported On Description 11912 ======================================================== =================== ============ =================== 11913 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 11914 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11915 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 11916 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11917 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 11918 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11919 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 11920 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11921 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 11922 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11923 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 11924 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11925 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 11926 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11927 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 11928 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11929 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 11930 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11931 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 11932 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11933 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 11934 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11935 Specific 11936 (wavefrontsize64) 11937 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 11938 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11939 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 11940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11941 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 11942 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11943 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 11944 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11945 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 11946 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11947 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 11948 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 11949 Possible values are defined in 11950 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 11951 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 11952 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 11953 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11954 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 11955 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11956 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11957 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 11958 Used to calculate ACCUM_OFFSET in 11959 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11960 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 11961 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11962 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11963 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 11964 scratch memory. Used to calculate 11965 GRANULATED_WAVEFRONT_SGPR_COUNT in 11966 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11967 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 11968 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 11969 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11970 (xnack) 11971 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 11972 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11973 Possible values are defined in 11974 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11975 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 11976 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11977 Possible values are defined in 11978 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 11979 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 11980 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11981 Possible values are defined in 11982 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11983 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 11984 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11985 Possible values are defined in 11986 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 11987 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 11988 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11989 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 11990 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11991 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 11992 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11993 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 11994 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 11995 Specific 11996 (tgsplit) 11997 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 11998 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 11999 Specific 12000 (cumode) 12001 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 12002 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12003 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 12004 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12005 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 12006 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12007 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 12008 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12009 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 12010 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12011 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 12012 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12013 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 12014 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12015 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 12016 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12017 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 12018 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12019 ======================================================== =================== ============ =================== 12020 12021.amdgpu_metadata 12022++++++++++++++++ 12023 12024Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 12025note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 12026 12027The contents must be in the [YAML]_ markup format, with the same structure and 12028semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 12029:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 12030 12031This directive is terminated by an ``.end_amdgpu_metadata`` directive. 12032 12033.. _amdgpu-amdhsa-assembler-example-v3-v4: 12034 12035Code Object V3 to V4 Example Source Code 12036~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12037 12038Here is an example of a minimal assembly source file, defining one HSA kernel: 12039 12040.. code:: 12041 :number-lines: 12042 12043 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12044 12045 .text 12046 .globl hello_world 12047 .p2align 8 12048 .type hello_world,@function 12049 hello_world: 12050 s_load_dwordx2 s[0:1], s[0:1] 0x0 12051 v_mov_b32 v0, 3.14159 12052 s_waitcnt lgkmcnt(0) 12053 v_mov_b32 v1, s0 12054 v_mov_b32 v2, s1 12055 flat_store_dword v[1:2], v0 12056 s_endpgm 12057 .Lfunc_end0: 12058 .size hello_world, .Lfunc_end0-hello_world 12059 12060 .rodata 12061 .p2align 6 12062 .amdhsa_kernel hello_world 12063 .amdhsa_user_sgpr_kernarg_segment_ptr 1 12064 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12065 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12066 .end_amdhsa_kernel 12067 12068 .amdgpu_metadata 12069 --- 12070 amdhsa.version: 12071 - 1 12072 - 0 12073 amdhsa.kernels: 12074 - .name: hello_world 12075 .symbol: hello_world.kd 12076 .kernarg_segment_size: 48 12077 .group_segment_fixed_size: 0 12078 .private_segment_fixed_size: 0 12079 .kernarg_segment_align: 4 12080 .wavefront_size: 64 12081 .sgpr_count: 2 12082 .vgpr_count: 3 12083 .max_flat_workgroup_size: 256 12084 ... 12085 .end_amdgpu_metadata 12086 12087If an assembly source file contains multiple kernels and/or functions, the 12088:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 12089:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 12090the ``.set <symbol>, <expression>`` directive. For example, in the case of two 12091kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 12092to group the function with the kernel that calls it and reset the symbols 12093between the two connected components: 12094 12095.. code:: 12096 :number-lines: 12097 12098 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12099 12100 // gpr tracking symbols are implicitly set to zero 12101 12102 .text 12103 .globl kern0 12104 .p2align 8 12105 .type kern0,@function 12106 kern0: 12107 // ... 12108 s_endpgm 12109 .Lkern0_end: 12110 .size kern0, .Lkern0_end-kern0 12111 12112 .rodata 12113 .p2align 6 12114 .amdhsa_kernel kern0 12115 // ... 12116 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12117 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12118 .end_amdhsa_kernel 12119 12120 // reset symbols to begin tracking usage in func1 and kern1 12121 .set .amdgcn.next_free_vgpr, 0 12122 .set .amdgcn.next_free_sgpr, 0 12123 12124 .text 12125 .hidden func1 12126 .global func1 12127 .p2align 2 12128 .type func1,@function 12129 func1: 12130 // ... 12131 s_setpc_b64 s[30:31] 12132 .Lfunc1_end: 12133 .size func1, .Lfunc1_end-func1 12134 12135 .globl kern1 12136 .p2align 8 12137 .type kern1,@function 12138 kern1: 12139 // ... 12140 s_getpc_b64 s[4:5] 12141 s_add_u32 s4, s4, func1@rel32@lo+4 12142 s_addc_u32 s5, s5, func1@rel32@lo+4 12143 s_swappc_b64 s[30:31], s[4:5] 12144 // ... 12145 s_endpgm 12146 .Lkern1_end: 12147 .size kern1, .Lkern1_end-kern1 12148 12149 .rodata 12150 .p2align 6 12151 .amdhsa_kernel kern1 12152 // ... 12153 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12154 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12155 .end_amdhsa_kernel 12156 12157These symbols cannot identify connected components in order to automatically 12158track the usage for each kernel. However, in some cases careful organization of 12159the kernels and functions in the source file means there is minimal additional 12160effort required to accurately calculate GPR usage. 12161 12162Additional Documentation 12163======================== 12164 12165.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 12166.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 12167.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 12168.. [AMD-GCN-GFX9] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 12169.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 12170.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 12171.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 12172.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 12173.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 12174.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 12175.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 12176.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 12177.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12178.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12179.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12180.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12181.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12182.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12183.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12184.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12185.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12186.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12187.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12188