1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPU/AMDGPUAsmGFX1013 22 AMDGPU/AMDGPUAsmGFX1030 23 AMDGPUModifierSyntax 24 AMDGPUOperandSyntax 25 AMDGPUInstructionSyntax 26 AMDGPUInstructionNotation 27 AMDGPUDwarfExtensionsForHeterogeneousDebugging 28 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack 29 30Introduction 31============ 32 33The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 34R600 family up until the current GCN families. It lives in the 35``llvm/lib/Target/AMDGPU`` directory. 36 37LLVM 38==== 39 40.. _amdgpu-target-triples: 41 42Target Triples 43-------------- 44 45Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 46to specify the target triple: 47 48 .. table:: AMDGPU Architectures 49 :name: amdgpu-architecture-table 50 51 ============ ============================================================== 52 Architecture Description 53 ============ ============================================================== 54 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 55 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 56 ============ ============================================================== 57 58 .. table:: AMDGPU Vendors 59 :name: amdgpu-vendor-table 60 61 ============ ============================================================== 62 Vendor Description 63 ============ ============================================================== 64 ``amd`` Can be used for all AMD GPU usage. 65 ``mesa3d`` Can be used if the OS is ``mesa3d``. 66 ============ ============================================================== 67 68 .. table:: AMDGPU Operating Systems 69 :name: amdgpu-os 70 71 ============== ============================================================ 72 OS Description 73 ============== ============================================================ 74 *<empty>* Defaults to the *unknown* OS. 75 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 76 such as: 77 78 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 79 loader on Linux. See *AMD ROCm Platform Release Notes* 80 [AMD-ROCm-Release-Notes]_ for supported hardware and 81 software. 82 - AMD's PAL runtime using the *pal-amdhsa* loader on 83 Windows. 84 85 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 86 runtime using the *pal-amdpal* loader on Windows and Linux 87 Pro. 88 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 89 3D runtime using the *mesa-mesa3d* loader on Linux. 90 ============== ============================================================ 91 92 .. table:: AMDGPU Environments 93 :name: amdgpu-environment-table 94 95 ============ ============================================================== 96 Environment Description 97 ============ ============================================================== 98 *<empty>* Default. 99 ============ ============================================================== 100 101.. _amdgpu-processors: 102 103Processors 104---------- 105 106Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 107specify the AMDGPU processor together with optional target features. See 108:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 109specific information. 110 111Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 112 113* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 114 115 116 .. table:: AMDGPU Processors 117 :name: amdgpu-processor-table 118 119 =========== =============== ============ ===== ================= =============== =============== ====================== 120 Processor Alternative Target dGPU/ Target Target OS Support Example 121 Processor Triple APU Features Properties *(see* Products 122 Architecture Supported `amdgpu-os`_ 123 *and 124 corresponding 125 runtime release 126 notes for 127 current 128 information and 129 level of 130 support)* 131 =========== =============== ============ ===== ================= =============== =============== ====================== 132 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 133 ----------------------------------------------------------------------------------------------------------------------- 134 ``r600`` ``r600`` dGPU - Does not 135 support 136 generic 137 address 138 space 139 ``r630`` ``r600`` dGPU - Does not 140 support 141 generic 142 address 143 space 144 ``rs880`` ``r600`` dGPU - Does not 145 support 146 generic 147 address 148 space 149 ``rv670`` ``r600`` dGPU - Does not 150 support 151 generic 152 address 153 space 154 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 155 ----------------------------------------------------------------------------------------------------------------------- 156 ``rv710`` ``r600`` dGPU - Does not 157 support 158 generic 159 address 160 space 161 ``rv730`` ``r600`` dGPU - Does not 162 support 163 generic 164 address 165 space 166 ``rv770`` ``r600`` dGPU - Does not 167 support 168 generic 169 address 170 space 171 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 172 ----------------------------------------------------------------------------------------------------------------------- 173 ``cedar`` ``r600`` dGPU - Does not 174 support 175 generic 176 address 177 space 178 ``cypress`` ``r600`` dGPU - Does not 179 support 180 generic 181 address 182 space 183 ``juniper`` ``r600`` dGPU - Does not 184 support 185 generic 186 address 187 space 188 ``redwood`` ``r600`` dGPU - Does not 189 support 190 generic 191 address 192 space 193 ``sumo`` ``r600`` dGPU - Does not 194 support 195 generic 196 address 197 space 198 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 199 ----------------------------------------------------------------------------------------------------------------------- 200 ``barts`` ``r600`` dGPU - Does not 201 support 202 generic 203 address 204 space 205 ``caicos`` ``r600`` dGPU - Does not 206 support 207 generic 208 address 209 space 210 ``cayman`` ``r600`` dGPU - Does not 211 support 212 generic 213 address 214 space 215 ``turks`` ``r600`` dGPU - Does not 216 support 217 generic 218 address 219 space 220 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 221 ----------------------------------------------------------------------------------------------------------------------- 222 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 223 support 224 generic 225 address 226 space 227 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 228 - ``verde`` support 229 generic 230 address 231 space 232 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 233 - ``oland`` support 234 generic 235 address 236 space 237 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 238 ----------------------------------------------------------------------------------------------------------------------- 239 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 240 flat - *pal-amdhsa* - A6 Pro-7050B 241 scratch - *pal-amdpal* - A8-7100 242 - A8 Pro-7150B 243 - A10-7300 244 - A10 Pro-7350B 245 - FX-7500 246 - A8-7200P 247 - A10-7400P 248 - FX-7600P 249 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 250 flat - *pal-amdhsa* - FirePro W9100 251 scratch - *pal-amdpal* - FirePro S9150 252 - FirePro S9170 253 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 254 flat - *pal-amdhsa* - Radeon R9 290x 255 scratch - *pal-amdpal* - Radeon R390 256 - Radeon R390x 257 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 258 - ``mullins`` flat - *pal-amdpal* - E1-2200 259 scratch - E1-2500 260 - E2-3000 261 - E2-3800 262 - A4-5000 263 - A4-5100 264 - A6-5200 265 - A4 Pro-3340B 266 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 267 flat - *pal-amdpal* - Radeon HD 8770 268 scratch - R7 260 269 - R7 260X 270 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 271 flat - *pal-amdpal* 272 scratch .. TODO:: 273 274 Add product 275 names. 276 277 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 278 ----------------------------------------------------------------------------------------------------------------------- 279 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 280 flat - *pal-amdhsa* - Pro A6-8500B 281 scratch - *pal-amdpal* - A8-8600P 282 - Pro A8-8600B 283 - FX-8800P 284 - Pro A12-8800B 285 - A10-8700P 286 - Pro A10-8700B 287 - A10-8780P 288 - A10-9600P 289 - A10-9630P 290 - A12-9700P 291 - A12-9730P 292 - FX-9800P 293 - FX-9830P 294 - E2-9010 295 - A6-9210 296 - A9-9410 297 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 298 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 299 scratch - *pal-amdpal* - Radeon R9 385 300 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 301 - *pal-amdhsa* - Radeon R9 Fury 302 - *pal-amdpal* - Radeon R9 FuryX 303 - Radeon Pro Duo 304 - FirePro S9300x2 305 - Radeon Instinct MI8 306 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 307 flat - *pal-amdhsa* - Radeon RX 480 308 scratch - *pal-amdpal* - Radeon Instinct MI6 309 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 310 flat - *pal-amdhsa* 311 scratch - *pal-amdpal* 312 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 313 flat - *pal-amdhsa* - FirePro S7100 314 scratch - *pal-amdpal* - FirePro W7100 315 - Mobile FirePro 316 M7170 317 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 318 flat - *pal-amdhsa* 319 scratch - *pal-amdpal* .. TODO:: 320 321 Add product 322 names. 323 324 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ [AMD-GCN-GFX90A-CDNA2]_ 325 ----------------------------------------------------------------------------------------------------------------------- 326 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 327 flat - *pal-amdhsa* Frontier Edition 328 scratch - *pal-amdpal* - Radeon RX Vega 56 329 - Radeon RX Vega 64 330 - Radeon RX Vega 64 331 Liquid 332 - Radeon Instinct MI25 333 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 334 flat - *pal-amdhsa* - Ryzen 5 2400G 335 scratch - *pal-amdpal* 336 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 337 - *pal-amdhsa* 338 - *pal-amdpal* .. TODO:: 339 340 Add product 341 names. 342 343 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 344 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 345 scratch - *pal-amdpal* - Radeon VII 346 - Radeon Pro VII 347 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 348 - xnack - Absolute 349 flat 350 scratch 351 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 352 flat 353 scratch .. TODO:: 354 355 Add product 356 names. 357 358 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 359 - tgsplit flat 360 - xnack scratch .. TODO:: 361 - Packed 362 work-item Add product 363 IDs names. 364 365 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 366 flat - Ryzen 7 4700GE 367 scratch - Ryzen 5 4600G 368 - Ryzen 5 4600GE 369 - Ryzen 3 4300G 370 - Ryzen 3 4300GE 371 - Ryzen Pro 4000G 372 - Ryzen 7 Pro 4700G 373 - Ryzen 7 Pro 4750GE 374 - Ryzen 5 Pro 4650G 375 - Ryzen 5 Pro 4650GE 376 - Ryzen 3 Pro 4350G 377 - Ryzen 3 Pro 4350GE 378 379 ``gfx940`` ``amdgcn`` dGPU - sramecc - Architected *TBA* 380 - tgsplit flat 381 - xnack scratch .. TODO:: 382 - Packed 383 work-item Add product 384 IDs names. 385 386 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 387 ----------------------------------------------------------------------------------------------------------------------- 388 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 389 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 390 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 391 - Radeon Pro 5600M 392 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 393 - wavefrontsize64 - Absolute - *pal-amdhsa* 394 - xnack flat - *pal-amdpal* 395 scratch 396 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 397 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 398 - xnack scratch - *pal-amdpal* 399 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 400 - wavefrontsize64 flat - *pal-amdhsa* 401 - xnack scratch - *pal-amdpal* .. TODO:: 402 403 Add product 404 names. 405 406 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 407 ----------------------------------------------------------------------------------------------------------------------- 408 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 409 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 410 scratch - *pal-amdpal* - Radeon RX 6900 XT 411 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 412 - wavefrontsize64 flat - *pal-amdhsa* 413 scratch - *pal-amdpal* 414 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 415 - wavefrontsize64 flat - *pal-amdhsa* 416 scratch - *pal-amdpal* .. TODO:: 417 418 Add product 419 names. 420 421 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 422 - wavefrontsize64 flat 423 scratch .. TODO:: 424 425 Add product 426 names. 427 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 428 - wavefrontsize64 flat 429 scratch .. TODO:: 430 431 Add product 432 names. 433 434 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 435 - wavefrontsize64 flat 436 scratch .. TODO:: 437 Add product 438 names. 439 440 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 441 - wavefrontsize64 flat 442 scratch .. TODO:: 443 444 Add product 445 names. 446 447 =========== =============== ============ ===== ================= =============== =============== ====================== 448 449.. _amdgpu-target-features: 450 451Target Features 452--------------- 453 454Target features control how code is generated to support certain 455processor specific features. Not all target features are supported by 456all processors. The runtime must ensure that the features supported by 457the device used to execute the code match the features enabled when 458generating the code. A mismatch of features may result in incorrect 459execution, or a reduction in performance. 460 461The target features supported by each processor is listed in 462:ref:`amdgpu-processor-table`. 463 464Target features are controlled by exactly one of the following Clang 465options: 466 467``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 468 469 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 470 optional components of the target ID. If omitted, the target feature has the 471 ``any`` value. See :ref:`amdgpu-target-id`. 472 473``-m[no-]<target-feature>`` 474 475 Target features not specified by the target ID are specified using a 476 separate option. These target features can have an ``on`` or ``off`` 477 value. ``on`` is specified by omitting the ``no-`` prefix, and 478 ``off`` is specified by including the ``no-`` prefix. The default 479 if not specified is ``off``. 480 481For example: 482 483``-mcpu=gfx908:xnack+`` 484 Enable the ``xnack`` feature. 485``-mcpu=gfx908:xnack-`` 486 Disable the ``xnack`` feature. 487``-mcumode`` 488 Enable the ``cumode`` feature. 489``-mno-cumode`` 490 Disable the ``cumode`` feature. 491 492 .. table:: AMDGPU Target Features 493 :name: amdgpu-target-features-table 494 495 =============== ============================ ================================================== 496 Target Feature Clang Option to Control Description 497 Name 498 =============== ============================ ================================================== 499 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 500 when generating code for kernels. When disabled 501 native WGP wavefront execution mode is used, 502 when enabled CU wavefront execution mode is used 503 (see :ref:`amdgpu-amdhsa-memory-model`). 504 505 sramecc - ``-mcpu`` If specified, generate code that can only be 506 - ``--offload-arch`` loaded and executed in a process that has a 507 matching setting for SRAMECC. 508 509 If not specified for code object V2 to V3, generate 510 code that can be loaded and executed in a process 511 with SRAMECC enabled. 512 513 If not specified for code object V4 or above, generate 514 code that can be loaded and executed in a process 515 with either setting of SRAMECC. 516 517 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 518 work-groups are launched in threadgroup split mode. 519 When enabled the waves of a work-group may be 520 launched in different CUs. 521 522 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 523 generating code for kernels. When disabled 524 native wavefront size 32 is used, when enabled 525 wavefront size 64 is used. 526 527 xnack - ``-mcpu`` If specified, generate code that can only be 528 - ``--offload-arch`` loaded and executed in a process that has a 529 matching setting for XNACK replay. 530 531 If not specified for code object V2 to V3, generate 532 code that can be loaded and executed in a process 533 with XNACK replay enabled. 534 535 If not specified for code object V4 or above, generate 536 code that can be loaded and executed in a process 537 with either setting of XNACK replay. 538 539 XNACK replay can be used for demand paging and 540 page migration. If enabled in the device, then if 541 a page fault occurs the code may execute 542 incorrectly unless generated with XNACK replay 543 enabled, or generated for code object V4 or above without 544 specifying XNACK replay. Executing code that was 545 generated with XNACK replay enabled, or generated 546 for code object V4 or above without specifying XNACK replay, 547 on a device that does not have XNACK replay 548 enabled will execute correctly but may be less 549 performant than code generated for XNACK replay 550 disabled. 551 =============== ============================ ================================================== 552 553.. _amdgpu-target-id: 554 555Target ID 556--------- 557 558AMDGPU supports target IDs. See `Clang Offload Bundler 559<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 560description. The AMDGPU target specific information is: 561 562**processor** 563 Is an AMDGPU processor or alternative processor name specified in 564 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 565 the primary processor and alternative processor names. The canonical form 566 target ID only allow the primary processor name. 567 568**target-feature** 569 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 570 is supported by the processor. The target features supported by each processor 571 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 572 a target ID are marked as being controlled by ``-mcpu`` and 573 ``--offload-arch``. Each target feature must appear at most once in a target 574 ID. The non-canonical form target ID allows the target features to be 575 specified in any order. The canonical form target ID requires the target 576 features to be specified in alphabetic order. 577 578.. _amdgpu-target-id-v2-v3: 579 580Code Object V2 to V3 Target ID 581~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 582 583The target ID syntax for code object V2 to V3 is the same as defined in `Clang 584Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 585when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 586directive and the bundle entry ID. In those cases it has the following BNF 587syntax: 588 589.. code:: 590 591 <target-id> ::== <processor> ( "+" <target-feature> )* 592 593Where a target feature is omitted if *Off* and present if *On* or *Any*. 594 595.. note:: 596 597 The code object V2 to V3 cannot represent *Any* and treats it the same as 598 *On*. 599 600.. _amdgpu-embedding-bundled-objects: 601 602Embedding Bundled Code Objects 603------------------------------ 604 605AMDGPU supports the HIP and OpenMP languages that perform code object embedding 606as described in `Clang Offload Bundler 607<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 608 609.. note:: 610 611 The target ID syntax used for code object V2 to V3 for a bundle entry ID 612 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 613 614.. _amdgpu-address-spaces: 615 616Address Spaces 617-------------- 618 619The AMDGPU architecture supports a number of memory address spaces. The address 620space names use the OpenCL standard names, with some additions. 621 622The AMDGPU address spaces correspond to target architecture specific LLVM 623address space numbers used in LLVM IR. 624 625The AMDGPU address spaces are described in 626:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 627supported for the ``amdgcn`` target. 628 629 .. table:: AMDGPU Address Spaces 630 :name: amdgpu-address-spaces-table 631 632 ================================= =============== =========== ================ ======= ============================ 633 .. 64-Bit Process Address Space 634 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 635 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 636 Space Number Name Name Size 637 ================================= =============== =========== ================ ======= ============================ 638 Generic 0 flat flat 64 0x0000000000000000 639 Global 1 global global 64 0x0000000000000000 640 Region 2 N/A GDS 32 *not implemented for AMDHSA* 641 Local 3 group LDS 32 0xFFFFFFFF 642 Constant 4 constant *same as global* 64 0x0000000000000000 643 Private 5 private scratch 32 0xFFFFFFFF 644 Constant 32-bit 6 *TODO* 0x00000000 645 Buffer Fat Pointer (experimental) 7 *TODO* 646 ================================= =============== =========== ================ ======= ============================ 647 648**Generic** 649 The generic address space is supported unless the *Target Properties* column 650 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 651 space*. 652 653 The generic address space uses the hardware flat address support for two fixed 654 ranges of virtual addresses (the private and local apertures), that are 655 outside the range of addressable global memory, to map from a flat address to 656 a private or local address. This uses FLAT instructions that can take a flat 657 address and access global, private (scratch), and group (LDS) memory depending 658 on if the address is within one of the aperture ranges. 659 660 Flat access to scratch requires hardware aperture setup and setup in the 661 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 662 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 663 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 664 665 To convert between a private or group address space address (termed a segment 666 address) and a flat address the base address of the corresponding aperture 667 can be used. For GFX7-GFX8 these are available in the 668 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 669 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 670 GFX9-GFX10 the aperture base addresses are directly available as inline 671 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 672 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 673 aligned to 2^32 which makes it easier to convert from flat to segment or 674 segment to flat. 675 676 A global address space address has the same value when used as a flat address 677 so no conversion is needed. 678 679**Global and Constant** 680 The global and constant address spaces both use global virtual addresses, 681 which are the same virtual address space used by the CPU. However, some 682 virtual addresses may only be accessible to the CPU, some only accessible 683 by the GPU, and some by both. 684 685 Using the constant address space indicates that the data will not change 686 during the execution of the kernel. This allows scalar read instructions to 687 be used. As the constant address space could only be modified on the host 688 side, a generic pointer loaded from the constant address space is safe to be 689 assumed as a global pointer since only the device global memory is visible 690 and managed on the host side. The vector and scalar L1 caches are invalidated 691 of volatile data before each kernel dispatch execution to allow constant 692 memory to change values between kernel dispatches. 693 694**Region** 695 The region address space uses the hardware Global Data Store (GDS). All 696 wavefronts executing on the same device will access the same memory for any 697 given region address. However, the same region address accessed by wavefronts 698 executing on different devices will access different memory. It is higher 699 performance than global memory. It is allocated by the runtime. The data 700 store (DS) instructions can be used to access it. 701 702**Local** 703 The local address space uses the hardware Local Data Store (LDS) which is 704 automatically allocated when the hardware creates the wavefronts of a 705 work-group, and freed when all the wavefronts of a work-group have 706 terminated. All wavefronts belonging to the same work-group will access the 707 same memory for any given local address. However, the same local address 708 accessed by wavefronts belonging to different work-groups will access 709 different memory. It is higher performance than global memory. The data store 710 (DS) instructions can be used to access it. 711 712**Private** 713 The private address space uses the hardware scratch memory support which 714 automatically allocates memory when it creates a wavefront and frees it when 715 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 716 given private address will be different to the memory accessed by another lane 717 of the same or different wavefront for the same private address. 718 719 If a kernel dispatch uses scratch, then the hardware allocates memory from a 720 pool of backing memory allocated by the runtime for each wavefront. The lanes 721 of the wavefront access this using dword (4 byte) interleaving. The mapping 722 used from private address to backing memory address is: 723 724 ``wavefront-scratch-base + 725 ((private-address / 4) * wavefront-size * 4) + 726 (wavefront-lane-id * 4) + (private-address % 4)`` 727 728 If each lane of a wavefront accesses the same private address, the 729 interleaving results in adjacent dwords being accessed and hence requires 730 fewer cache lines to be fetched. 731 732 There are different ways that the wavefront scratch base address is 733 determined by a wavefront (see 734 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 735 736 Scratch memory can be accessed in an interleaved manner using buffer 737 instructions with the scratch buffer descriptor and per wavefront scratch 738 offset, by the scratch instructions, or by flat instructions. Multi-dword 739 access is not supported except by flat and scratch instructions in 740 GFX9-GFX10. 741 742**Constant 32-bit** 743 *TODO* 744 745**Buffer Fat Pointer** 746 The buffer fat pointer is an experimental address space that is currently 747 unsupported in the backend. It exposes a non-integral pointer that is in 748 the future intended to support the modelling of 128-bit buffer descriptors 749 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 750 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 751 model the buffer descriptors used heavily in graphics workloads targeting 752 the backend. 753 754.. _amdgpu-memory-scopes: 755 756Memory Scopes 757------------- 758 759This section provides LLVM memory synchronization scopes supported by the AMDGPU 760backend memory model when the target triple OS is ``amdhsa`` (see 761:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 762 763The memory model supported is based on the HSA memory model [HSA]_ which is 764based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 765relation is transitive over the synchronizes-with relation independent of scope 766and synchronizes-with allows the memory scope instances to be inclusive (see 767table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 768 769This is different to the OpenCL [OpenCL]_ memory model which does not have scope 770inclusion and requires the memory scopes to exactly match. However, this 771is conservatively correct for OpenCL. 772 773 .. table:: AMDHSA LLVM Sync Scopes 774 :name: amdgpu-amdhsa-llvm-sync-scopes-table 775 776 ======================= =================================================== 777 LLVM Sync Scope Description 778 ======================= =================================================== 779 *none* The default: ``system``. 780 781 Synchronizes with, and participates in modification 782 and seq_cst total orderings with, other operations 783 (except image operations) for all address spaces 784 (except private, or generic that accesses private) 785 provided the other operation's sync scope is: 786 787 - ``system``. 788 - ``agent`` and executed by a thread on the same 789 agent. 790 - ``workgroup`` and executed by a thread in the 791 same work-group. 792 - ``wavefront`` and executed by a thread in the 793 same wavefront. 794 795 ``agent`` Synchronizes with, and participates in modification 796 and seq_cst total orderings with, other operations 797 (except image operations) for all address spaces 798 (except private, or generic that accesses private) 799 provided the other operation's sync scope is: 800 801 - ``system`` or ``agent`` and executed by a thread 802 on the same agent. 803 - ``workgroup`` and executed by a thread in the 804 same work-group. 805 - ``wavefront`` and executed by a thread in the 806 same wavefront. 807 808 ``workgroup`` Synchronizes with, and participates in modification 809 and seq_cst total orderings with, other operations 810 (except image operations) for all address spaces 811 (except private, or generic that accesses private) 812 provided the other operation's sync scope is: 813 814 - ``system``, ``agent`` or ``workgroup`` and 815 executed by a thread in the same work-group. 816 - ``wavefront`` and executed by a thread in the 817 same wavefront. 818 819 ``wavefront`` Synchronizes with, and participates in modification 820 and seq_cst total orderings with, other operations 821 (except image operations) for all address spaces 822 (except private, or generic that accesses private) 823 provided the other operation's sync scope is: 824 825 - ``system``, ``agent``, ``workgroup`` or 826 ``wavefront`` and executed by a thread in the 827 same wavefront. 828 829 ``singlethread`` Only synchronizes with and participates in 830 modification and seq_cst total orderings with, 831 other operations (except image operations) running 832 in the same thread for all address spaces (for 833 example, in signal handlers). 834 835 ``one-as`` Same as ``system`` but only synchronizes with other 836 operations within the same address space. 837 838 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 839 operations within the same address space. 840 841 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 842 other operations within the same address space. 843 844 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 845 other operations within the same address space. 846 847 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 848 other operations within the same address space. 849 ======================= =================================================== 850 851LLVM IR Intrinsics 852------------------ 853 854The AMDGPU backend implements the following LLVM IR intrinsics. 855 856*This section is WIP.* 857 858.. TODO:: 859 860 List AMDGPU intrinsics. 861 862LLVM IR Attributes 863------------------ 864 865The AMDGPU backend supports the following LLVM IR attributes. 866 867 .. table:: AMDGPU LLVM IR Attributes 868 :name: amdgpu-llvm-ir-attributes-table 869 870 ======================================= ========================================================== 871 LLVM Attribute Description 872 ======================================= ========================================================== 873 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 874 will be specified when the kernel is dispatched. Generated 875 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 876 The implied default value is 1,1024. 877 878 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 879 argument block size for the implicit arguments. This 880 varies by OS and language (for OpenCL see 881 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 882 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 883 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 884 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 885 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 886 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 887 execution unit. Generated by the ``amdgpu_waves_per_eu`` 888 CLANG attribute [CLANG-ATTR]_. This is an optimization hint, 889 and the backend may not be able to satisfy the request. If 890 the specified range is incompatible with the function's 891 "amdgpu-flat-work-group-size" value, the implied occupancy 892 bounds by the workgroup size takes precedence. 893 894 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 895 mode register to be set on entry. Overrides the default for 896 the calling convention. 897 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 898 the mode register to be set on entry. Overrides the default 899 for the calling convention. 900 901 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 902 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 903 attribute, or reached through a call site marked with this attribute, 904 the value returned by the intrinsic is undefined. The backend can 905 generally infer this during code generation, so typically there is no 906 benefit to frontends marking functions with this. 907 908 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 909 llvm.amdgcn.workitem.id.y intrinsic. 910 911 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 912 llvm.amdgcn.workitem.id.z intrinsic. 913 914 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 915 llvm.amdgcn.workgroup.id.x intrinsic. 916 917 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 918 llvm.amdgcn.workgroup.id.y intrinsic. 919 920 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 921 llvm.amdgcn.workgroup.id.z intrinsic. 922 923 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 924 llvm.amdgcn.dispatch.ptr intrinsic. 925 926 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 927 llvm.amdgcn.implicitarg.ptr intrinsic. 928 929 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 930 llvm.amdgcn.dispatch.id intrinsic. 931 932 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 933 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 934 attributes, the queue pointer may be required in situations where the 935 intrinsic call does not directly appear in the program. Some subtargets 936 require the queue pointer for to handle some addrspacecasts, as well 937 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 938 llvm.debug intrinsics. 939 940 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 941 kernel argument that holds the pointer to the hostcall buffer. If this 942 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 943 944 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 945 kernel argument that holds the pointer to an initialized memory buffer 946 that conforms to the requirements of the malloc/free device library V1 947 version implementation. If this attribute is absent, then the 948 amdgpu-no-implicitarg-ptr is also removed. 949 950 ======================================= ========================================================== 951 952.. _amdgpu-elf-code-object: 953 954ELF Code Object 955=============== 956 957The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 958can be linked by ``lld`` to produce a standard ELF shared code object which can 959be loaded and executed on an AMDGPU target. 960 961.. _amdgpu-elf-header: 962 963Header 964------ 965 966The AMDGPU backend uses the following ELF header: 967 968 .. table:: AMDGPU ELF Header 969 :name: amdgpu-elf-header-table 970 971 ========================== =============================== 972 Field Value 973 ========================== =============================== 974 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 975 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 976 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 977 - ``ELFOSABI_AMDGPU_HSA`` 978 - ``ELFOSABI_AMDGPU_PAL`` 979 - ``ELFOSABI_AMDGPU_MESA3D`` 980 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 981 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 982 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 983 - ``ELFABIVERSION_AMDGPU_HSA_V5`` 984 - ``ELFABIVERSION_AMDGPU_PAL`` 985 - ``ELFABIVERSION_AMDGPU_MESA3D`` 986 ``e_type`` - ``ET_REL`` 987 - ``ET_DYN`` 988 ``e_machine`` ``EM_AMDGPU`` 989 ``e_entry`` 0 990 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 991 :ref:`amdgpu-elf-header-e_flags-table-v3`, 992 and :ref:`amdgpu-elf-header-e_flags-table-v4-onwards` 993 ========================== =============================== 994 995.. 996 997 .. table:: AMDGPU ELF Header Enumeration Values 998 :name: amdgpu-elf-header-enumeration-values-table 999 1000 =============================== ===== 1001 Name Value 1002 =============================== ===== 1003 ``EM_AMDGPU`` 224 1004 ``ELFOSABI_NONE`` 0 1005 ``ELFOSABI_AMDGPU_HSA`` 64 1006 ``ELFOSABI_AMDGPU_PAL`` 65 1007 ``ELFOSABI_AMDGPU_MESA3D`` 66 1008 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 1009 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 1010 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 1011 ``ELFABIVERSION_AMDGPU_HSA_V5`` 3 1012 ``ELFABIVERSION_AMDGPU_PAL`` 0 1013 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 1014 =============================== ===== 1015 1016``e_ident[EI_CLASS]`` 1017 The ELF class is: 1018 1019 * ``ELFCLASS32`` for ``r600`` architecture. 1020 1021 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 1022 process address space applications. 1023 1024``e_ident[EI_DATA]`` 1025 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 1026 1027``e_ident[EI_OSABI]`` 1028 One of the following AMDGPU target architecture specific OS ABIs 1029 (see :ref:`amdgpu-os`): 1030 1031 * ``ELFOSABI_NONE`` for *unknown* OS. 1032 1033 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 1034 1035 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1036 1037 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1038 1039``e_ident[EI_ABIVERSION]`` 1040 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1041 object conforms: 1042 1043 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1044 runtime ABI for code object V2. Specify using the Clang option 1045 ``-mcode-object-version=2``. 1046 1047 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1048 runtime ABI for code object V3. Specify using the Clang option 1049 ``-mcode-object-version=3``. 1050 1051 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1052 runtime ABI for code object V4. Specify using the Clang option 1053 ``-mcode-object-version=4``. This is the default code object 1054 version if not specified. 1055 1056 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA 1057 runtime ABI for code object V5. Specify using the Clang option 1058 ``-mcode-object-version=5``. 1059 1060 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1061 runtime ABI. 1062 1063 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1064 3D runtime ABI. 1065 1066``e_type`` 1067 Can be one of the following values: 1068 1069 1070 ``ET_REL`` 1071 The type produced by the AMDGPU backend compiler as it is relocatable code 1072 object. 1073 1074 ``ET_DYN`` 1075 The type produced by the linker as it is a shared code object. 1076 1077 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1078 1079``e_machine`` 1080 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1081 by the ``r600`` and ``amdgcn`` architectures (see 1082 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1083 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1084 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1085 ``e_flags`` for code object V3 and above (see 1086 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1087 :ref:`amdgpu-elf-header-e_flags-table-v4-onwards`). 1088 1089``e_entry`` 1090 The entry point is 0 as the entry points for individual kernels must be 1091 selected in order to invoke them through AQL packets. 1092 1093``e_flags`` 1094 The AMDGPU backend uses the following ELF header flags: 1095 1096 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1097 :name: amdgpu-elf-header-e_flags-v2-table 1098 1099 ===================================== ===== ============================= 1100 Name Value Description 1101 ===================================== ===== ============================= 1102 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1103 target feature is 1104 enabled for all code 1105 contained in the code object. 1106 If the processor 1107 does not support the 1108 ``xnack`` target 1109 feature then must 1110 be 0. 1111 See 1112 :ref:`amdgpu-target-features`. 1113 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1114 handler is enabled for all 1115 code contained in the code 1116 object. If the processor 1117 does not support a trap 1118 handler then must be 0. 1119 See 1120 :ref:`amdgpu-target-features`. 1121 ===================================== ===== ============================= 1122 1123 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1124 :name: amdgpu-elf-header-e_flags-table-v3 1125 1126 ================================= ===== ============================= 1127 Name Value Description 1128 ================================= ===== ============================= 1129 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1130 mask for 1131 ``EF_AMDGPU_MACH_xxx`` values 1132 defined in 1133 :ref:`amdgpu-ef-amdgpu-mach-table`. 1134 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1135 target feature is 1136 enabled for all code 1137 contained in the code object. 1138 If the processor 1139 does not support the 1140 ``xnack`` target 1141 feature then must 1142 be 0. 1143 See 1144 :ref:`amdgpu-target-features`. 1145 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1146 target feature is 1147 enabled for all code 1148 contained in the code object. 1149 If the processor 1150 does not support the 1151 ``sramecc`` target 1152 feature then must 1153 be 0. 1154 See 1155 :ref:`amdgpu-target-features`. 1156 ================================= ===== ============================= 1157 1158 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and After 1159 :name: amdgpu-elf-header-e_flags-table-v4-onwards 1160 1161 ============================================ ===== =================================== 1162 Name Value Description 1163 ============================================ ===== =================================== 1164 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1165 mask for 1166 ``EF_AMDGPU_MACH_xxx`` values 1167 defined in 1168 :ref:`amdgpu-ef-amdgpu-mach-table`. 1169 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1170 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1171 values. 1172 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1173 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1174 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1175 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1176 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1177 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1178 values. 1179 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1180 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1181 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1182 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1183 ============================================ ===== =================================== 1184 1185 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1186 :name: amdgpu-ef-amdgpu-mach-table 1187 1188 ==================================== ========== ============================= 1189 Name Value Description (see 1190 :ref:`amdgpu-processor-table`) 1191 ==================================== ========== ============================= 1192 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1193 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1194 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1195 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1196 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1197 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1198 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1199 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1200 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1201 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1202 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1203 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1204 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1205 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1206 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1207 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1208 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1209 *reserved* 0x011 - Reserved for ``r600`` 1210 0x01f architecture processors. 1211 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1212 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1213 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1214 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1215 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1216 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1217 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1218 *reserved* 0x027 Reserved. 1219 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1220 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1221 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1222 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1223 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1224 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1225 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1226 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1227 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1228 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1229 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1230 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1231 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1232 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1233 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1234 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1235 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1236 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1237 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1238 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1239 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1240 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1241 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1242 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1243 ``EF_AMDGPU_MACH_AMDGCN_GFX940`` 0x040 ``gfx940`` 1244 *reserved* 0x041 Reserved. 1245 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1246 *reserved* 0x043 Reserved. 1247 *reserved* 0x044 Reserved. 1248 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036`` 1249 ==================================== ========== ============================= 1250 1251Sections 1252-------- 1253 1254An AMDGPU target ELF code object has the standard ELF sections which include: 1255 1256 .. table:: AMDGPU ELF Sections 1257 :name: amdgpu-elf-sections-table 1258 1259 ================== ================ ================================= 1260 Name Type Attributes 1261 ================== ================ ================================= 1262 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1263 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1264 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1265 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1266 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1267 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1268 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1269 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1270 ``.note`` ``SHT_NOTE`` *none* 1271 ``.rela``\ *name* ``SHT_RELA`` *none* 1272 ``.rela.dyn`` ``SHT_RELA`` *none* 1273 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1274 ``.shstrtab`` ``SHT_STRTAB`` *none* 1275 ``.strtab`` ``SHT_STRTAB`` *none* 1276 ``.symtab`` ``SHT_SYMTAB`` *none* 1277 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1278 ================== ================ ================================= 1279 1280These sections have their standard meanings (see [ELF]_) and are only generated 1281if needed. 1282 1283``.debug``\ *\** 1284 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1285 information on the DWARF produced by the AMDGPU backend. 1286 1287``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1288 The standard sections used by a dynamic loader. 1289 1290``.note`` 1291 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1292 backend. 1293 1294``.rela``\ *name*, ``.rela.dyn`` 1295 For relocatable code objects, *name* is the name of the section that the 1296 relocation records apply. For example, ``.rela.text`` is the section name for 1297 relocation records associated with the ``.text`` section. 1298 1299 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1300 records from each of the relocatable code object's ``.rela``\ *name* sections. 1301 1302 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1303 the AMDGPU backend. 1304 1305``.text`` 1306 The executable machine code for the kernels and functions they call. Generated 1307 as position independent code. See :ref:`amdgpu-code-conventions` for 1308 information on conventions used in the isa generation. 1309 1310.. _amdgpu-note-records: 1311 1312Note Records 1313------------ 1314 1315The AMDGPU backend code object contains ELF note records in the ``.note`` 1316section. The set of generated notes and their semantics depend on the code 1317object version; see :ref:`amdgpu-note-records-v2` and 1318:ref:`amdgpu-note-records-v3-onwards`. 1319 1320As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1321must be generated after the ``name`` field to ensure the ``desc`` field is 4 1322byte aligned. In addition, minimal zero-byte padding must be generated to 1323ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1324field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1325alignment. 1326 1327.. _amdgpu-note-records-v2: 1328 1329Code Object V2 Note Records 1330~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1331 1332.. warning:: 1333 Code object V2 is not the default code object version emitted by 1334 this version of LLVM. 1335 1336The AMDGPU backend code object uses the following ELF note record in the 1337``.note`` section when compiling for code object V2. 1338 1339The note record vendor field is "AMD". 1340 1341Additional note records may be present, but any which are not documented here 1342are deprecated and should not be used. 1343 1344 .. table:: AMDGPU Code Object V2 ELF Note Records 1345 :name: amdgpu-elf-note-records-v2-table 1346 1347 ===== ===================================== ====================================== 1348 Name Type Description 1349 ===== ===================================== ====================================== 1350 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1351 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1352 Finalizer and not the LLVM compiler. 1353 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1354 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1355 YAML [YAML]_ textual format. 1356 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1357 ===== ===================================== ====================================== 1358 1359.. 1360 1361 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1362 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1363 1364 ===================================== ===== 1365 Name Value 1366 ===================================== ===== 1367 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1368 ``NT_AMD_HSA_HSAIL`` 2 1369 ``NT_AMD_HSA_ISA_VERSION`` 3 1370 *reserved* 4-9 1371 ``NT_AMD_HSA_METADATA`` 10 1372 ``NT_AMD_HSA_ISA_NAME`` 11 1373 ===================================== ===== 1374 1375``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1376 Specifies the code object version number. The description field has the 1377 following layout: 1378 1379 .. code:: c 1380 1381 struct amdgpu_hsa_note_code_object_version_s { 1382 uint32_t major_version; 1383 uint32_t minor_version; 1384 }; 1385 1386 The ``major_version`` has a value less than or equal to 2. 1387 1388``NT_AMD_HSA_HSAIL`` 1389 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1390 field has the following layout: 1391 1392 .. code:: c 1393 1394 struct amdgpu_hsa_note_hsail_s { 1395 uint32_t hsail_major_version; 1396 uint32_t hsail_minor_version; 1397 uint8_t profile; 1398 uint8_t machine_model; 1399 uint8_t default_float_round; 1400 }; 1401 1402``NT_AMD_HSA_ISA_VERSION`` 1403 Specifies the target ISA version. The description field has the following layout: 1404 1405 .. code:: c 1406 1407 struct amdgpu_hsa_note_isa_s { 1408 uint16_t vendor_name_size; 1409 uint16_t architecture_name_size; 1410 uint32_t major; 1411 uint32_t minor; 1412 uint32_t stepping; 1413 char vendor_and_architecture_name[1]; 1414 }; 1415 1416 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1417 vendor and architecture names respectively, including the NUL character. 1418 1419 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1420 vendor, immediately followed by the NUL terminated string for the 1421 architecture. 1422 1423 This note record is used by the HSA runtime loader. 1424 1425 Code object V2 only supports a limited number of processors and has fixed 1426 settings for target features. See 1427 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1428 processors and the corresponding target ID. In the table the note record ISA 1429 name is a concatenation of the vendor name, architecture name, major, minor, 1430 and stepping separated by a ":". 1431 1432 The target ID column shows the processor name and fixed target features used 1433 by the LLVM compiler. The LLVM compiler does not generate a 1434 ``NT_AMD_HSA_HSAIL`` note record. 1435 1436 A code object generated by the Finalizer also uses code object V2 and always 1437 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1438 ``sramecc`` target feature is as shown in 1439 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1440 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1441 bit. 1442 1443``NT_AMD_HSA_ISA_NAME`` 1444 Specifies the target ISA name as a non-NUL terminated string. 1445 1446 This note record is not used by the HSA runtime loader. 1447 1448 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1449 V2's limited support of processors and fixed settings for target features. 1450 1451 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1452 from the string to the corresponding target ID. If the ``xnack`` target 1453 feature is supported and enabled, the string produced by the LLVM compiler 1454 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1455 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1456 1457``NT_AMD_HSA_METADATA`` 1458 Specifies extensible metadata associated with the code objects executed on HSA 1459 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1460 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1461 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1462 metadata string. 1463 1464 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1465 :name: amdgpu-elf-note-record-supported_processors-v2-table 1466 1467 ===================== ========================== 1468 Note Record ISA Name Target ID 1469 ===================== ========================== 1470 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1471 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1472 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1473 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1474 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1475 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1476 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1477 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1478 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1479 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1480 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1481 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1482 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1483 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1484 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1485 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1486 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1487 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1488 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1489 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1490 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1491 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1492 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1493 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1494 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1495 ===================== ========================== 1496 1497.. _amdgpu-note-records-v3-onwards: 1498 1499Code Object V3 and Above Note Records 1500~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1501 1502The AMDGPU backend code object uses the following ELF note record in the 1503``.note`` section when compiling for code object V3 and above. 1504 1505The note record vendor field is "AMDGPU". 1506 1507Additional note records may be present, but any which are not documented here 1508are deprecated and should not be used. 1509 1510 .. table:: AMDGPU Code Object V3 and Above ELF Note Records 1511 :name: amdgpu-elf-note-records-table-v3-onwards 1512 1513 ======== ============================== ====================================== 1514 Name Type Description 1515 ======== ============================== ====================================== 1516 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1517 binary format. 1518 ======== ============================== ====================================== 1519 1520.. 1521 1522 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values 1523 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards 1524 1525 ============================== ===== 1526 Name Value 1527 ============================== ===== 1528 *reserved* 0-31 1529 ``NT_AMDGPU_METADATA`` 32 1530 ============================== ===== 1531 1532``NT_AMDGPU_METADATA`` 1533 Specifies extensible metadata associated with an AMDGPU code object. It is 1534 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1535 :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 1536 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and 1537 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the 1538 ``amdhsa`` OS. 1539 1540.. _amdgpu-symbols: 1541 1542Symbols 1543------- 1544 1545Symbols include the following: 1546 1547 .. table:: AMDGPU ELF Symbols 1548 :name: amdgpu-elf-symbols-table 1549 1550 ===================== ================== ================ ================== 1551 Name Type Section Description 1552 ===================== ================== ================ ================== 1553 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1554 - ``.rodata`` 1555 - ``.bss`` 1556 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1557 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1558 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1559 ===================== ================== ================ ================== 1560 1561Global variable 1562 Global variables both used and defined by the compilation unit. 1563 1564 If the symbol is defined in the compilation unit then it is allocated in the 1565 appropriate section according to if it has initialized data or is readonly. 1566 1567 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1568 will resolve relocations using the definition provided by another code object 1569 or explicitly defined by the runtime. 1570 1571 If the symbol resides in local/group memory (LDS) then its section is the 1572 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1573 ``st_value`` field describes alignment requirements as it does for common 1574 symbols. 1575 1576 .. TODO:: 1577 1578 Add description of linked shared object symbols. Seems undefined symbols 1579 are marked as STT_NOTYPE. 1580 1581Kernel descriptor 1582 Every HSA kernel has an associated kernel descriptor. It is the address of the 1583 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1584 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1585 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1586 1587Kernel entry point 1588 Every HSA kernel also has a symbol for its machine code entry point. 1589 1590.. _amdgpu-relocation-records: 1591 1592Relocation Records 1593------------------ 1594 1595AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1596relocatable fields are: 1597 1598``word32`` 1599 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1600 alignment. These values use the same byte order as other word values in the 1601 AMDGPU architecture. 1602 1603``word64`` 1604 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1605 alignment. These values use the same byte order as other word values in the 1606 AMDGPU architecture. 1607 1608Following notations are used for specifying relocation calculations: 1609 1610**A** 1611 Represents the addend used to compute the value of the relocatable field. 1612 1613**G** 1614 Represents the offset into the global offset table at which the relocation 1615 entry's symbol will reside during execution. 1616 1617**GOT** 1618 Represents the address of the global offset table. 1619 1620**P** 1621 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1622 of the storage unit being relocated (computed using ``r_offset``). 1623 1624**S** 1625 Represents the value of the symbol whose index resides in the relocation 1626 entry. Relocations not using this must specify a symbol index of 1627 ``STN_UNDEF``. 1628 1629**B** 1630 Represents the base address of a loaded executable or shared object which is 1631 the difference between the ELF address and the actual load address. 1632 Relocations using this are only valid in executable or shared objects. 1633 1634The following relocation types are supported: 1635 1636 .. table:: AMDGPU ELF Relocation Records 1637 :name: amdgpu-elf-relocation-records-table 1638 1639 ========================== ======= ===== ========== ============================== 1640 Relocation Type Kind Value Field Calculation 1641 ========================== ======= ===== ========== ============================== 1642 ``R_AMDGPU_NONE`` 0 *none* *none* 1643 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1644 Dynamic 1645 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1646 Dynamic 1647 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1648 Dynamic 1649 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1650 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1651 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1652 Dynamic 1653 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1654 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1655 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1656 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1657 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1658 *reserved* 12 1659 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1660 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1661 ========================== ======= ===== ========== ============================== 1662 1663``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1664the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1665 1666There is no current OS loader support for 32-bit programs and so 1667``R_AMDGPU_ABS32`` is not used. 1668 1669.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1670 1671Loaded Code Object Path Uniform Resource Identifier (URI) 1672--------------------------------------------------------- 1673 1674The AMD GPU code object loader represents the path of the ELF shared object from 1675which the code object was loaded as a textual Uniform Resource Identifier (URI). 1676Note that the code object is the in memory loaded relocated form of the ELF 1677shared object. Multiple code objects may be loaded at different memory 1678addresses in the same process from the same ELF shared object. 1679 1680The loaded code object path URI syntax is defined by the following BNF syntax: 1681 1682.. code:: 1683 1684 code_object_uri ::== file_uri | memory_uri 1685 file_uri ::== "file://" file_path [ range_specifier ] 1686 memory_uri ::== "memory://" process_id range_specifier 1687 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1688 file_path ::== URI_ENCODED_OS_FILE_PATH 1689 process_id ::== DECIMAL_NUMBER 1690 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1691 1692**number** 1693 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1694 and octal values by "0". 1695 1696**file_path** 1697 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1698 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1699 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1700 the path are separated by "/". 1701 1702**offset** 1703 Is a 0-based byte offset to the start of the code object. For a file URI, it 1704 is from the start of the file specified by the ``file_path``, and if omitted 1705 defaults to 0. For a memory URI, it is the memory address and is required. 1706 1707**size** 1708 Is the number of bytes in the code object. For a file URI, if omitted it 1709 defaults to the size of the file. It is required for a memory URI. 1710 1711**process_id** 1712 Is the identity of the process owning the memory. For Linux it is the C 1713 unsigned integral decimal literal for the process ID (PID). 1714 1715For example: 1716 1717.. code:: 1718 1719 file:///dir1/dir2/file1 1720 file:///dir3/dir4/file2#offset=0x2000&size=3000 1721 memory://1234#offset=0x20000&size=3000 1722 1723.. _amdgpu-dwarf-debug-information: 1724 1725DWARF Debug Information 1726======================= 1727 1728.. warning:: 1729 1730 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1731 is not currently fully implemented and is subject to change. 1732 1733AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1734:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1735object executable code and data to the source language constructs. It can be 1736used by tools such as debuggers and profilers. It uses features defined in 1737:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1738DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1739 1740This section defines the AMDGPU target architecture specific DWARF mappings. 1741 1742.. _amdgpu-dwarf-register-identifier: 1743 1744Register Identifier 1745------------------- 1746 1747This section defines the AMDGPU target architecture register numbers used in 1748DWARF operation expressions (see DWARF Version 5 section 2.5 and 1749:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1750instructions (see DWARF Version 5 section 6.4 and 1751:ref:`amdgpu-dwarf-call-frame-information`). 1752 1753A single code object can contain code for kernels that have different wavefront 1754sizes. The vector registers and some scalar registers are based on the wavefront 1755size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1756simplifies the consumer of the DWARF so that each register has a fixed size, 1757rather than being dynamic according to the wavefront size mode. Similarly, 1758distinct DWARF registers are defined for those registers that vary in size 1759according to the process address size. This allows a consumer to treat a 1760specific AMDGPU processor as a single architecture regardless of how it is 1761configured at run time. The compiler explicitly specifies the DWARF registers 1762that match the mode in which the code it is generating will be executed. 1763 1764DWARF registers are encoded as numbers, which are mapped to architecture 1765registers. The mapping for AMDGPU is defined in 1766:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1767mapping. 1768 1769.. table:: AMDGPU DWARF Register Mapping 1770 :name: amdgpu-dwarf-register-mapping-table 1771 1772 ============== ================= ======== ================================== 1773 DWARF Register AMDGPU Register Bit Size Description 1774 ============== ================= ======== ================================== 1775 0 PC_32 32 Program Counter (PC) when 1776 executing in a 32-bit process 1777 address space. Used in the CFI to 1778 describe the PC of the calling 1779 frame. 1780 1 EXEC_MASK_32 32 Execution Mask Register when 1781 executing in wavefront 32 mode. 1782 2-15 *Reserved* *Reserved for highly accessed 1783 registers using DWARF shortcut.* 1784 16 PC_64 64 Program Counter (PC) when 1785 executing in a 64-bit process 1786 address space. Used in the CFI to 1787 describe the PC of the calling 1788 frame. 1789 17 EXEC_MASK_64 64 Execution Mask Register when 1790 executing in wavefront 64 mode. 1791 18-31 *Reserved* *Reserved for highly accessed 1792 registers using DWARF shortcut.* 1793 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1794 Registers. 1795 96-127 *Reserved* *Reserved for frequently accessed 1796 registers using DWARF 1-byte ULEB.* 1797 128 STATUS 32 Status Register. 1798 129-511 *Reserved* *Reserved for future Scalar 1799 Architectural Registers.* 1800 512 VCC_32 32 Vector Condition Code Register 1801 when executing in wavefront 32 1802 mode. 1803 513-767 *Reserved* *Reserved for future Vector 1804 Architectural Registers when 1805 executing in wavefront 32 mode.* 1806 768 VCC_64 64 Vector Condition Code Register 1807 when executing in wavefront 64 1808 mode. 1809 769-1023 *Reserved* *Reserved for future Vector 1810 Architectural Registers when 1811 executing in wavefront 64 mode.* 1812 1024-1087 *Reserved* *Reserved for padding.* 1813 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1814 1130-1535 *Reserved* *Reserved for future Scalar 1815 General Purpose Registers.* 1816 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1817 when executing in wavefront 32 1818 mode. 1819 1792-2047 *Reserved* *Reserved for future Vector 1820 General Purpose Registers when 1821 executing in wavefront 32 mode.* 1822 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1823 when executing in wavefront 32 1824 mode. 1825 2304-2559 *Reserved* *Reserved for future Vector 1826 Accumulation Registers when 1827 executing in wavefront 32 mode.* 1828 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1829 when executing in wavefront 64 1830 mode. 1831 2816-3071 *Reserved* *Reserved for future Vector 1832 General Purpose Registers when 1833 executing in wavefront 64 mode.* 1834 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1835 when executing in wavefront 64 1836 mode. 1837 3328-3583 *Reserved* *Reserved for future Vector 1838 Accumulation Registers when 1839 executing in wavefront 64 mode.* 1840 ============== ================= ======== ================================== 1841 1842The vector registers are represented as the full size for the wavefront. They 1843are organized as consecutive dwords (32-bits), one per lane, with the dword at 1844the least significant bit position corresponding to lane 0 and so forth. DWARF 1845location expressions involving the ``DW_OP_LLVM_offset`` and 1846``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1847register corresponding to the lane that is executing the current thread of 1848execution in languages that are implemented using a SIMD or SIMT execution 1849model. 1850 1851If the wavefront size is 32 lanes then the wavefront 32 mode register 1852definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1853mode register definitions are used. Some AMDGPU targets support executing in 1854both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1855to the wavefront mode of the generated code will be used. 1856 1857If code is generated to execute in a 32-bit process address space, then the 185832-bit process address space register definitions are used. If code is generated 1859to execute in a 64-bit process address space, then the 64-bit process address 1860space register definitions are used. The ``amdgcn`` target only supports the 186164-bit process address space. 1862 1863.. _amdgpu-dwarf-address-class-identifier: 1864 1865Address Class Identifier 1866------------------------ 1867 1868The DWARF address class represents the source language memory space. See DWARF 1869Version 5 section 2.12 which is updated by the *DWARF Extensions For 1870Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1871 1872The DWARF address class mapping used for AMDGPU is defined in 1873:ref:`amdgpu-dwarf-address-class-mapping-table`. 1874 1875.. table:: AMDGPU DWARF Address Class Mapping 1876 :name: amdgpu-dwarf-address-class-mapping-table 1877 1878 ========================= ====== ================= 1879 DWARF AMDGPU 1880 -------------------------------- ----------------- 1881 Address Class Name Value Address Space 1882 ========================= ====== ================= 1883 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1884 ``DW_ADDR_LLVM_global`` 0x0001 Global 1885 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1886 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1887 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1888 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1889 ========================= ====== ================= 1890 1891The DWARF address class values defined in the *DWARF Extensions For 1892Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1893 1894In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1895available for use for the AMD extension for access to the hardware GDS memory 1896which is scratchpad memory allocated per device. 1897 1898For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1899address class of ``DW_ADDR_none`` is used. 1900 1901See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1902mapping of DWARF address classes to DWARF address spaces, including address size 1903and NULL value. 1904 1905.. _amdgpu-dwarf-address-space-identifier: 1906 1907Address Space Identifier 1908------------------------ 1909 1910DWARF address spaces correspond to target architecture specific linear 1911addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1912For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1913 1914The DWARF address space mapping used for AMDGPU is defined in 1915:ref:`amdgpu-dwarf-address-space-mapping-table`. 1916 1917.. table:: AMDGPU DWARF Address Space Mapping 1918 :name: amdgpu-dwarf-address-space-mapping-table 1919 1920 ======================================= ===== ======= ======== ================= ======================= 1921 DWARF AMDGPU Notes 1922 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1923 Address Space Name Value Address Bit Size Address Space 1924 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1925 .. 64-bit 32-bit 1926 process process 1927 address address 1928 space space 1929 ======================================= ===== ======= ======== ================= ======================= 1930 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1931 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1932 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1933 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1934 *Reserved* 0x04 1935 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1936 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1937 ======================================= ===== ======= ======== ================= ======================= 1938 1939See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1940including address size and NULL value. 1941 1942The ``DW_ASPACE_none`` address space is the default target architecture address 1943space used in DWARF operations that do not specify an address space. It 1944therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1945related operations can refer to addresses in the program code. 1946 1947The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1948specify the flat address space. If the address corresponds to an address in the 1949local address space, then it corresponds to the wavefront that is executing the 1950focused thread of execution. If the address corresponds to an address in the 1951private address space, then it corresponds to the lane that is executing the 1952focused thread of execution for languages that are implemented using a SIMD or 1953SIMT execution model. 1954 1955.. note:: 1956 1957 CUDA-like languages such as HIP that do not have address spaces in the 1958 language type system, but do allow variables to be allocated in different 1959 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1960 address space in the DWARF expression operations as the default address space 1961 is the global address space. 1962 1963The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1964specify the local address space corresponding to the wavefront that is executing 1965the focused thread of execution. 1966 1967The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1968to specify the private address space corresponding to the lane that is executing 1969the focused thread of execution for languages that are implemented using a SIMD 1970or SIMT execution model. 1971 1972The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1973to specify the unswizzled private address space corresponding to the wavefront 1974that is executing the focused thread of execution. The wavefront view of private 1975memory is the per wavefront unswizzled backing memory layout defined in 1976:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1977location for the backing memory of the wavefront (namely the address is not 1978offset by ``wavefront-scratch-base``). The following formula can be used to 1979convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1980``DW_ASPACE_AMDGPU_private_wave`` address: 1981 1982:: 1983 1984 private-address-wavefront = 1985 ((private-address-lane / 4) * wavefront-size * 4) + 1986 (wavefront-lane-id * 4) + (private-address-lane % 4) 1987 1988If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1989of the dwords for each lane starting with lane 0 is required, then this 1990simplifies to: 1991 1992:: 1993 1994 private-address-wavefront = 1995 private-address-lane * wavefront-size 1996 1997A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1998complete spilled vector register back into a complete vector register in the 1999CFI. The frame pointer can be a private lane address which is dword aligned, 2000which can be shifted to multiply by the wavefront size, and then used to form a 2001private wavefront address that gives a location for a contiguous set of dwords, 2002one per lane, where the vector register dwords are spilled. The compiler knows 2003the wavefront size since it generates the code. Note that the type of the 2004address may have to be converted as the size of a 2005``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 2006``DW_ASPACE_AMDGPU_private_wave`` address. 2007 2008.. _amdgpu-dwarf-lane-identifier: 2009 2010Lane identifier 2011--------------- 2012 2013DWARF lane identifies specify a target architecture lane position for hardware 2014that executes in a SIMD or SIMT manner, and on which a source language maps its 2015threads of execution onto those lanes. The DWARF lane identifier is pushed by 2016the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 2017section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 2018section :ref:`amdgpu-dwarf-operation-expressions`. 2019 2020For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 2021wavefront. It is numbered from 0 to the wavefront size minus 1. 2022 2023Operation Expressions 2024--------------------- 2025 2026DWARF expressions are used to compute program values and the locations of 2027program objects. See DWARF Version 5 section 2.5 and 2028:ref:`amdgpu-dwarf-operation-expressions`. 2029 2030DWARF location descriptions describe how to access storage which includes memory 2031and registers. When accessing storage on AMDGPU, bytes are ordered with least 2032significant bytes first, and bits are ordered within bytes with least 2033significant bits first. 2034 2035For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 2036unwinding vector registers that are spilled under the execution mask to memory: 2037the zero-single location description is the vector register, and the one-single 2038location description is the spilled memory location description. The 2039``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 2040memory location description. 2041 2042In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2043``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2044controlled by the execution mask. An undefined location description together 2045with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2046to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2047 2048Debugger Information Entry Attributes 2049------------------------------------- 2050 2051This section describes how certain debugger information entry attributes are 2052used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.1 2053which are updated by *DWARF Extensions For Heterogeneous Debugging* section 2054:ref:`amdgpu-dwarf-low-level-information` and 2055:ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`. 2056 2057.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2058 2059``DW_AT_LLVM_lane_pc`` 2060~~~~~~~~~~~~~~~~~~~~~~ 2061 2062For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2063location of the separate lanes of a SIMT thread. 2064 2065If the lane is an active lane then this will be the same as the current program 2066location. 2067 2068If the lane is inactive, but was active on entry to the subprogram, then this is 2069the program location in the subprogram at which execution of the lane is 2070conceptual positioned. 2071 2072If the lane was not active on entry to the subprogram, then this will be the 2073undefined location. A client debugger can check if the lane is part of a valid 2074work-group by checking that the lane is in the range of the associated 2075work-group within the grid, accounting for partial work-groups. If it is not, 2076then the debugger can omit any information for the lane. Otherwise, the debugger 2077may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2078calling subprogram until it finds a non-undefined location. Conceptually the 2079lane only has the call frames that it has a non-undefined 2080``DW_AT_LLVM_lane_pc``. 2081 2082The following example illustrates how the AMDGPU backend can generate a DWARF 2083location list expression for the nested ``IF/THEN/ELSE`` structures of the 2084following subprogram pseudo code for a target with 64 lanes per wavefront. 2085 2086.. code:: 2087 :number-lines: 2088 2089 SUBPROGRAM X 2090 BEGIN 2091 a; 2092 IF (c1) THEN 2093 b; 2094 IF (c2) THEN 2095 c; 2096 ELSE 2097 d; 2098 ENDIF 2099 e; 2100 ELSE 2101 f; 2102 ENDIF 2103 g; 2104 END 2105 2106The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2107execution mask (``EXEC``) to linearize the control flow. The condition is 2108evaluated to make a mask of the lanes for which the condition evaluates to true. 2109First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2110logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2111``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2112the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2113region the ``EXEC`` mask is restored to the value it had at the beginning of the 2114region. This is shown below. Other approaches are possible, but the basic 2115concept is the same. 2116 2117.. code:: 2118 :number-lines: 2119 2120 $lex_start: 2121 a; 2122 %1 = EXEC 2123 %2 = c1 2124 $lex_1_start: 2125 EXEC = %1 & %2 2126 $if_1_then: 2127 b; 2128 %3 = EXEC 2129 %4 = c2 2130 $lex_1_1_start: 2131 EXEC = %3 & %4 2132 $lex_1_1_then: 2133 c; 2134 EXEC = ~EXEC & %3 2135 $lex_1_1_else: 2136 d; 2137 EXEC = %3 2138 $lex_1_1_end: 2139 e; 2140 EXEC = ~EXEC & %1 2141 $lex_1_else: 2142 f; 2143 EXEC = %1 2144 $lex_1_end: 2145 g; 2146 $lex_end: 2147 2148To create the DWARF location list expression that defines the location 2149description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2150pseudo instruction can be used to annotate the linearized control flow. This can 2151be done by defining an artificial variable for the lane PC. The DWARF location 2152list expression created for it is used as the value of the 2153``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2154 2155A DWARF procedure is defined for each well nested structured control flow region 2156which provides the conceptual lane program location for a lane if it is not 2157active (namely it is divergent). The DWARF operation expression for each region 2158conceptually inherits the value of the immediately enclosing region and modifies 2159it according to the semantics of the region. 2160 2161For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2162the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2163region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2164region since the ``THEN`` region has completed. 2165 2166The lane PC artificial variable is assigned at each region transition. It uses 2167the immediately enclosing region's DWARF procedure to compute the program 2168location for each lane assuming they are divergent, and then modifies the result 2169by inserting the current program location for each lane that the ``EXEC`` mask 2170indicates is active. 2171 2172By having separate DWARF procedures for each region, they can be reused to 2173define the value for any nested region. This reduces the total size of the DWARF 2174operation expressions. 2175 2176The following provides an example using pseudo LLVM MIR. 2177 2178.. code:: 2179 :number-lines: 2180 2181 $lex_start: 2182 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2183 DW_AT_name = "__uint64"; 2184 DW_AT_byte_size = 8; 2185 DW_AT_encoding = DW_ATE_unsigned; 2186 ]; 2187 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2188 DW_AT_name = "__active_lane_pc"; 2189 DW_AT_location = [ 2190 DW_OP_regx PC; 2191 DW_OP_LLVM_extend 64, 64; 2192 DW_OP_regval_type EXEC, %uint_64; 2193 DW_OP_LLVM_select_bit_piece 64, 64; 2194 ]; 2195 ]; 2196 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2197 DW_AT_name = "__divergent_lane_pc"; 2198 DW_AT_location = [ 2199 DW_OP_LLVM_undefined; 2200 DW_OP_LLVM_extend 64, 64; 2201 ]; 2202 ]; 2203 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2204 DW_OP_call_ref %__divergent_lane_pc; 2205 DW_OP_call_ref %__active_lane_pc; 2206 ]; 2207 a; 2208 %1 = EXEC; 2209 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2210 %2 = c1; 2211 $lex_1_start: 2212 EXEC = %1 & %2; 2213 $lex_1_then: 2214 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2215 DW_AT_name = "__divergent_lane_pc_1_then"; 2216 DW_AT_location = DIExpression[ 2217 DW_OP_call_ref %__divergent_lane_pc; 2218 DW_OP_addrx &lex_1_start; 2219 DW_OP_stack_value; 2220 DW_OP_LLVM_extend 64, 64; 2221 DW_OP_call_ref %__lex_1_save_exec; 2222 DW_OP_deref_type 64, %__uint_64; 2223 DW_OP_LLVM_select_bit_piece 64, 64; 2224 ]; 2225 ]; 2226 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2227 DW_OP_call_ref %__divergent_lane_pc_1_then; 2228 DW_OP_call_ref %__active_lane_pc; 2229 ]; 2230 b; 2231 %3 = EXEC; 2232 DBG_VALUE %3, %__lex_1_1_save_exec; 2233 %4 = c2; 2234 $lex_1_1_start: 2235 EXEC = %3 & %4; 2236 $lex_1_1_then: 2237 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2238 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2239 DW_AT_location = DIExpression[ 2240 DW_OP_call_ref %__divergent_lane_pc_1_then; 2241 DW_OP_addrx &lex_1_1_start; 2242 DW_OP_stack_value; 2243 DW_OP_LLVM_extend 64, 64; 2244 DW_OP_call_ref %__lex_1_1_save_exec; 2245 DW_OP_deref_type 64, %__uint_64; 2246 DW_OP_LLVM_select_bit_piece 64, 64; 2247 ]; 2248 ]; 2249 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2250 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2251 DW_OP_call_ref %__active_lane_pc; 2252 ]; 2253 c; 2254 EXEC = ~EXEC & %3; 2255 $lex_1_1_else: 2256 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2257 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2258 DW_AT_location = DIExpression[ 2259 DW_OP_call_ref %__divergent_lane_pc_1_then; 2260 DW_OP_addrx &lex_1_1_end; 2261 DW_OP_stack_value; 2262 DW_OP_LLVM_extend 64, 64; 2263 DW_OP_call_ref %__lex_1_1_save_exec; 2264 DW_OP_deref_type 64, %__uint_64; 2265 DW_OP_LLVM_select_bit_piece 64, 64; 2266 ]; 2267 ]; 2268 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2269 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2270 DW_OP_call_ref %__active_lane_pc; 2271 ]; 2272 d; 2273 EXEC = %3; 2274 $lex_1_1_end: 2275 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2276 DW_OP_call_ref %__divergent_lane_pc; 2277 DW_OP_call_ref %__active_lane_pc; 2278 ]; 2279 e; 2280 EXEC = ~EXEC & %1; 2281 $lex_1_else: 2282 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2283 DW_AT_name = "__divergent_lane_pc_1_else"; 2284 DW_AT_location = DIExpression[ 2285 DW_OP_call_ref %__divergent_lane_pc; 2286 DW_OP_addrx &lex_1_end; 2287 DW_OP_stack_value; 2288 DW_OP_LLVM_extend 64, 64; 2289 DW_OP_call_ref %__lex_1_save_exec; 2290 DW_OP_deref_type 64, %__uint_64; 2291 DW_OP_LLVM_select_bit_piece 64, 64; 2292 ]; 2293 ]; 2294 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2295 DW_OP_call_ref %__divergent_lane_pc_1_else; 2296 DW_OP_call_ref %__active_lane_pc; 2297 ]; 2298 f; 2299 EXEC = %1; 2300 $lex_1_end: 2301 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2302 DW_OP_call_ref %__divergent_lane_pc; 2303 DW_OP_call_ref %__active_lane_pc; 2304 ]; 2305 g; 2306 $lex_end: 2307 2308The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2309that are active, with the current program location. 2310 2311Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2312the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2313instruction, location list entries will be created that describe where the 2314artificial variables are allocated at any given program location. The compiler 2315may allocate them to registers or spill them to memory. 2316 2317The DWARF procedures for each region use the values of the saved execution mask 2318artificial variables to only update the lanes that are active on entry to the 2319region. All other lanes retain the value of the enclosing region where they were 2320last active. If they were not active on entry to the subprogram, then will have 2321the undefined location description. 2322 2323Other structured control flow regions can be handled similarly. For example, 2324loops would set the divergent program location for the region at the end of the 2325loop. Any lanes active will be in the loop, and any lanes not active must have 2326exited the loop. 2327 2328An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2329``IF/THEN/ELSE`` regions. 2330 2331The DWARF procedures can use the active lane artificial variable described in 2332:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2333``EXEC`` mask in order to support whole or quad wavefront mode. 2334 2335.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2336 2337``DW_AT_LLVM_active_lane`` 2338~~~~~~~~~~~~~~~~~~~~~~~~~~ 2339 2340The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2341entry is used to specify the lanes that are conceptually active for a SIMT 2342thread. 2343 2344The execution mask may be modified to implement whole or quad wavefront mode 2345operations. For example, all lanes may need to temporarily be made active to 2346execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2347update it to enable the necessary lanes, perform the operations, and then 2348restore the ``EXEC`` mask from the saved value. While executing the whole 2349wavefront region, the conceptual execution mask is the saved value, not the 2350``EXEC`` value. 2351 2352This is handled by defining an artificial variable for the active lane mask. The 2353active lane mask artificial variable would be the actual ``EXEC`` mask for 2354normal regions, and the saved execution mask for regions where the mask is 2355temporarily updated. The location list expression created for this artificial 2356variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2357attribute. 2358 2359``DW_AT_LLVM_augmentation`` 2360~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2361 2362For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2363debugger information entry has the following value for the augmentation string: 2364 2365:: 2366 2367 [amdgpu:v0.0] 2368 2369The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2370extensions used in the DWARF of the compilation unit. The version number 2371conforms to [SEMVER]_. 2372 2373Call Frame Information 2374---------------------- 2375 2376DWARF Call Frame Information (CFI) describes how a consumer can virtually 2377*unwind* call frames in a running process or core dump. See DWARF Version 5 2378section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2379 2380For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2381 23821. ``augmentation`` string contains the following null-terminated UTF-8 string: 2383 2384 :: 2385 2386 [amd:v0.0] 2387 2388 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2389 extensions used in this CIE or to the FDEs that use it. The version number 2390 conforms to [SEMVER]_. 2391 23922. ``address_size`` for the ``Global`` address space is defined in 2393 :ref:`amdgpu-dwarf-address-space-identifier`. 2394 23953. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2396 23974. ``code_alignment_factor`` is 4 bytes. 2398 2399 .. TODO:: 2400 2401 Add to :ref:`amdgpu-processor-table` table. 2402 24035. ``data_alignment_factor`` is 4 bytes. 2404 2405 .. TODO:: 2406 2407 Add to :ref:`amdgpu-processor-table` table. 2408 24096. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2410 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2411 24127. ``initial_instructions`` Since a subprogram X with fewer registers can be 2413 called from subprogram Y that has more allocated, X will not change any of 2414 the extra registers as it cannot access them. Therefore, the default rule 2415 for all columns is ``same value``. 2416 2417For AMDGPU the register number follows the numbering defined in 2418:ref:`amdgpu-dwarf-register-identifier`. 2419 2420For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2421the return address to get the address of a byte within the call site 2422instructions. See DWARF Version 5 section 6.4.4. 2423 2424Accelerated Access 2425------------------ 2426 2427See DWARF Version 5 section 6.1. 2428 2429Lookup By Name Section Header 2430~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2431 2432See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2433 2434For AMDGPU the lookup by name section header table: 2435 2436``augmentation_string_size`` (uword) 2437 2438 Set to the length of the ``augmentation_string`` value which is always a 2439 multiple of 4. 2440 2441``augmentation_string`` (sequence of UTF-8 characters) 2442 2443 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2444 2445 :: 2446 2447 [amdgpu:v0.0] 2448 2449 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2450 extensions used in the DWARF of this index. The version number conforms to 2451 [SEMVER]_. 2452 2453 .. note:: 2454 2455 This is different to the DWARF Version 5 definition that requires the first 2456 4 characters to be the vendor ID. But this is consistent with the other 2457 augmentation strings and does allow multiple vendor contributions. However, 2458 backwards compatibility may be more desirable. 2459 2460Lookup By Address Section Header 2461~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2462 2463See DWARF Version 5 section 6.1.2. 2464 2465For AMDGPU the lookup by address section header table: 2466 2467``address_size`` (ubyte) 2468 2469 Match the address size for the ``Global`` address space defined in 2470 :ref:`amdgpu-dwarf-address-space-identifier`. 2471 2472``segment_selector_size`` (ubyte) 2473 2474 AMDGPU does not use a segment selector so this is 0. The entries in the 2475 ``.debug_aranges`` do not have a segment selector. 2476 2477Line Number Information 2478----------------------- 2479 2480See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2481 2482AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2483The instruction set must be obtained from the ELF file header ``e_flags`` field 2484in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2485<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2486 2487.. TODO:: 2488 2489 Should the ``isa`` state machine register be used to indicate if the code is 2490 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2491 2492For AMDGPU the line number program header fields have the following values (see 2493DWARF Version 5 section 6.2.4): 2494 2495``address_size`` (ubyte) 2496 Matches the address size for the ``Global`` address space defined in 2497 :ref:`amdgpu-dwarf-address-space-identifier`. 2498 2499``segment_selector_size`` (ubyte) 2500 AMDGPU does not use a segment selector so this is 0. 2501 2502``minimum_instruction_length`` (ubyte) 2503 For GFX9-GFX10 this is 4. 2504 2505``maximum_operations_per_instruction`` (ubyte) 2506 For GFX9-GFX10 this is 1. 2507 2508Source text for online-compiled programs (for example, those compiled by the 2509OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2510See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2511Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2512<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2513 2514The Clang option used to control source embedding in AMDGPU is defined in 2515:ref:`amdgpu-clang-debug-options-table`. 2516 2517 .. table:: AMDGPU Clang Debug Options 2518 :name: amdgpu-clang-debug-options-table 2519 2520 ==================== ================================================== 2521 Debug Flag Description 2522 ==================== ================================================== 2523 -g[no-]embed-source Enable/disable embedding source text in DWARF 2524 debug sections. Useful for environments where 2525 source cannot be written to disk, such as 2526 when performing online compilation. 2527 ==================== ================================================== 2528 2529For example: 2530 2531``-gembed-source`` 2532 Enable the embedded source. 2533 2534``-gno-embed-source`` 2535 Disable the embedded source. 2536 253732-Bit and 64-Bit DWARF Formats 2538------------------------------- 2539 2540See DWARF Version 5 section 7.4 and 2541:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2542 2543For AMDGPU: 2544 2545* For the ``amdgcn`` target architecture only the 64-bit process address space 2546 is supported. 2547 2548* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2549 the 32-bit DWARF format. 2550 2551Unit Headers 2552------------ 2553 2554For AMDGPU the following values apply for each of the unit headers described in 2555DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2556 2557``address_size`` (ubyte) 2558 Matches the address size for the ``Global`` address space defined in 2559 :ref:`amdgpu-dwarf-address-space-identifier`. 2560 2561.. _amdgpu-code-conventions: 2562 2563Code Conventions 2564================ 2565 2566This section provides code conventions used for each supported target triple OS 2567(see :ref:`amdgpu-target-triples`). 2568 2569AMDHSA 2570------ 2571 2572This section provides code conventions used when the target triple OS is 2573``amdhsa`` (see :ref:`amdgpu-target-triples`). 2574 2575.. _amdgpu-amdhsa-code-object-metadata: 2576 2577Code Object Metadata 2578~~~~~~~~~~~~~~~~~~~~ 2579 2580The code object metadata specifies extensible metadata associated with the code 2581objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2582encoding and semantics of this metadata depends on the code object version; see 2583:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2584:ref:`amdgpu-amdhsa-code-object-metadata-v3`, 2585:ref:`amdgpu-amdhsa-code-object-metadata-v4` and 2586:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 2587 2588Code object metadata is specified in a note record (see 2589:ref:`amdgpu-note-records`) and is required when the target triple OS is 2590``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2591information necessary to support the HSA compatible runtime kernel queries. For 2592example, the segment sizes needed in a dispatch packet. In addition, a 2593high-level language runtime may require other information to be included. For 2594example, the AMD OpenCL runtime records kernel argument information. 2595 2596.. _amdgpu-amdhsa-code-object-metadata-v2: 2597 2598Code Object V2 Metadata 2599+++++++++++++++++++++++ 2600 2601.. warning:: 2602 Code object V2 is not the default code object version emitted by this version 2603 of LLVM. 2604 2605Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2606(see :ref:`amdgpu-note-records-v2`). 2607 2608The metadata is specified as a YAML formatted string (see [YAML]_ and 2609:doc:`YamlIO`). 2610 2611.. TODO:: 2612 2613 Is the string null terminated? It probably should not if YAML allows it to 2614 contain null characters, otherwise it should be. 2615 2616The metadata is represented as a single YAML document comprised of the mapping 2617defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2618referenced tables. 2619 2620For boolean values, the string values of ``false`` and ``true`` are used for 2621false and true respectively. 2622 2623Additional information can be added to the mappings. To avoid conflicts, any 2624non-AMD key names should be prefixed by "*vendor-name*.". 2625 2626 .. table:: AMDHSA Code Object V2 Metadata Map 2627 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2628 2629 ========== ============== ========= ======================================= 2630 String Key Value Type Required? Description 2631 ========== ============== ========= ======================================= 2632 "Version" sequence of Required - The first integer is the major 2633 2 integers version. Currently 1. 2634 - The second integer is the minor 2635 version. Currently 0. 2636 "Printf" sequence of Each string is encoded information 2637 strings about a printf function call. The 2638 encoded information is organized as 2639 fields separated by colon (':'): 2640 2641 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2642 2643 where: 2644 2645 ``ID`` 2646 A 32-bit integer as a unique id for 2647 each printf function call 2648 2649 ``N`` 2650 A 32-bit integer equal to the number 2651 of arguments of printf function call 2652 minus 1 2653 2654 ``S[i]`` (where i = 0, 1, ... , N-1) 2655 32-bit integers for the size in bytes 2656 of the i-th FormatString argument of 2657 the printf function call 2658 2659 FormatString 2660 The format string passed to the 2661 printf function call. 2662 "Kernels" sequence of Required Sequence of the mappings for each 2663 mapping kernel in the code object. See 2664 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2665 for the definition of the mapping. 2666 ========== ============== ========= ======================================= 2667 2668.. 2669 2670 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2671 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2672 2673 ================= ============== ========= ================================ 2674 String Key Value Type Required? Description 2675 ================= ============== ========= ================================ 2676 "Name" string Required Source name of the kernel. 2677 "SymbolName" string Required Name of the kernel 2678 descriptor ELF symbol. 2679 "Language" string Source language of the kernel. 2680 Values include: 2681 2682 - "OpenCL C" 2683 - "OpenCL C++" 2684 - "HCC" 2685 - "OpenMP" 2686 2687 "LanguageVersion" sequence of - The first integer is the major 2688 2 integers version. 2689 - The second integer is the 2690 minor version. 2691 "Attrs" mapping Mapping of kernel attributes. 2692 See 2693 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2694 for the mapping definition. 2695 "Args" sequence of Sequence of mappings of the 2696 mapping kernel arguments. See 2697 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2698 for the definition of the mapping. 2699 "CodeProps" mapping Mapping of properties related to 2700 the kernel code. See 2701 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2702 for the mapping definition. 2703 ================= ============== ========= ================================ 2704 2705.. 2706 2707 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2708 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2709 2710 =================== ============== ========= ============================== 2711 String Key Value Type Required? Description 2712 =================== ============== ========= ============================== 2713 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2714 3 integers must be >=1 and the dispatch 2715 work-group size X, Y, Z must 2716 correspond to the specified 2717 values. Defaults to 0, 0, 0. 2718 2719 Corresponds to the OpenCL 2720 ``reqd_work_group_size`` 2721 attribute. 2722 "WorkGroupSizeHint" sequence of The dispatch work-group size 2723 3 integers X, Y, Z is likely to be the 2724 specified values. 2725 2726 Corresponds to the OpenCL 2727 ``work_group_size_hint`` 2728 attribute. 2729 "VecTypeHint" string The name of a scalar or vector 2730 type. 2731 2732 Corresponds to the OpenCL 2733 ``vec_type_hint`` attribute. 2734 2735 "RuntimeHandle" string The external symbol name 2736 associated with a kernel. 2737 OpenCL runtime allocates a 2738 global buffer for the symbol 2739 and saves the kernel's address 2740 to it, which is used for 2741 device side enqueueing. Only 2742 available for device side 2743 enqueued kernels. 2744 =================== ============== ========= ============================== 2745 2746.. 2747 2748 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2749 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2750 2751 ================= ============== ========= ================================ 2752 String Key Value Type Required? Description 2753 ================= ============== ========= ================================ 2754 "Name" string Kernel argument name. 2755 "TypeName" string Kernel argument type name. 2756 "Size" integer Required Kernel argument size in bytes. 2757 "Align" integer Required Kernel argument alignment in 2758 bytes. Must be a power of two. 2759 "ValueKind" string Required Kernel argument kind that 2760 specifies how to set up the 2761 corresponding argument. 2762 Values include: 2763 2764 "ByValue" 2765 The argument is copied 2766 directly into the kernarg. 2767 2768 "GlobalBuffer" 2769 A global address space pointer 2770 to the buffer data is passed 2771 in the kernarg. 2772 2773 "DynamicSharedPointer" 2774 A group address space pointer 2775 to dynamically allocated LDS 2776 is passed in the kernarg. 2777 2778 "Sampler" 2779 A global address space 2780 pointer to a S# is passed in 2781 the kernarg. 2782 2783 "Image" 2784 A global address space 2785 pointer to a T# is passed in 2786 the kernarg. 2787 2788 "Pipe" 2789 A global address space pointer 2790 to an OpenCL pipe is passed in 2791 the kernarg. 2792 2793 "Queue" 2794 A global address space pointer 2795 to an OpenCL device enqueue 2796 queue is passed in the 2797 kernarg. 2798 2799 "HiddenGlobalOffsetX" 2800 The OpenCL grid dispatch 2801 global offset for the X 2802 dimension is passed in the 2803 kernarg. 2804 2805 "HiddenGlobalOffsetY" 2806 The OpenCL grid dispatch 2807 global offset for the Y 2808 dimension is passed in the 2809 kernarg. 2810 2811 "HiddenGlobalOffsetZ" 2812 The OpenCL grid dispatch 2813 global offset for the Z 2814 dimension is passed in the 2815 kernarg. 2816 2817 "HiddenNone" 2818 An argument that is not used 2819 by the kernel. Space needs to 2820 be left for it, but it does 2821 not need to be set up. 2822 2823 "HiddenPrintfBuffer" 2824 A global address space pointer 2825 to the runtime printf buffer 2826 is passed in kernarg. Mutually 2827 exclusive with 2828 "HiddenHostcallBuffer". 2829 2830 "HiddenHostcallBuffer" 2831 A global address space pointer 2832 to the runtime hostcall buffer 2833 is passed in kernarg. Mutually 2834 exclusive with 2835 "HiddenPrintfBuffer". 2836 2837 "HiddenDefaultQueue" 2838 A global address space pointer 2839 to the OpenCL device enqueue 2840 queue that should be used by 2841 the kernel by default is 2842 passed in the kernarg. 2843 2844 "HiddenCompletionAction" 2845 A global address space pointer 2846 to help link enqueued kernels into 2847 the ancestor tree for determining 2848 when the parent kernel has finished. 2849 2850 "HiddenMultiGridSyncArg" 2851 A global address space pointer for 2852 multi-grid synchronization is 2853 passed in the kernarg. 2854 2855 "ValueType" string Unused and deprecated. This should no longer 2856 be emitted, but is accepted for compatibility. 2857 2858 2859 "PointeeAlign" integer Alignment in bytes of pointee 2860 type for pointer type kernel 2861 argument. Must be a power 2862 of 2. Only present if 2863 "ValueKind" is 2864 "DynamicSharedPointer". 2865 "AddrSpaceQual" string Kernel argument address space 2866 qualifier. Only present if 2867 "ValueKind" is "GlobalBuffer" or 2868 "DynamicSharedPointer". Values 2869 are: 2870 2871 - "Private" 2872 - "Global" 2873 - "Constant" 2874 - "Local" 2875 - "Generic" 2876 - "Region" 2877 2878 .. TODO:: 2879 2880 Is GlobalBuffer only Global 2881 or Constant? Is 2882 DynamicSharedPointer always 2883 Local? Can HCC allow Generic? 2884 How can Private or Region 2885 ever happen? 2886 2887 "AccQual" string Kernel argument access 2888 qualifier. Only present if 2889 "ValueKind" is "Image" or 2890 "Pipe". Values 2891 are: 2892 2893 - "ReadOnly" 2894 - "WriteOnly" 2895 - "ReadWrite" 2896 2897 .. TODO:: 2898 2899 Does this apply to 2900 GlobalBuffer? 2901 2902 "ActualAccQual" string The actual memory accesses 2903 performed by the kernel on the 2904 kernel argument. Only present if 2905 "ValueKind" is "GlobalBuffer", 2906 "Image", or "Pipe". This may be 2907 more restrictive than indicated 2908 by "AccQual" to reflect what the 2909 kernel actual does. If not 2910 present then the runtime must 2911 assume what is implied by 2912 "AccQual" and "IsConst". Values 2913 are: 2914 2915 - "ReadOnly" 2916 - "WriteOnly" 2917 - "ReadWrite" 2918 2919 "IsConst" boolean Indicates if the kernel argument 2920 is const qualified. Only present 2921 if "ValueKind" is 2922 "GlobalBuffer". 2923 2924 "IsRestrict" boolean Indicates if the kernel argument 2925 is restrict qualified. Only 2926 present if "ValueKind" is 2927 "GlobalBuffer". 2928 2929 "IsVolatile" boolean Indicates if the kernel argument 2930 is volatile qualified. Only 2931 present if "ValueKind" is 2932 "GlobalBuffer". 2933 2934 "IsPipe" boolean Indicates if the kernel argument 2935 is pipe qualified. Only present 2936 if "ValueKind" is "Pipe". 2937 2938 .. TODO:: 2939 2940 Can GlobalBuffer be pipe 2941 qualified? 2942 2943 ================= ============== ========= ================================ 2944 2945.. 2946 2947 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2948 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2949 2950 ============================ ============== ========= ===================== 2951 String Key Value Type Required? Description 2952 ============================ ============== ========= ===================== 2953 "KernargSegmentSize" integer Required The size in bytes of 2954 the kernarg segment 2955 that holds the values 2956 of the arguments to 2957 the kernel. 2958 "GroupSegmentFixedSize" integer Required The amount of group 2959 segment memory 2960 required by a 2961 work-group in 2962 bytes. This does not 2963 include any 2964 dynamically allocated 2965 group segment memory 2966 that may be added 2967 when the kernel is 2968 dispatched. 2969 "PrivateSegmentFixedSize" integer Required The amount of fixed 2970 private address space 2971 memory required for a 2972 work-item in 2973 bytes. If the kernel 2974 uses a dynamic call 2975 stack then additional 2976 space must be added 2977 to this value for the 2978 call stack. 2979 "KernargSegmentAlign" integer Required The maximum byte 2980 alignment of 2981 arguments in the 2982 kernarg segment. Must 2983 be a power of 2. 2984 "WavefrontSize" integer Required Wavefront size. Must 2985 be a power of 2. 2986 "NumSGPRs" integer Required Number of scalar 2987 registers used by a 2988 wavefront for 2989 GFX6-GFX10. This 2990 includes the special 2991 SGPRs for VCC, Flat 2992 Scratch (GFX7-GFX10) 2993 and XNACK (for 2994 GFX8-GFX10). It does 2995 not include the 16 2996 SGPR added if a trap 2997 handler is 2998 enabled. It is not 2999 rounded up to the 3000 allocation 3001 granularity. 3002 "NumVGPRs" integer Required Number of vector 3003 registers used by 3004 each work-item for 3005 GFX6-GFX10 3006 "MaxFlatWorkGroupSize" integer Required Maximum flat 3007 work-group size 3008 supported by the 3009 kernel in work-items. 3010 Must be >=1 and 3011 consistent with 3012 ReqdWorkGroupSize if 3013 not 0, 0, 0. 3014 "NumSpilledSGPRs" integer Number of stores from 3015 a scalar register to 3016 a register allocator 3017 created spill 3018 location. 3019 "NumSpilledVGPRs" integer Number of stores from 3020 a vector register to 3021 a register allocator 3022 created spill 3023 location. 3024 ============================ ============== ========= ===================== 3025 3026.. _amdgpu-amdhsa-code-object-metadata-v3: 3027 3028Code Object V3 Metadata 3029+++++++++++++++++++++++ 3030 3031.. warning:: 3032 Code object V3 is not the default code object version emitted by this version 3033 of LLVM. 3034 3035Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note 3036record (see :ref:`amdgpu-note-records-v3-onwards`). 3037 3038The metadata is represented as Message Pack formatted binary data (see 3039[MsgPack]_). The top level is a Message Pack map that includes the 3040keys defined in table 3041:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 3042tables. 3043 3044Additional information can be added to the maps. To avoid conflicts, 3045any key names should be prefixed by "*vendor-name*." where 3046``vendor-name`` can be the name of the vendor and specific vendor 3047tool that generates the information. The prefix is abbreviated to 3048simply "." when it appears within a map that has been added by the 3049same *vendor-name*. 3050 3051 .. table:: AMDHSA Code Object V3 Metadata Map 3052 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3053 3054 ================= ============== ========= ======================================= 3055 String Key Value Type Required? Description 3056 ================= ============== ========= ======================================= 3057 "amdhsa.version" sequence of Required - The first integer is the major 3058 2 integers version. Currently 1. 3059 - The second integer is the minor 3060 version. Currently 0. 3061 "amdhsa.printf" sequence of Each string is encoded information 3062 strings about a printf function call. The 3063 encoded information is organized as 3064 fields separated by colon (':'): 3065 3066 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3067 3068 where: 3069 3070 ``ID`` 3071 A 32-bit integer as a unique id for 3072 each printf function call 3073 3074 ``N`` 3075 A 32-bit integer equal to the number 3076 of arguments of printf function call 3077 minus 1 3078 3079 ``S[i]`` (where i = 0, 1, ... , N-1) 3080 32-bit integers for the size in bytes 3081 of the i-th FormatString argument of 3082 the printf function call 3083 3084 FormatString 3085 The format string passed to the 3086 printf function call. 3087 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3088 map kernel in the code object. See 3089 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3090 for the definition of the keys included 3091 in that map. 3092 ================= ============== ========= ======================================= 3093 3094.. 3095 3096 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3097 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3098 3099 =================================== ============== ========= ================================ 3100 String Key Value Type Required? Description 3101 =================================== ============== ========= ================================ 3102 ".name" string Required Source name of the kernel. 3103 ".symbol" string Required Name of the kernel 3104 descriptor ELF symbol. 3105 ".language" string Source language of the kernel. 3106 Values include: 3107 3108 - "OpenCL C" 3109 - "OpenCL C++" 3110 - "HCC" 3111 - "HIP" 3112 - "OpenMP" 3113 - "Assembler" 3114 3115 ".language_version" sequence of - The first integer is the major 3116 2 integers version. 3117 - The second integer is the 3118 minor version. 3119 ".args" sequence of Sequence of maps of the 3120 map kernel arguments. See 3121 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3122 for the definition of the keys 3123 included in that map. 3124 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3125 3 integers must be >=1 and the dispatch 3126 work-group size X, Y, Z must 3127 correspond to the specified 3128 values. Defaults to 0, 0, 0. 3129 3130 Corresponds to the OpenCL 3131 ``reqd_work_group_size`` 3132 attribute. 3133 ".workgroup_size_hint" sequence of The dispatch work-group size 3134 3 integers X, Y, Z is likely to be the 3135 specified values. 3136 3137 Corresponds to the OpenCL 3138 ``work_group_size_hint`` 3139 attribute. 3140 ".vec_type_hint" string The name of a scalar or vector 3141 type. 3142 3143 Corresponds to the OpenCL 3144 ``vec_type_hint`` attribute. 3145 3146 ".device_enqueue_symbol" string The external symbol name 3147 associated with a kernel. 3148 OpenCL runtime allocates a 3149 global buffer for the symbol 3150 and saves the kernel's address 3151 to it, which is used for 3152 device side enqueueing. Only 3153 available for device side 3154 enqueued kernels. 3155 ".kernarg_segment_size" integer Required The size in bytes of 3156 the kernarg segment 3157 that holds the values 3158 of the arguments to 3159 the kernel. 3160 ".group_segment_fixed_size" integer Required The amount of group 3161 segment memory 3162 required by a 3163 work-group in 3164 bytes. This does not 3165 include any 3166 dynamically allocated 3167 group segment memory 3168 that may be added 3169 when the kernel is 3170 dispatched. 3171 ".private_segment_fixed_size" integer Required The amount of fixed 3172 private address space 3173 memory required for a 3174 work-item in 3175 bytes. If the kernel 3176 uses a dynamic call 3177 stack then additional 3178 space must be added 3179 to this value for the 3180 call stack. 3181 ".kernarg_segment_align" integer Required The maximum byte 3182 alignment of 3183 arguments in the 3184 kernarg segment. Must 3185 be a power of 2. 3186 ".wavefront_size" integer Required Wavefront size. Must 3187 be a power of 2. 3188 ".sgpr_count" integer Required Number of scalar 3189 registers required by a 3190 wavefront for 3191 GFX6-GFX9. A register 3192 is required if it is 3193 used explicitly, or 3194 if a higher numbered 3195 register is used 3196 explicitly. This 3197 includes the special 3198 SGPRs for VCC, Flat 3199 Scratch (GFX7-GFX9) 3200 and XNACK (for 3201 GFX8-GFX9). It does 3202 not include the 16 3203 SGPR added if a trap 3204 handler is 3205 enabled. It is not 3206 rounded up to the 3207 allocation 3208 granularity. 3209 ".vgpr_count" integer Required Number of vector 3210 registers required by 3211 each work-item for 3212 GFX6-GFX9. A register 3213 is required if it is 3214 used explicitly, or 3215 if a higher numbered 3216 register is used 3217 explicitly. 3218 ".agpr_count" integer Required Number of accumulator 3219 registers required by 3220 each work-item for 3221 GFX90A, GFX908. 3222 ".max_flat_workgroup_size" integer Required Maximum flat 3223 work-group size 3224 supported by the 3225 kernel in work-items. 3226 Must be >=1 and 3227 consistent with 3228 ReqdWorkGroupSize if 3229 not 0, 0, 0. 3230 ".sgpr_spill_count" integer Number of stores from 3231 a scalar register to 3232 a register allocator 3233 created spill 3234 location. 3235 ".vgpr_spill_count" integer Number of stores from 3236 a vector register to 3237 a register allocator 3238 created spill 3239 location. 3240 ".kind" string The kind of the kernel 3241 with the following 3242 values: 3243 3244 "normal" 3245 Regular kernels. 3246 3247 "init" 3248 These kernels must be 3249 invoked after loading 3250 the containing code 3251 object and must 3252 complete before any 3253 normal and fini 3254 kernels in the same 3255 code object are 3256 invoked. 3257 3258 "fini" 3259 These kernels must be 3260 invoked before 3261 unloading the 3262 containing code object 3263 and after all init and 3264 normal kernels in the 3265 same code object have 3266 been invoked and 3267 completed. 3268 3269 If omitted, "normal" is 3270 assumed. 3271 =================================== ============== ========= ================================ 3272 3273.. 3274 3275 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3276 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3277 3278 ====================== ============== ========= ================================ 3279 String Key Value Type Required? Description 3280 ====================== ============== ========= ================================ 3281 ".name" string Kernel argument name. 3282 ".type_name" string Kernel argument type name. 3283 ".size" integer Required Kernel argument size in bytes. 3284 ".offset" integer Required Kernel argument offset in 3285 bytes. The offset must be a 3286 multiple of the alignment 3287 required by the argument. 3288 ".value_kind" string Required Kernel argument kind that 3289 specifies how to set up the 3290 corresponding argument. 3291 Values include: 3292 3293 "by_value" 3294 The argument is copied 3295 directly into the kernarg. 3296 3297 "global_buffer" 3298 A global address space pointer 3299 to the buffer data is passed 3300 in the kernarg. 3301 3302 "dynamic_shared_pointer" 3303 A group address space pointer 3304 to dynamically allocated LDS 3305 is passed in the kernarg. 3306 3307 "sampler" 3308 A global address space 3309 pointer to a S# is passed in 3310 the kernarg. 3311 3312 "image" 3313 A global address space 3314 pointer to a T# is passed in 3315 the kernarg. 3316 3317 "pipe" 3318 A global address space pointer 3319 to an OpenCL pipe is passed in 3320 the kernarg. 3321 3322 "queue" 3323 A global address space pointer 3324 to an OpenCL device enqueue 3325 queue is passed in the 3326 kernarg. 3327 3328 "hidden_global_offset_x" 3329 The OpenCL grid dispatch 3330 global offset for the X 3331 dimension is passed in the 3332 kernarg. 3333 3334 "hidden_global_offset_y" 3335 The OpenCL grid dispatch 3336 global offset for the Y 3337 dimension is passed in the 3338 kernarg. 3339 3340 "hidden_global_offset_z" 3341 The OpenCL grid dispatch 3342 global offset for the Z 3343 dimension is passed in the 3344 kernarg. 3345 3346 "hidden_none" 3347 An argument that is not used 3348 by the kernel. Space needs to 3349 be left for it, but it does 3350 not need to be set up. 3351 3352 "hidden_printf_buffer" 3353 A global address space pointer 3354 to the runtime printf buffer 3355 is passed in kernarg. Mutually 3356 exclusive with 3357 "hidden_hostcall_buffer" 3358 before Code Object V5. 3359 3360 "hidden_hostcall_buffer" 3361 A global address space pointer 3362 to the runtime hostcall buffer 3363 is passed in kernarg. Mutually 3364 exclusive with 3365 "hidden_printf_buffer" 3366 before Code Object V5. 3367 3368 "hidden_default_queue" 3369 A global address space pointer 3370 to the OpenCL device enqueue 3371 queue that should be used by 3372 the kernel by default is 3373 passed in the kernarg. 3374 3375 "hidden_completion_action" 3376 A global address space pointer 3377 to help link enqueued kernels into 3378 the ancestor tree for determining 3379 when the parent kernel has finished. 3380 3381 "hidden_multigrid_sync_arg" 3382 A global address space pointer for 3383 multi-grid synchronization is 3384 passed in the kernarg. 3385 3386 ".value_type" string Unused and deprecated. This should no longer 3387 be emitted, but is accepted for compatibility. 3388 3389 ".pointee_align" integer Alignment in bytes of pointee 3390 type for pointer type kernel 3391 argument. Must be a power 3392 of 2. Only present if 3393 ".value_kind" is 3394 "dynamic_shared_pointer". 3395 ".address_space" string Kernel argument address space 3396 qualifier. Only present if 3397 ".value_kind" is "global_buffer" or 3398 "dynamic_shared_pointer". Values 3399 are: 3400 3401 - "private" 3402 - "global" 3403 - "constant" 3404 - "local" 3405 - "generic" 3406 - "region" 3407 3408 .. TODO:: 3409 3410 Is "global_buffer" only "global" 3411 or "constant"? Is 3412 "dynamic_shared_pointer" always 3413 "local"? Can HCC allow "generic"? 3414 How can "private" or "region" 3415 ever happen? 3416 3417 ".access" string Kernel argument access 3418 qualifier. Only present if 3419 ".value_kind" is "image" or 3420 "pipe". Values 3421 are: 3422 3423 - "read_only" 3424 - "write_only" 3425 - "read_write" 3426 3427 .. TODO:: 3428 3429 Does this apply to 3430 "global_buffer"? 3431 3432 ".actual_access" string The actual memory accesses 3433 performed by the kernel on the 3434 kernel argument. Only present if 3435 ".value_kind" is "global_buffer", 3436 "image", or "pipe". This may be 3437 more restrictive than indicated 3438 by ".access" to reflect what the 3439 kernel actual does. If not 3440 present then the runtime must 3441 assume what is implied by 3442 ".access" and ".is_const" . Values 3443 are: 3444 3445 - "read_only" 3446 - "write_only" 3447 - "read_write" 3448 3449 ".is_const" boolean Indicates if the kernel argument 3450 is const qualified. Only present 3451 if ".value_kind" is 3452 "global_buffer". 3453 3454 ".is_restrict" boolean Indicates if the kernel argument 3455 is restrict qualified. Only 3456 present if ".value_kind" is 3457 "global_buffer". 3458 3459 ".is_volatile" boolean Indicates if the kernel argument 3460 is volatile qualified. Only 3461 present if ".value_kind" is 3462 "global_buffer". 3463 3464 ".is_pipe" boolean Indicates if the kernel argument 3465 is pipe qualified. Only present 3466 if ".value_kind" is "pipe". 3467 3468 .. TODO:: 3469 3470 Can "global_buffer" be pipe 3471 qualified? 3472 3473 ====================== ============== ========= ================================ 3474 3475.. _amdgpu-amdhsa-code-object-metadata-v4: 3476 3477Code Object V4 Metadata 3478+++++++++++++++++++++++ 3479 3480Code object V4 metadata is the same as 3481:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3482defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`. 3483 3484 .. table:: AMDHSA Code Object V4 Metadata Map Changes 3485 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3486 3487 ================= ============== ========= ======================================= 3488 String Key Value Type Required? Description 3489 ================= ============== ========= ======================================= 3490 "amdhsa.version" sequence of Required - The first integer is the major 3491 2 integers version. Currently 1. 3492 - The second integer is the minor 3493 version. Currently 1. 3494 "amdhsa.target" string Required The target name of the code using the syntax: 3495 3496 .. code:: 3497 3498 <target-triple> [ "-" <target-id> ] 3499 3500 A canonical target ID must be 3501 used. See :ref:`amdgpu-target-triples` 3502 and :ref:`amdgpu-target-id`. 3503 ================= ============== ========= ======================================= 3504 3505.. _amdgpu-amdhsa-code-object-metadata-v5: 3506 3507Code Object V5 Metadata 3508+++++++++++++++++++++++ 3509 3510.. warning:: 3511 Code object V5 is not the default code object version emitted by this version 3512 of LLVM. 3513 3514 3515Code object V5 metadata is the same as 3516:ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table 3517:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5` and table 3518:ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`. 3519 3520 .. table:: AMDHSA Code Object V5 Metadata Map Changes 3521 :name: amdgpu-amdhsa-code-object-metadata-map-table-v5 3522 3523 ================= ============== ========= ======================================= 3524 String Key Value Type Required? Description 3525 ================= ============== ========= ======================================= 3526 "amdhsa.version" sequence of Required - The first integer is the major 3527 2 integers version. Currently 1. 3528 - The second integer is the minor 3529 version. Currently 2. 3530 ================= ============== ========= ======================================= 3531 3532.. 3533 3534 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes 3535 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5 3536 3537 ====================== ============== ========= ================================ 3538 String Key Value Type Required? Description 3539 ====================== ============== ========= ================================ 3540 ".value_kind" string Required Kernel argument kind that 3541 specifies how to set up the 3542 corresponding argument. 3543 Values include: 3544 the same as code object V3 metadata 3545 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`) 3546 with the following additions: 3547 3548 "hidden_block_count_x" 3549 The grid dispatch work-group count for the X dimension 3550 is passed in the kernarg. Some languages, such as OpenCL, 3551 support a last work-group in each dimension being partial. 3552 This count only includes the non-partial work-group count. 3553 This is not the same as the value in the AQL dispatch packet, 3554 which has the grid size in work-items. 3555 3556 "hidden_block_count_y" 3557 The grid dispatch work-group count for the Y dimension 3558 is passed in the kernarg. Some languages, such as OpenCL, 3559 support a last work-group in each dimension being partial. 3560 This count only includes the non-partial work-group count. 3561 This is not the same as the value in the AQL dispatch packet, 3562 which has the grid size in work-items. If the grid dimensionality 3563 is 1, then must be 1. 3564 3565 "hidden_block_count_z" 3566 The grid dispatch work-group count for the Z dimension 3567 is passed in the kernarg. Some languages, such as OpenCL, 3568 support a last work-group in each dimension being partial. 3569 This count only includes the non-partial work-group count. 3570 This is not the same as the value in the AQL dispatch packet, 3571 which has the grid size in work-items. If the grid dimensionality 3572 is 1 or 2, then must be 1. 3573 3574 "hidden_group_size_x" 3575 The grid dispatch work-group size for the X dimension is 3576 passed in the kernarg. This size only applies to the 3577 non-partial work-groups. This is the same value as the AQL 3578 dispatch packet work-group size. 3579 3580 "hidden_group_size_y" 3581 The grid dispatch work-group size for the Y dimension is 3582 passed in the kernarg. This size only applies to the 3583 non-partial work-groups. This is the same value as the AQL 3584 dispatch packet work-group size. If the grid dimensionality 3585 is 1, then must be 1. 3586 3587 "hidden_group_size_z" 3588 The grid dispatch work-group size for the Z dimension is 3589 passed in the kernarg. This size only applies to the 3590 non-partial work-groups. This is the same value as the AQL 3591 dispatch packet work-group size. If the grid dimensionality 3592 is 1 or 2, then must be 1. 3593 3594 "hidden_remainder_x" 3595 The grid dispatch work group size of the the partial work group 3596 of the X dimension, if it exists. Must be zero if a partial 3597 work group does not exist in the X dimension. 3598 3599 "hidden_remainder_y" 3600 The grid dispatch work group size of the the partial work group 3601 of the Y dimension, if it exists. Must be zero if a partial 3602 work group does not exist in the Y dimension. 3603 3604 "hidden_remainder_z" 3605 The grid dispatch work group size of the the partial work group 3606 of the Z dimension, if it exists. Must be zero if a partial 3607 work group does not exist in the Z dimension. 3608 3609 "hidden_grid_dims" 3610 The grid dispatch dimensionality. This is the same value 3611 as the AQL dispatch packet dimensionality. Must be a value 3612 between 1 and 3. 3613 3614 "hidden_heap_v1" 3615 A global address space pointer to an initialized memory 3616 buffer that conforms to the requirements of the malloc/free 3617 device library V1 version implementation. 3618 3619 "hidden_private_base" 3620 The high 32 bits of the flat addressing private aperture base. 3621 Only used by GFX8 to allow conversion between private segment 3622 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3623 3624 "hidden_shared_base" 3625 The high 32 bits of the flat addressing shared aperture base. 3626 Only used by GFX8 to allow conversion between shared segment 3627 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3628 3629 "hidden_queue_ptr" 3630 A global memory address space pointer to the ROCm runtime 3631 ``struct amd_queue_t`` structure for the HSA queue of the 3632 associated dispatch AQL packet. It is only required for pre-GFX9 3633 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`). 3634 3635 ====================== ============== ========= ================================ 3636 3637.. 3638 3639Kernel Dispatch 3640~~~~~~~~~~~~~~~ 3641 3642The HSA architected queuing language (AQL) defines a user space memory interface 3643that can be used to control the dispatch of kernels, in an agent independent 3644way. An agent can have zero or more AQL queues created for it using an HSA 3645compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3646are 64 bytes) can be placed. See the *HSA Platform System Architecture 3647Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3648 3649The packet processor of a kernel agent is responsible for detecting and 3650dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3651packet processor is implemented by the hardware command processor (CP), 3652asynchronous dispatch controller (ADC) and shader processor input controller 3653(SPI). 3654 3655An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3656the kernel mode driver to initialize and register the AQL queue with CP. 3657 3658To dispatch a kernel the following actions are performed. This can occur in the 3659CPU host program, or from an HSA kernel executing on a GPU. 3660 36611. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3662 executed is obtained. 36632. A pointer to the kernel descriptor (see 3664 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3665 It must be for a kernel that is contained in a code object that was loaded 3666 by an HSA compatible runtime on the kernel agent with which the AQL queue is 3667 associated. 36683. Space is allocated for the kernel arguments using the HSA compatible runtime 3669 allocator for a memory region with the kernarg property for the kernel agent 3670 that will execute the kernel. It must be at least 16-byte aligned. 36714. Kernel argument values are assigned to the kernel argument memory 3672 allocation. The layout is defined in the *HSA Programmer's Language 3673 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3674 kernel argument memory in the same way constant memory is accessed. (Note 3675 that the HSA specification allows an implementation to copy the kernel 3676 argument contents to another location that is accessed by the kernel.) 36775. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3678 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3679 for the packet. The packet must be set up, and the final write must use an 3680 atomic store release to set the packet kind to ensure the packet contents are 3681 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3682 notify the kernel agent that the AQL queue has been updated. These rules, and 3683 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3684 System Architecture Specification* [HSA]_. 36856. A kernel dispatch packet includes information about the actual dispatch, 3686 such as grid and work-group size, together with information from the code 3687 object about the kernel, such as segment sizes. The HSA compatible runtime 3688 queries on the kernel symbol can be used to obtain the code object values 3689 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 36907. CP executes micro-code and is responsible for detecting and setting up the 3691 GPU to execute the wavefronts of a kernel dispatch. 36928. CP ensures that when the a wavefront starts executing the kernel machine 3693 code, the scalar general purpose registers (SGPR) and vector general purpose 3694 registers (VGPR) are set up as required by the machine code. The required 3695 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3696 register state is defined in 3697 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 36989. The prolog of the kernel machine code (see 3699 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3700 before continuing executing the machine code that corresponds to the kernel. 370110. When the kernel dispatch has completed execution, CP signals the completion 3702 signal specified in the kernel dispatch packet if not 0. 3703 3704.. _amdgpu-amdhsa-memory-spaces: 3705 3706Memory Spaces 3707~~~~~~~~~~~~~ 3708 3709The memory space properties are: 3710 3711 .. table:: AMDHSA Memory Spaces 3712 :name: amdgpu-amdhsa-memory-spaces-table 3713 3714 ================= =========== ======== ======= ================== 3715 Memory Space Name HSA Segment Hardware Address NULL Value 3716 Name Name Size 3717 ================= =========== ======== ======= ================== 3718 Private private scratch 32 0x00000000 3719 Local group LDS 32 0xFFFFFFFF 3720 Global global global 64 0x0000000000000000 3721 Constant constant *same as 64 0x0000000000000000 3722 global* 3723 Generic flat flat 64 0x0000000000000000 3724 Region N/A GDS 32 *not implemented 3725 for AMDHSA* 3726 ================= =========== ======== ======= ================== 3727 3728The global and constant memory spaces both use global virtual addresses, which 3729are the same virtual address space used by the CPU. However, some virtual 3730addresses may only be accessible to the CPU, some only accessible by the GPU, 3731and some by both. 3732 3733Using the constant memory space indicates that the data will not change during 3734the execution of the kernel. This allows scalar read instructions to be 3735used. The vector and scalar L1 caches are invalidated of volatile data before 3736each kernel dispatch execution to allow constant memory to change values between 3737kernel dispatches. 3738 3739The local memory space uses the hardware Local Data Store (LDS) which is 3740automatically allocated when the hardware creates work-groups of wavefronts, and 3741freed when all the wavefronts of a work-group have terminated. The data store 3742(DS) instructions can be used to access it. 3743 3744The private memory space uses the hardware scratch memory support. If the kernel 3745uses scratch, then the hardware allocates memory that is accessed using 3746wavefront lane dword (4 byte) interleaving. The mapping used from private 3747address to physical address is: 3748 3749 ``wavefront-scratch-base + 3750 (private-address * wavefront-size * 4) + 3751 (wavefront-lane-id * 4)`` 3752 3753There are different ways that the wavefront scratch base address is determined 3754by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3755memory can be accessed in an interleaved manner using buffer instruction with 3756the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3757instructions, or by flat instructions. If each lane of a wavefront accesses the 3758same private address, the interleaving results in adjacent dwords being accessed 3759and hence requires fewer cache lines to be fetched. Multi-dword access is not 3760supported except by flat and scratch instructions in GFX9-GFX10. 3761 3762The generic address space uses the hardware flat address support available in 3763GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3764local apertures), that are outside the range of addressible global memory, to 3765map from a flat address to a private or local address. 3766 3767FLAT instructions can take a flat address and access global, private (scratch) 3768and group (LDS) memory depending on if the address is within one of the 3769aperture ranges. Flat access to scratch requires hardware aperture setup and 3770setup in the kernel prologue (see 3771:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3772hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3773:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3774 3775To convert between a segment address and a flat address the base address of the 3776apertures address can be used. For GFX7-GFX8 these are available in the 3777:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3778Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3779GFX9-GFX10 the aperture base addresses are directly available as inline constant 3780registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3781address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3782which makes it easier to convert from flat to segment or segment to flat. 3783 3784Image and Samplers 3785~~~~~~~~~~~~~~~~~~ 3786 3787Image and sample handles created by an HSA compatible runtime (see 3788:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3789object respectively. In order to support the HSA ``query_sampler`` operations 3790two extra dwords are used to store the HSA BRIG enumeration values for the 3791queries that are not trivially deducible from the S# representation. 3792 3793HSA Signals 3794~~~~~~~~~~~ 3795 3796HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3797are 64-bit addresses of a structure allocated in memory accessible from both the 3798CPU and GPU. The structure is defined by the runtime and subject to change 3799between releases. For example, see [AMD-ROCm-github]_. 3800 3801.. _amdgpu-amdhsa-hsa-aql-queue: 3802 3803HSA AQL Queue 3804~~~~~~~~~~~~~ 3805 3806The HSA AQL queue structure is defined by an HSA compatible runtime (see 3807:ref:`amdgpu-os`) and subject to change between releases. For example, see 3808[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3809certain language features such as the flat address aperture bases. It also 3810contains fields used by CP such as managing the allocation of scratch memory. 3811 3812.. _amdgpu-amdhsa-kernel-descriptor: 3813 3814Kernel Descriptor 3815~~~~~~~~~~~~~~~~~ 3816 3817A kernel descriptor consists of the information needed by CP to initiate the 3818execution of a kernel, including the entry point address of the machine code 3819that implements the kernel. 3820 3821Code Object V3 Kernel Descriptor 3822++++++++++++++++++++++++++++++++ 3823 3824CP microcode requires the Kernel descriptor to be allocated on 64-byte 3825alignment. 3826 3827The fields used by CP for code objects before V3 also match those specified in 3828:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3829 3830 .. table:: Code Object V3 Kernel Descriptor 3831 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3832 3833 ======= ======= =============================== ============================ 3834 Bits Size Field Name Description 3835 ======= ======= =============================== ============================ 3836 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3837 address space memory 3838 required for a work-group 3839 in bytes. This does not 3840 include any dynamically 3841 allocated local address 3842 space memory that may be 3843 added when the kernel is 3844 dispatched. 3845 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3846 private address space 3847 memory required for a 3848 work-item in bytes. 3849 Additional space may need to 3850 be added to this value if 3851 the call stack has 3852 non-inlined function calls. 3853 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3854 memory pointed to by the 3855 AQL dispatch packet. The 3856 kernarg memory is used to 3857 pass arguments to the 3858 kernel. 3859 3860 * If the kernarg pointer in 3861 the dispatch packet is NULL 3862 then there are no kernel 3863 arguments. 3864 * If the kernarg pointer in 3865 the dispatch packet is 3866 not NULL and this value 3867 is 0 then the kernarg 3868 memory size is 3869 unspecified. 3870 * If the kernarg pointer in 3871 the dispatch packet is 3872 not NULL and this value 3873 is not 0 then the value 3874 specifies the kernarg 3875 memory size in bytes. It 3876 is recommended to provide 3877 a value as it may be used 3878 by CP to optimize making 3879 the kernarg memory 3880 visible to the kernel 3881 code. 3882 3883 127:96 4 bytes Reserved, must be 0. 3884 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3885 negative) from base 3886 address of kernel 3887 descriptor to kernel's 3888 entry point instruction 3889 which must be 256 byte 3890 aligned. 3891 351:272 20 Reserved, must be 0. 3892 bytes 3893 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3894 Reserved, must be 0. 3895 GFX90A, GFX940 3896 Compute Shader (CS) 3897 program settings used by 3898 CP to set up 3899 ``COMPUTE_PGM_RSRC3`` 3900 configuration 3901 register. See 3902 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3903 GFX10 3904 Compute Shader (CS) 3905 program settings used by 3906 CP to set up 3907 ``COMPUTE_PGM_RSRC3`` 3908 configuration 3909 register. See 3910 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3911 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3912 program settings used by 3913 CP to set up 3914 ``COMPUTE_PGM_RSRC1`` 3915 configuration 3916 register. See 3917 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3918 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3919 program settings used by 3920 CP to set up 3921 ``COMPUTE_PGM_RSRC2`` 3922 configuration 3923 register. See 3924 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3925 458:448 7 bits *See separate bits below.* Enable the setup of the 3926 SGPR user data registers 3927 (see 3928 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3929 3930 The total number of SGPR 3931 user data registers 3932 requested must not exceed 3933 16 and match value in 3934 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3935 Any requests beyond 16 3936 will be ignored. 3937 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3938 _BUFFER column of 3939 :ref:`amdgpu-processor-table` 3940 specifies *Architected flat 3941 scratch* then not supported 3942 and must be 0, 3943 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3944 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3945 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3946 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3947 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3948 column of 3949 :ref:`amdgpu-processor-table` 3950 specifies *Architected flat 3951 scratch* then not supported 3952 and must be 0, 3953 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3954 _SIZE 3955 457:455 3 bits Reserved, must be 0. 3956 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3957 Reserved, must be 0. 3958 GFX10 3959 - If 0 execute in 3960 wavefront size 64 mode. 3961 - If 1 execute in 3962 native wavefront size 3963 32 mode. 3964 463:459 1 bit Reserved, must be 0. 3965 464 1 bit RESERVED_464 Deprecated, must be 0. 3966 467:465 3 bits Reserved, must be 0. 3967 468 1 bit RESERVED_468 Deprecated, must be 0. 3968 469:471 3 bits Reserved, must be 0. 3969 511:472 5 bytes Reserved, must be 0. 3970 512 **Total size 64 bytes.** 3971 ======= ==================================================================== 3972 3973.. 3974 3975 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3976 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3977 3978 ======= ======= =============================== =========================================================================== 3979 Bits Size Field Name Description 3980 ======= ======= =============================== =========================================================================== 3981 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3982 blocks used by each work-item; 3983 granularity is device 3984 specific: 3985 3986 GFX6-GFX9 3987 - vgprs_used 0..256 3988 - max(0, ceil(vgprs_used / 4) - 1) 3989 GFX90A, GFX940 3990 - vgprs_used 0..512 3991 - vgprs_used = align(arch_vgprs, 4) 3992 + acc_vgprs 3993 - max(0, ceil(vgprs_used / 8) - 1) 3994 GFX10 (wavefront size 64) 3995 - max_vgpr 1..256 3996 - max(0, ceil(vgprs_used / 4) - 1) 3997 GFX10 (wavefront size 32) 3998 - max_vgpr 1..256 3999 - max(0, ceil(vgprs_used / 8) - 1) 4000 4001 Where vgprs_used is defined 4002 as the highest VGPR number 4003 explicitly referenced plus 4004 one. 4005 4006 Used by CP to set up 4007 ``COMPUTE_PGM_RSRC1.VGPRS``. 4008 4009 The 4010 :ref:`amdgpu-assembler` 4011 calculates this 4012 automatically for the 4013 selected processor from 4014 values provided to the 4015 `.amdhsa_kernel` directive 4016 by the 4017 `.amdhsa_next_free_vgpr` 4018 nested directive (see 4019 :ref:`amdhsa-kernel-directives-table`). 4020 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 4021 blocks used by a wavefront; 4022 granularity is device 4023 specific: 4024 4025 GFX6-GFX8 4026 - sgprs_used 0..112 4027 - max(0, ceil(sgprs_used / 8) - 1) 4028 GFX9 4029 - sgprs_used 0..112 4030 - 2 * max(0, ceil(sgprs_used / 16) - 1) 4031 GFX10 4032 Reserved, must be 0. 4033 (128 SGPRs always 4034 allocated.) 4035 4036 Where sgprs_used is 4037 defined as the highest 4038 SGPR number explicitly 4039 referenced plus one, plus 4040 a target specific number 4041 of additional special 4042 SGPRs for VCC, 4043 FLAT_SCRATCH (GFX7+) and 4044 XNACK_MASK (GFX8+), and 4045 any additional 4046 target specific 4047 limitations. It does not 4048 include the 16 SGPRs added 4049 if a trap handler is 4050 enabled. 4051 4052 The target specific 4053 limitations and special 4054 SGPR layout are defined in 4055 the hardware 4056 documentation, which can 4057 be found in the 4058 :ref:`amdgpu-processors` 4059 table. 4060 4061 Used by CP to set up 4062 ``COMPUTE_PGM_RSRC1.SGPRS``. 4063 4064 The 4065 :ref:`amdgpu-assembler` 4066 calculates this 4067 automatically for the 4068 selected processor from 4069 values provided to the 4070 `.amdhsa_kernel` directive 4071 by the 4072 `.amdhsa_next_free_sgpr` 4073 and `.amdhsa_reserve_*` 4074 nested directives (see 4075 :ref:`amdhsa-kernel-directives-table`). 4076 11:10 2 bits PRIORITY Must be 0. 4077 4078 Start executing wavefront 4079 at the specified priority. 4080 4081 CP is responsible for 4082 filling in 4083 ``COMPUTE_PGM_RSRC1.PRIORITY``. 4084 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 4085 with specified rounding 4086 mode for single (32 4087 bit) floating point 4088 precision floating point 4089 operations. 4090 4091 Floating point rounding 4092 mode values are defined in 4093 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4094 4095 Used by CP to set up 4096 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4097 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 4098 with specified rounding 4099 denorm mode for half/double (16 4100 and 64-bit) floating point 4101 precision floating point 4102 operations. 4103 4104 Floating point rounding 4105 mode values are defined in 4106 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4107 4108 Used by CP to set up 4109 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4110 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 4111 with specified denorm mode 4112 for single (32 4113 bit) floating point 4114 precision floating point 4115 operations. 4116 4117 Floating point denorm mode 4118 values are defined in 4119 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4120 4121 Used by CP to set up 4122 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4123 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 4124 with specified denorm mode 4125 for half/double (16 4126 and 64-bit) floating point 4127 precision floating point 4128 operations. 4129 4130 Floating point denorm mode 4131 values are defined in 4132 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4133 4134 Used by CP to set up 4135 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4136 20 1 bit PRIV Must be 0. 4137 4138 Start executing wavefront 4139 in privilege trap handler 4140 mode. 4141 4142 CP is responsible for 4143 filling in 4144 ``COMPUTE_PGM_RSRC1.PRIV``. 4145 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 4146 with DX10 clamp mode 4147 enabled. Used by the vector 4148 ALU to force DX10 style 4149 treatment of NaN's (when 4150 set, clamp NaN to zero, 4151 otherwise pass NaN 4152 through). 4153 4154 Used by CP to set up 4155 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 4156 22 1 bit DEBUG_MODE Must be 0. 4157 4158 Start executing wavefront 4159 in single step mode. 4160 4161 CP is responsible for 4162 filling in 4163 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 4164 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 4165 with IEEE mode 4166 enabled. Floating point 4167 opcodes that support 4168 exception flag gathering 4169 will quiet and propagate 4170 signaling-NaN inputs per 4171 IEEE 754-2008. Min_dx10 and 4172 max_dx10 become IEEE 4173 754-2008 compliant due to 4174 signaling-NaN propagation 4175 and quieting. 4176 4177 Used by CP to set up 4178 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 4179 24 1 bit BULKY Must be 0. 4180 4181 Only one work-group allowed 4182 to execute on a compute 4183 unit. 4184 4185 CP is responsible for 4186 filling in 4187 ``COMPUTE_PGM_RSRC1.BULKY``. 4188 25 1 bit CDBG_USER Must be 0. 4189 4190 Flag that can be used to 4191 control debugging code. 4192 4193 CP is responsible for 4194 filling in 4195 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4196 26 1 bit FP16_OVFL GFX6-GFX8 4197 Reserved, must be 0. 4198 GFX9-GFX10 4199 Wavefront starts execution 4200 with specified fp16 overflow 4201 mode. 4202 4203 - If 0, fp16 overflow generates 4204 +/-INF values. 4205 - If 1, fp16 overflow that is the 4206 result of an +/-INF input value 4207 or divide by 0 produces a +/-INF, 4208 otherwise clamps computed 4209 overflow to +/-MAX_FP16 as 4210 appropriate. 4211 4212 Used by CP to set up 4213 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4214 28:27 2 bits Reserved, must be 0. 4215 29 1 bit WGP_MODE GFX6-GFX9 4216 Reserved, must be 0. 4217 GFX10 4218 - If 0 execute work-groups in 4219 CU wavefront execution mode. 4220 - If 1 execute work-groups on 4221 in WGP wavefront execution mode. 4222 4223 See :ref:`amdgpu-amdhsa-memory-model`. 4224 4225 Used by CP to set up 4226 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4227 30 1 bit MEM_ORDERED GFX6-GFX9 4228 Reserved, must be 0. 4229 GFX10 4230 Controls the behavior of the 4231 s_waitcnt's vmcnt and vscnt 4232 counters. 4233 4234 - If 0 vmcnt reports completion 4235 of load and atomic with return 4236 out of order with sample 4237 instructions, and the vscnt 4238 reports the completion of 4239 store and atomic without 4240 return in order. 4241 - If 1 vmcnt reports completion 4242 of load, atomic with return 4243 and sample instructions in 4244 order, and the vscnt reports 4245 the completion of store and 4246 atomic without return in order. 4247 4248 Used by CP to set up 4249 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4250 31 1 bit FWD_PROGRESS GFX6-GFX9 4251 Reserved, must be 0. 4252 GFX10 4253 - If 0 execute SIMD wavefronts 4254 using oldest first policy. 4255 - If 1 execute SIMD wavefronts to 4256 ensure wavefronts will make some 4257 forward progress. 4258 4259 Used by CP to set up 4260 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4261 32 **Total size 4 bytes** 4262 ======= =================================================================================================================== 4263 4264.. 4265 4266 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4267 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4268 4269 ======= ======= =============================== =========================================================================== 4270 Bits Size Field Name Description 4271 ======= ======= =============================== =========================================================================== 4272 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4273 private segment. 4274 * If the *Target Properties* 4275 column of 4276 :ref:`amdgpu-processor-table` 4277 does not specify 4278 *Architected flat 4279 scratch* then enable the 4280 setup of the SGPR 4281 wavefront scratch offset 4282 system register (see 4283 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4284 * If the *Target Properties* 4285 column of 4286 :ref:`amdgpu-processor-table` 4287 specifies *Architected 4288 flat scratch* then enable 4289 the setup of the 4290 FLAT_SCRATCH register 4291 pair (see 4292 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4293 4294 Used by CP to set up 4295 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4296 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4297 user data 4298 registers requested. This 4299 number must be greater than 4300 or equal to the number of user 4301 data registers enabled. 4302 4303 Used by CP to set up 4304 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4305 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4306 4307 This bit represents 4308 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4309 which is set by the CP if 4310 the runtime has installed a 4311 trap handler. 4312 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4313 system SGPR register for 4314 the work-group id in the X 4315 dimension (see 4316 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4317 4318 Used by CP to set up 4319 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4320 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4321 system SGPR register for 4322 the work-group id in the Y 4323 dimension (see 4324 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4325 4326 Used by CP to set up 4327 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4328 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4329 system SGPR register for 4330 the work-group id in the Z 4331 dimension (see 4332 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4333 4334 Used by CP to set up 4335 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4336 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4337 system SGPR register for 4338 work-group information (see 4339 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4340 4341 Used by CP to set up 4342 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4343 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4344 VGPR system registers used 4345 for the work-item ID. 4346 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4347 defines the values. 4348 4349 Used by CP to set up 4350 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4351 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4352 4353 Wavefront starts execution 4354 with address watch 4355 exceptions enabled which 4356 are generated when L1 has 4357 witnessed a thread access 4358 an *address of 4359 interest*. 4360 4361 CP is responsible for 4362 filling in the address 4363 watch bit in 4364 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4365 according to what the 4366 runtime requests. 4367 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4368 4369 Wavefront starts execution 4370 with memory violation 4371 exceptions exceptions 4372 enabled which are generated 4373 when a memory violation has 4374 occurred for this wavefront from 4375 L1 or LDS 4376 (write-to-read-only-memory, 4377 mis-aligned atomic, LDS 4378 address out of range, 4379 illegal address, etc.). 4380 4381 CP sets the memory 4382 violation bit in 4383 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4384 according to what the 4385 runtime requests. 4386 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4387 4388 CP uses the rounded value 4389 from the dispatch packet, 4390 not this value, as the 4391 dispatch may contain 4392 dynamically allocated group 4393 segment memory. CP writes 4394 directly to 4395 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4396 4397 Amount of group segment 4398 (LDS) to allocate for each 4399 work-group. Granularity is 4400 device specific: 4401 4402 GFX6 4403 roundup(lds-size / (64 * 4)) 4404 GFX7-GFX10 4405 roundup(lds-size / (128 * 4)) 4406 4407 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4408 _INVALID_OPERATION with specified exceptions 4409 enabled. 4410 4411 Used by CP to set up 4412 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4413 (set from bits 0..6). 4414 4415 IEEE 754 FP Invalid 4416 Operation 4417 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4418 _SOURCE input operands is a 4419 denormal number 4420 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4421 _DIVISION_BY_ZERO Zero 4422 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4423 _OVERFLOW 4424 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4425 _UNDERFLOW 4426 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4427 _INEXACT 4428 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4429 _ZERO (rcp_iflag_f32 instruction 4430 only) 4431 31 1 bit Reserved, must be 0. 4432 32 **Total size 4 bytes.** 4433 ======= =================================================================================================================== 4434 4435.. 4436 4437 .. table:: compute_pgm_rsrc3 for GFX90A, GFX940 4438 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4439 4440 ======= ======= =============================== =========================================================================== 4441 Bits Size Field Name Description 4442 ======= ======= =============================== =========================================================================== 4443 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4444 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4445 63 - accum-offset = 256. 4446 6:15 10 Reserved, must be 0. 4447 bits 4448 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4449 launched in the same CU. 4450 - If 1 the waves of a work-group can be 4451 launched in different CUs. The waves 4452 cannot use S_BARRIER or LDS. 4453 17:31 15 Reserved, must be 0. 4454 bits 4455 32 **Total size 4 bytes.** 4456 ======= =================================================================================================================== 4457 4458.. 4459 4460 .. table:: compute_pgm_rsrc3 for GFX10 4461 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4462 4463 ======= ======= =============================== =========================================================================== 4464 Bits Size Field Name Description 4465 ======= ======= =============================== =========================================================================== 4466 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPR blocks when executing in subvector mode. For 4467 wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity 4468 of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does 4469 not exceed 256. For wavefront size 32 shared_vgpr_count must be 0. 4470 31:4 28 Reserved, must be 0. 4471 bits 4472 32 **Total size 4 bytes.** 4473 ======= =================================================================================================================== 4474 4475.. 4476 4477 .. table:: Floating Point Rounding Mode Enumeration Values 4478 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4479 4480 ====================================== ===== ============================== 4481 Enumeration Name Value Description 4482 ====================================== ===== ============================== 4483 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4484 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4485 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4486 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4487 ====================================== ===== ============================== 4488 4489.. 4490 4491 .. table:: Floating Point Denorm Mode Enumeration Values 4492 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4493 4494 ====================================== ===== ============================== 4495 Enumeration Name Value Description 4496 ====================================== ===== ============================== 4497 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4498 Denorms 4499 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4500 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4501 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4502 ====================================== ===== ============================== 4503 4504.. 4505 4506 .. table:: System VGPR Work-Item ID Enumeration Values 4507 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4508 4509 ======================================== ===== ============================ 4510 Enumeration Name Value Description 4511 ======================================== ===== ============================ 4512 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4513 ID. 4514 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4515 dimensions ID. 4516 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4517 dimensions ID. 4518 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4519 ======================================== ===== ============================ 4520 4521.. _amdgpu-amdhsa-initial-kernel-execution-state: 4522 4523Initial Kernel Execution State 4524~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4525 4526This section defines the register state that will be set up by the packet 4527processor prior to the start of execution of every wavefront. This is limited by 4528the constraints of the hardware controllers of CP/ADC/SPI. 4529 4530The order of the SGPR registers is defined, but the compiler can specify which 4531ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4532fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4533for enabled registers are dense starting at SGPR0: the first enabled register is 4534SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4535an SGPR number. 4536 4537The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4538all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4539using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4540actually initialized. These are then immediately followed by the System SGPRs 4541that are set up by ADC/SPI and can have different values for each wavefront of 4542the grid dispatch. 4543 4544SGPR register initial state is defined in 4545:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4546 4547 .. table:: SGPR Register Set Up Order 4548 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4549 4550 ========== ========================== ====== ============================== 4551 SGPR Order Name Number Description 4552 (kernel descriptor enable of 4553 field) SGPRs 4554 ========== ========================== ====== ============================== 4555 First Private Segment Buffer 4 See 4556 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4557 _segment_buffer) 4558 then Dispatch Ptr 2 64-bit address of AQL dispatch 4559 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4560 actually executing. 4561 then Queue Ptr 2 64-bit address of amd_queue_t 4562 (enable_sgpr_queue_ptr) object for AQL queue on which 4563 the dispatch packet was 4564 queued. 4565 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4566 (enable_sgpr_kernarg segment. This is directly 4567 _segment_ptr) copied from the 4568 kernarg_address in the kernel 4569 dispatch packet. 4570 4571 Having CP load it once avoids 4572 loading it at the beginning of 4573 every wavefront. 4574 then Dispatch Id 2 64-bit Dispatch ID of the 4575 (enable_sgpr_dispatch_id) dispatch packet being 4576 executed. 4577 then Flat Scratch Init 2 See 4578 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4579 _init) 4580 then Private Segment Size 1 The 32-bit byte size of a 4581 (enable_sgpr_private single work-item's memory 4582 _segment_size) allocation. This is the 4583 value from the kernel 4584 dispatch packet Private 4585 Segment Byte Size rounded up 4586 by CP to a multiple of 4587 DWORD. 4588 4589 Having CP load it once avoids 4590 loading it at the beginning of 4591 every wavefront. 4592 4593 This is not used for 4594 GFX7-GFX8 since it is the same 4595 value as the second SGPR of 4596 Flat Scratch Init. However, it 4597 may be needed for GFX9-GFX10 which 4598 changes the meaning of the 4599 Flat Scratch Init value. 4600 then Work-Group Id X 1 32-bit work-group id in X 4601 (enable_sgpr_workgroup_id dimension of grid for 4602 _X) wavefront. 4603 then Work-Group Id Y 1 32-bit work-group id in Y 4604 (enable_sgpr_workgroup_id dimension of grid for 4605 _Y) wavefront. 4606 then Work-Group Id Z 1 32-bit work-group id in Z 4607 (enable_sgpr_workgroup_id dimension of grid for 4608 _Z) wavefront. 4609 then Work-Group Info 1 {first_wavefront, 14'b0000, 4610 (enable_sgpr_workgroup ordered_append_term[10:0], 4611 _info) threadgroup_size_in_wavefronts[5:0]} 4612 then Scratch Wavefront Offset 1 See 4613 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4614 _segment_wavefront_offset) and 4615 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4616 ========== ========================== ====== ============================== 4617 4618The order of the VGPR registers is defined, but the compiler can specify which 4619ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4620fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4621for enabled registers are dense starting at VGPR0: the first enabled register is 4622VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4623VGPR number. 4624 4625There are different methods used for the VGPR initial state: 4626 4627* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4628 specifies otherwise, a separate VGPR register is used per work-item ID. The 4629 VGPR register initial state for this method is defined in 4630 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4631* If *Target Properties* column of :ref:`amdgpu-processor-table` 4632 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4633 for all work-item IDs. The register layout for this method is defined in 4634 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4635 4636 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4637 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4638 4639 ========== ========================== ====== ============================== 4640 VGPR Order Name Number Description 4641 (kernel descriptor enable of 4642 field) VGPRs 4643 ========== ========================== ====== ============================== 4644 First Work-Item Id X 1 32-bit work-item id in X 4645 (Always initialized) dimension of work-group for 4646 wavefront lane. 4647 then Work-Item Id Y 1 32-bit work-item id in Y 4648 (enable_vgpr_workitem_id dimension of work-group for 4649 > 0) wavefront lane. 4650 then Work-Item Id Z 1 32-bit work-item id in Z 4651 (enable_vgpr_workitem_id dimension of work-group for 4652 > 1) wavefront lane. 4653 ========== ========================== ====== ============================== 4654 4655.. 4656 4657 .. table:: Register Layout for Packed Work-Item ID Method 4658 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4659 4660 ======= ======= ================ ========================================= 4661 Bits Size Field Name Description 4662 ======= ======= ================ ========================================= 4663 0:9 10 bits Work-Item Id X Work-item id in X 4664 dimension of work-group for 4665 wavefront lane. 4666 4667 Always initialized. 4668 4669 10:19 10 bits Work-Item Id Y Work-item id in Y 4670 dimension of work-group for 4671 wavefront lane. 4672 4673 Initialized if enable_vgpr_workitem_id > 4674 0, otherwise set to 0. 4675 20:29 10 bits Work-Item Id Z Work-item id in Z 4676 dimension of work-group for 4677 wavefront lane. 4678 4679 Initialized if enable_vgpr_workitem_id > 4680 1, otherwise set to 0. 4681 30:31 2 bits Reserved, set to 0. 4682 ======= ======= ================ ========================================= 4683 4684The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4685 46861. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4687 registers. 46882. Work-group Id registers X, Y, Z are set by ADC which supports any 4689 combination including none. 46903. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4691 its value cannot be included with the flat scratch init value which is per 4692 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 46934. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4694 or (X, Y, Z). 46955. Flat Scratch register pair initialization is described in 4696 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4697 4698The global segment can be accessed either using buffer instructions (GFX6 which 4699has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4700instructions (GFX9-GFX10). 4701 4702If buffer operations are used, then the compiler can generate a V# with the 4703following properties: 4704 4705* base address of 0 4706* no swizzle 4707* ATC: 1 if IOMMU present (such as APU) 4708* ptr64: 1 4709* MTYPE set to support memory coherence that matches the runtime (such as CC for 4710 APU and NC for dGPU). 4711 4712.. _amdgpu-amdhsa-kernel-prolog: 4713 4714Kernel Prolog 4715~~~~~~~~~~~~~ 4716 4717The compiler performs initialization in the kernel prologue depending on the 4718target and information about things like stack usage in the kernel and called 4719functions. Some of this initialization requires the compiler to request certain 4720User and System SGPRs be present in the 4721:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4722:ref:`amdgpu-amdhsa-kernel-descriptor`. 4723 4724.. _amdgpu-amdhsa-kernel-prolog-cfi: 4725 4726CFI 4727+++ 4728 47291. The CFI return address is undefined. 4730 47312. The CFI CFA is defined using an expression which evaluates to a location 4732 description that comprises one memory location description for the 4733 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4734 4735.. _amdgpu-amdhsa-kernel-prolog-m0: 4736 4737M0 4738++ 4739 4740GFX6-GFX8 4741 The M0 register must be initialized with a value at least the total LDS size 4742 if the kernel may access LDS via DS or flat operations. Total LDS size is 4743 available in dispatch packet. For M0, it is also possible to use maximum 4744 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4745 GFX7-GFX8). 4746GFX9-GFX10 4747 The M0 register is not used for range checking LDS accesses and so does not 4748 need to be initialized in the prolog. 4749 4750.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4751 4752Stack Pointer 4753+++++++++++++ 4754 4755If the kernel has function calls it must set up the ABI stack pointer described 4756in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4757SGPR32 to the unswizzled scratch offset of the address past the last local 4758allocation. 4759 4760.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4761 4762Frame Pointer 4763+++++++++++++ 4764 4765If the kernel needs a frame pointer for the reasons defined in 4766``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4767kernel prolog. If a frame pointer is not required then all uses of the frame 4768pointer are replaced with immediate ``0`` offsets. 4769 4770.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4771 4772Flat Scratch 4773++++++++++++ 4774 4775There are different methods used for initializing flat scratch: 4776 4777* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4778 specifies *Does not support generic address space*: 4779 4780 Flat scratch is not supported and there is no flat scratch register pair. 4781 4782* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4783 specifies *Offset flat scratch*: 4784 4785 If the kernel or any function it calls may use flat operations to access 4786 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4787 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4788 Scratch Wavefront Offset SGPR registers (see 4789 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4790 4791 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4792 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4793 being managed by SPI for the queue executing the kernel dispatch. This is 4794 the same value used in the Scratch Segment Buffer V# base address. 4795 4796 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4797 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4798 4799 The prolog must add the value of Scratch Wavefront Offset to get the 4800 wavefront's byte scratch backing memory offset from 4801 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4802 4803 The Scratch Wavefront Offset must also be used as an offset with Private 4804 segment address when using the Scratch Segment Buffer. 4805 4806 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4807 shifted by 8 before moving into FLAT_SCRATCH_HI. 4808 4809 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4810 SGPRn is the highest numbered SGPR allocated to the wavefront). 4811 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4812 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4813 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4814 aperture. 4815 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4816 work-items scratch memory usage. 4817 4818 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4819 checks that the value in the kernel dispatch packet Private Segment Byte 4820 Size is not larger and requests the runtime to increase the queue's scratch 4821 size if necessary. 4822 4823 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4824 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4825 loading it at the beginning of every wavefront. 4826 4827 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4828 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4829 in flat memory instructions. 4830 4831* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4832 specifies *Absolute flat scratch*: 4833 4834 If the kernel or any function it calls may use flat operations to access 4835 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4836 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4837 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4838 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4839 4840 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4841 memory being managed by SPI for the queue executing the kernel dispatch. 4842 4843 CP obtains this from the runtime. 4844 4845 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4846 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4847 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4848 memory instructions. 4849 4850 The Scratch Wavefront Offset must also be used as an offset with Private 4851 segment address when using the Scratch Segment Buffer (see 4852 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4853 4854* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4855 specifies *Architected flat scratch*: 4856 4857 If ENABLE_PRIVATE_SEGMENT is enabled in 4858 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4859 register pair will be initialized to the 64-bit address of the base of scratch 4860 backing memory being managed by SPI for the queue executing the kernel 4861 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4862 flat scratch base in flat memory instructions. 4863 4864.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4865 4866Private Segment Buffer 4867++++++++++++++++++++++ 4868 4869If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4870*Architected flat scratch* then a Private Segment Buffer is not supported. 4871Instead the flat SCRATCH instructions are used. 4872 4873Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4874that are used as a V# to access scratch. CP uses the value provided by the 4875runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4876access the private memory space using a segment address. See 4877:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4878 4879The scratch V# is a four-aligned SGPR and always selected for the kernel as 4880follows: 4881 4882 - If it is known during instruction selection that there is stack usage, 4883 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4884 optimizations are disabled (``-O0``), if stack objects already exist (for 4885 locals, etc.), or if there are any function calls. 4886 4887 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4888 are reserved for the tentative scratch V#. These will be used if it is 4889 determined that spilling is needed. 4890 4891 - If no use is made of the tentative scratch V#, then it is unreserved, 4892 and the register count is determined ignoring it. 4893 - If use is made of the tentative scratch V#, then its register numbers 4894 are shifted to the first four-aligned SGPR index after the highest one 4895 allocated by the register allocator, and all uses are updated. The 4896 register count includes them in the shifted location. 4897 - In either case, if the processor has the SGPR allocation bug, the 4898 tentative allocation is not shifted or unreserved in order to ensure 4899 the register count is higher to workaround the bug. 4900 4901 .. note:: 4902 4903 This approach of using a tentative scratch V# and shifting the register 4904 numbers if used avoids having to perform register allocation a second 4905 time if the tentative V# is eliminated. This is more efficient and 4906 avoids the problem that the second register allocation may perform 4907 spilling which will fail as there is no longer a scratch V#. 4908 4909When the kernel prolog code is being emitted it is known whether the scratch V# 4910described above is actually used. If it is, the prolog code must set it up by 4911copying the Private Segment Buffer to the scratch V# registers and then adding 4912the Private Segment Wavefront Offset to the queue base address in the V#. The 4913result is a V# with a base address pointing to the beginning of the wavefront 4914scratch backing memory. 4915 4916The Private Segment Buffer is always requested, but the Private Segment 4917Wavefront Offset is only requested if it is used (see 4918:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4919 4920.. _amdgpu-amdhsa-memory-model: 4921 4922Memory Model 4923~~~~~~~~~~~~ 4924 4925This section describes the mapping of the LLVM memory model onto AMDGPU machine 4926code (see :ref:`memmodel`). 4927 4928The AMDGPU backend supports the memory synchronization scopes specified in 4929:ref:`amdgpu-memory-scopes`. 4930 4931The code sequences used to implement the memory model specify the order of 4932instructions that a single thread must execute. The ``s_waitcnt`` and cache 4933management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4934to other memory instructions executed by the same thread. This allows them to be 4935moved earlier or later which can allow them to be combined with other instances 4936of the same instruction, or hoisted/sunk out of loops to improve performance. 4937Only the instructions related to the memory model are given; additional 4938``s_waitcnt`` instructions are required to ensure registers are defined before 4939being used. These may be able to be combined with the memory model ``s_waitcnt`` 4940instructions as described above. 4941 4942The AMDGPU backend supports the following memory models: 4943 4944 HSA Memory Model [HSA]_ 4945 The HSA memory model uses a single happens-before relation for all address 4946 spaces (see :ref:`amdgpu-address-spaces`). 4947 OpenCL Memory Model [OpenCL]_ 4948 The OpenCL memory model which has separate happens-before relations for the 4949 global and local address spaces. Only a fence specifying both global and 4950 local address space, and seq_cst instructions join the relationships. Since 4951 the LLVM ``memfence`` instruction does not allow an address space to be 4952 specified the OpenCL fence has to conservatively assume both local and 4953 global address space was specified. However, optimizations can often be 4954 done to eliminate the additional ``s_waitcnt`` instructions when there are 4955 no intervening memory instructions which access the corresponding address 4956 space. The code sequences in the table indicate what can be omitted for the 4957 OpenCL memory. The target triple environment is used to determine if the 4958 source language is OpenCL (see :ref:`amdgpu-opencl`). 4959 4960``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4961operations. 4962 4963``buffer/global/flat_load/store/atomic`` instructions to global memory are 4964termed vector memory operations. 4965 4966Private address space uses ``buffer_load/store`` using the scratch V# 4967(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4968is accessing the memory, atomic memory orderings are not meaningful, and all 4969accesses are treated as non-atomic. 4970 4971Constant address space uses ``buffer/global_load`` instructions (or equivalent 4972scalar memory instructions). Since the constant address space contents do not 4973change during the execution of a kernel dispatch it is not legal to perform 4974stores, and atomic memory orderings are not meaningful, and all accesses are 4975treated as non-atomic. 4976 4977A memory synchronization scope wider than work-group is not meaningful for the 4978group (LDS) address space and is treated as work-group. 4979 4980The memory model does not support the region address space which is treated as 4981non-atomic. 4982 4983Acquire memory ordering is not meaningful on store atomic instructions and is 4984treated as non-atomic. 4985 4986Release memory ordering is not meaningful on load atomic instructions and is 4987treated a non-atomic. 4988 4989Acquire-release memory ordering is not meaningful on load or store atomic 4990instructions and is treated as acquire and release respectively. 4991 4992The memory order also adds the single thread optimization constraints defined in 4993table 4994:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4995 4996 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4997 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4998 4999 ============ ============================================================== 5000 LLVM Memory Optimization Constraints 5001 Ordering 5002 ============ ============================================================== 5003 unordered *none* 5004 monotonic *none* 5005 acquire - If a load atomic/atomicrmw then no following load/load 5006 atomic/store/store atomic/atomicrmw/fence instruction can be 5007 moved before the acquire. 5008 - If a fence then same as load atomic, plus no preceding 5009 associated fence-paired-atomic can be moved after the fence. 5010 release - If a store atomic/atomicrmw then no preceding load/load 5011 atomic/store/store atomic/atomicrmw/fence instruction can be 5012 moved after the release. 5013 - If a fence then same as store atomic, plus no following 5014 associated fence-paired-atomic can be moved before the 5015 fence. 5016 acq_rel Same constraints as both acquire and release. 5017 seq_cst - If a load atomic then same constraints as acquire, plus no 5018 preceding sequentially consistent load atomic/store 5019 atomic/atomicrmw/fence instruction can be moved after the 5020 seq_cst. 5021 - If a store atomic then the same constraints as release, plus 5022 no following sequentially consistent load atomic/store 5023 atomic/atomicrmw/fence instruction can be moved before the 5024 seq_cst. 5025 - If an atomicrmw/fence then same constraints as acq_rel. 5026 ============ ============================================================== 5027 5028The code sequences used to implement the memory model are defined in the 5029following sections: 5030 5031* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 5032* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 5033* :ref:`amdgpu-amdhsa-memory-model-gfx940` 5034* :ref:`amdgpu-amdhsa-memory-model-gfx10` 5035 5036.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 5037 5038Memory Model GFX6-GFX9 5039++++++++++++++++++++++ 5040 5041For GFX6-GFX9: 5042 5043* Each agent has multiple shader arrays (SA). 5044* Each SA has multiple compute units (CU). 5045* Each CU has multiple SIMDs that execute wavefronts. 5046* The wavefronts for a single work-group are executed in the same CU but may be 5047 executed by different SIMDs. 5048* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5049 executing on it. 5050* All LDS operations of a CU are performed as wavefront wide operations in a 5051 global order and involve no caching. Completion is reported to a wavefront in 5052 execution order. 5053* The LDS memory has multiple request queues shared by the SIMDs of a 5054 CU. Therefore, the LDS operations performed by different wavefronts of a 5055 work-group can be reordered relative to each other, which can result in 5056 reordering the visibility of vector memory operations with respect to LDS 5057 operations of other wavefronts in the same work-group. A ``s_waitcnt 5058 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5059 vector memory operations between wavefronts of a work-group, but not between 5060 operations performed by the same wavefront. 5061* The vector memory operations are performed as wavefront wide operations and 5062 completion is reported to a wavefront in execution order. The exception is 5063 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 5064 vector memory order if they access LDS memory, and out of LDS operation order 5065 if they access global memory. 5066* The vector memory operations access a single vector L1 cache shared by all 5067 SIMDs a CU. Therefore, no special action is required for coherence between the 5068 lanes of a single wavefront, or for coherence between wavefronts in the same 5069 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 5070 wavefronts executing in different work-groups as they may be executing on 5071 different CUs. 5072* The scalar memory operations access a scalar L1 cache shared by all wavefronts 5073 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 5074 scalar operations are used in a restricted way so do not impact the memory 5075 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 5076* The vector and scalar memory operations use an L2 cache shared by all CUs on 5077 the same agent. 5078* The L2 cache has independent channels to service disjoint ranges of virtual 5079 addresses. 5080* Each CU has a separate request queue per channel. Therefore, the vector and 5081 scalar memory operations performed by wavefronts executing in different 5082 work-groups (which may be executing on different CUs) of an agent can be 5083 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 5084 ensure synchronization between vector memory operations of different CUs. It 5085 ensures a previous vector memory operation has completed before executing a 5086 subsequent vector memory or LDS operation and so can be used to meet the 5087 requirements of acquire and release. 5088* The L2 cache can be kept coherent with other agents on some targets, or ranges 5089 of virtual addresses can be set up to bypass it to ensure system coherence. 5090 5091Scalar memory operations are only used to access memory that is proven to not 5092change during the execution of the kernel dispatch. This includes constant 5093address space and global address space for program scope ``const`` variables. 5094Therefore, the kernel machine code does not have to maintain the scalar cache to 5095ensure it is coherent with the vector caches. The scalar and vector caches are 5096invalidated between kernel dispatches by CP since constant address space data 5097may change between kernel dispatch executions. See 5098:ref:`amdgpu-amdhsa-memory-spaces`. 5099 5100The one exception is if scalar writes are used to spill SGPR registers. In this 5101case the AMDGPU backend ensures the memory location used to spill is never 5102accessed by vector memory operations at the same time. If scalar writes are used 5103then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5104return since the locations may be used for vector memory instructions by a 5105future wavefront that uses the same scratch area, or a function call that 5106creates a frame at the same address, respectively. There is no need for a 5107``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5108 5109For kernarg backing memory: 5110 5111* CP invalidates the L1 cache at the start of each kernel dispatch. 5112* On dGPU the kernarg backing memory is allocated in host memory accessed as 5113 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 5114 causes it to be treated as non-volatile and so is not invalidated by 5115 ``*_vol``. 5116* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 5117 and so the L2 cache will be coherent with the CPU and other agents. 5118 5119Scratch backing memory (which is used for the private address space) is accessed 5120with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 5121only accessed by a single thread, and is always write-before-read, there is 5122never a need to invalidate these entries from the L1 cache. Hence all cache 5123invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 5124 5125The code sequences used to implement the memory model for GFX6-GFX9 are defined 5126in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 5127 5128 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 5129 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 5130 5131 ============ ============ ============== ========== ================================ 5132 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5133 Ordering Sync Scope Address GFX6-GFX9 5134 Space 5135 ============ ============ ============== ========== ================================ 5136 **Non-Atomic** 5137 ------------------------------------------------------------------------------------ 5138 load *none* *none* - global - !volatile & !nontemporal 5139 - generic 5140 - private 1. buffer/global/flat_load 5141 - constant 5142 - !volatile & nontemporal 5143 5144 1. buffer/global/flat_load 5145 glc=1 slc=1 5146 5147 - volatile 5148 5149 1. buffer/global/flat_load 5150 glc=1 5151 2. s_waitcnt vmcnt(0) 5152 5153 - Must happen before 5154 any following volatile 5155 global/generic 5156 load/store. 5157 - Ensures that 5158 volatile 5159 operations to 5160 different 5161 addresses will not 5162 be reordered by 5163 hardware. 5164 5165 load *none* *none* - local 1. ds_load 5166 store *none* *none* - global - !volatile & !nontemporal 5167 - generic 5168 - private 1. buffer/global/flat_store 5169 - constant 5170 - !volatile & nontemporal 5171 5172 1. buffer/global/flat_store 5173 glc=1 slc=1 5174 5175 - volatile 5176 5177 1. buffer/global/flat_store 5178 2. s_waitcnt vmcnt(0) 5179 5180 - Must happen before 5181 any following volatile 5182 global/generic 5183 load/store. 5184 - Ensures that 5185 volatile 5186 operations to 5187 different 5188 addresses will not 5189 be reordered by 5190 hardware. 5191 5192 store *none* *none* - local 1. ds_store 5193 **Unordered Atomic** 5194 ------------------------------------------------------------------------------------ 5195 load atomic unordered *any* *any* *Same as non-atomic*. 5196 store atomic unordered *any* *any* *Same as non-atomic*. 5197 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5198 **Monotonic Atomic** 5199 ------------------------------------------------------------------------------------ 5200 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5201 - wavefront - local 5202 - workgroup - generic 5203 load atomic monotonic - agent - global 1. buffer/global/flat_load 5204 - system - generic glc=1 5205 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5206 - wavefront - generic 5207 - workgroup 5208 - agent 5209 - system 5210 store atomic monotonic - singlethread - local 1. ds_store 5211 - wavefront 5212 - workgroup 5213 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5214 - wavefront - generic 5215 - workgroup 5216 - agent 5217 - system 5218 atomicrmw monotonic - singlethread - local 1. ds_atomic 5219 - wavefront 5220 - workgroup 5221 **Acquire Atomic** 5222 ------------------------------------------------------------------------------------ 5223 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5224 - wavefront - local 5225 - generic 5226 load atomic acquire - workgroup - global 1. buffer/global_load 5227 load atomic acquire - workgroup - local 1. ds/flat_load 5228 - generic 2. s_waitcnt lgkmcnt(0) 5229 5230 - If OpenCL, omit. 5231 - Must happen before 5232 any following 5233 global/generic 5234 load/load 5235 atomic/store/store 5236 atomic/atomicrmw. 5237 - Ensures any 5238 following global 5239 data read is no 5240 older than a local load 5241 atomic value being 5242 acquired. 5243 5244 load atomic acquire - agent - global 1. buffer/global_load 5245 - system glc=1 5246 2. s_waitcnt vmcnt(0) 5247 5248 - Must happen before 5249 following 5250 buffer_wbinvl1_vol. 5251 - Ensures the load 5252 has completed 5253 before invalidating 5254 the cache. 5255 5256 3. buffer_wbinvl1_vol 5257 5258 - Must happen before 5259 any following 5260 global/generic 5261 load/load 5262 atomic/atomicrmw. 5263 - Ensures that 5264 following 5265 loads will not see 5266 stale global data. 5267 5268 load atomic acquire - agent - generic 1. flat_load glc=1 5269 - system 2. s_waitcnt vmcnt(0) & 5270 lgkmcnt(0) 5271 5272 - If OpenCL omit 5273 lgkmcnt(0). 5274 - Must happen before 5275 following 5276 buffer_wbinvl1_vol. 5277 - Ensures the flat_load 5278 has completed 5279 before invalidating 5280 the cache. 5281 5282 3. buffer_wbinvl1_vol 5283 5284 - Must happen before 5285 any following 5286 global/generic 5287 load/load 5288 atomic/atomicrmw. 5289 - Ensures that 5290 following loads 5291 will not see stale 5292 global data. 5293 5294 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5295 - wavefront - local 5296 - generic 5297 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5298 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5299 - generic 2. s_waitcnt lgkmcnt(0) 5300 5301 - If OpenCL, omit. 5302 - Must happen before 5303 any following 5304 global/generic 5305 load/load 5306 atomic/store/store 5307 atomic/atomicrmw. 5308 - Ensures any 5309 following global 5310 data read is no 5311 older than a local 5312 atomicrmw value 5313 being acquired. 5314 5315 atomicrmw acquire - agent - global 1. buffer/global_atomic 5316 - system 2. s_waitcnt vmcnt(0) 5317 5318 - Must happen before 5319 following 5320 buffer_wbinvl1_vol. 5321 - Ensures the 5322 atomicrmw has 5323 completed before 5324 invalidating the 5325 cache. 5326 5327 3. buffer_wbinvl1_vol 5328 5329 - Must happen before 5330 any following 5331 global/generic 5332 load/load 5333 atomic/atomicrmw. 5334 - Ensures that 5335 following loads 5336 will not see stale 5337 global data. 5338 5339 atomicrmw acquire - agent - generic 1. flat_atomic 5340 - system 2. s_waitcnt vmcnt(0) & 5341 lgkmcnt(0) 5342 5343 - If OpenCL, omit 5344 lgkmcnt(0). 5345 - Must happen before 5346 following 5347 buffer_wbinvl1_vol. 5348 - Ensures the 5349 atomicrmw has 5350 completed before 5351 invalidating the 5352 cache. 5353 5354 3. buffer_wbinvl1_vol 5355 5356 - Must happen before 5357 any following 5358 global/generic 5359 load/load 5360 atomic/atomicrmw. 5361 - Ensures that 5362 following loads 5363 will not see stale 5364 global data. 5365 5366 fence acquire - singlethread *none* *none* 5367 - wavefront 5368 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5369 5370 - If OpenCL and 5371 address space is 5372 not generic, omit. 5373 - However, since LLVM 5374 currently has no 5375 address space on 5376 the fence need to 5377 conservatively 5378 always generate. If 5379 fence had an 5380 address space then 5381 set to address 5382 space of OpenCL 5383 fence flag, or to 5384 generic if both 5385 local and global 5386 flags are 5387 specified. 5388 - Must happen after 5389 any preceding 5390 local/generic load 5391 atomic/atomicrmw 5392 with an equal or 5393 wider sync scope 5394 and memory ordering 5395 stronger than 5396 unordered (this is 5397 termed the 5398 fence-paired-atomic). 5399 - Must happen before 5400 any following 5401 global/generic 5402 load/load 5403 atomic/store/store 5404 atomic/atomicrmw. 5405 - Ensures any 5406 following global 5407 data read is no 5408 older than the 5409 value read by the 5410 fence-paired-atomic. 5411 5412 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5413 - system vmcnt(0) 5414 5415 - If OpenCL and 5416 address space is 5417 not generic, omit 5418 lgkmcnt(0). 5419 - However, since LLVM 5420 currently has no 5421 address space on 5422 the fence need to 5423 conservatively 5424 always generate 5425 (see comment for 5426 previous fence). 5427 - Could be split into 5428 separate s_waitcnt 5429 vmcnt(0) and 5430 s_waitcnt 5431 lgkmcnt(0) to allow 5432 them to be 5433 independently moved 5434 according to the 5435 following rules. 5436 - s_waitcnt vmcnt(0) 5437 must happen after 5438 any preceding 5439 global/generic load 5440 atomic/atomicrmw 5441 with an equal or 5442 wider sync scope 5443 and memory ordering 5444 stronger than 5445 unordered (this is 5446 termed the 5447 fence-paired-atomic). 5448 - s_waitcnt lgkmcnt(0) 5449 must happen after 5450 any preceding 5451 local/generic load 5452 atomic/atomicrmw 5453 with an equal or 5454 wider sync scope 5455 and memory ordering 5456 stronger than 5457 unordered (this is 5458 termed the 5459 fence-paired-atomic). 5460 - Must happen before 5461 the following 5462 buffer_wbinvl1_vol. 5463 - Ensures that the 5464 fence-paired atomic 5465 has completed 5466 before invalidating 5467 the 5468 cache. Therefore 5469 any following 5470 locations read must 5471 be no older than 5472 the value read by 5473 the 5474 fence-paired-atomic. 5475 5476 2. buffer_wbinvl1_vol 5477 5478 - Must happen before any 5479 following global/generic 5480 load/load 5481 atomic/store/store 5482 atomic/atomicrmw. 5483 - Ensures that 5484 following loads 5485 will not see stale 5486 global data. 5487 5488 **Release Atomic** 5489 ------------------------------------------------------------------------------------ 5490 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5491 - wavefront - local 5492 - generic 5493 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5494 - generic 5495 - If OpenCL, omit. 5496 - Must happen after 5497 any preceding 5498 local/generic 5499 load/store/load 5500 atomic/store 5501 atomic/atomicrmw. 5502 - Must happen before 5503 the following 5504 store. 5505 - Ensures that all 5506 memory operations 5507 to local have 5508 completed before 5509 performing the 5510 store that is being 5511 released. 5512 5513 2. buffer/global/flat_store 5514 store atomic release - workgroup - local 1. ds_store 5515 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5516 - system - generic vmcnt(0) 5517 5518 - If OpenCL and 5519 address space is 5520 not generic, omit 5521 lgkmcnt(0). 5522 - Could be split into 5523 separate s_waitcnt 5524 vmcnt(0) and 5525 s_waitcnt 5526 lgkmcnt(0) to allow 5527 them to be 5528 independently moved 5529 according to the 5530 following rules. 5531 - s_waitcnt vmcnt(0) 5532 must happen after 5533 any preceding 5534 global/generic 5535 load/store/load 5536 atomic/store 5537 atomic/atomicrmw. 5538 - s_waitcnt lgkmcnt(0) 5539 must happen after 5540 any preceding 5541 local/generic 5542 load/store/load 5543 atomic/store 5544 atomic/atomicrmw. 5545 - Must happen before 5546 the following 5547 store. 5548 - Ensures that all 5549 memory operations 5550 to memory have 5551 completed before 5552 performing the 5553 store that is being 5554 released. 5555 5556 2. buffer/global/flat_store 5557 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5558 - wavefront - local 5559 - generic 5560 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5561 - generic 5562 - If OpenCL, omit. 5563 - Must happen after 5564 any preceding 5565 local/generic 5566 load/store/load 5567 atomic/store 5568 atomic/atomicrmw. 5569 - Must happen before 5570 the following 5571 atomicrmw. 5572 - Ensures that all 5573 memory operations 5574 to local have 5575 completed before 5576 performing the 5577 atomicrmw that is 5578 being released. 5579 5580 2. buffer/global/flat_atomic 5581 atomicrmw release - workgroup - local 1. ds_atomic 5582 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5583 - system - generic vmcnt(0) 5584 5585 - If OpenCL, omit 5586 lgkmcnt(0). 5587 - Could be split into 5588 separate s_waitcnt 5589 vmcnt(0) and 5590 s_waitcnt 5591 lgkmcnt(0) to allow 5592 them to be 5593 independently moved 5594 according to the 5595 following rules. 5596 - s_waitcnt vmcnt(0) 5597 must happen after 5598 any preceding 5599 global/generic 5600 load/store/load 5601 atomic/store 5602 atomic/atomicrmw. 5603 - s_waitcnt lgkmcnt(0) 5604 must happen after 5605 any preceding 5606 local/generic 5607 load/store/load 5608 atomic/store 5609 atomic/atomicrmw. 5610 - Must happen before 5611 the following 5612 atomicrmw. 5613 - Ensures that all 5614 memory operations 5615 to global and local 5616 have completed 5617 before performing 5618 the atomicrmw that 5619 is being released. 5620 5621 2. buffer/global/flat_atomic 5622 fence release - singlethread *none* *none* 5623 - wavefront 5624 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5625 5626 - If OpenCL and 5627 address space is 5628 not generic, omit. 5629 - However, since LLVM 5630 currently has no 5631 address space on 5632 the fence need to 5633 conservatively 5634 always generate. If 5635 fence had an 5636 address space then 5637 set to address 5638 space of OpenCL 5639 fence flag, or to 5640 generic if both 5641 local and global 5642 flags are 5643 specified. 5644 - Must happen after 5645 any preceding 5646 local/generic 5647 load/load 5648 atomic/store/store 5649 atomic/atomicrmw. 5650 - Must happen before 5651 any following store 5652 atomic/atomicrmw 5653 with an equal or 5654 wider sync scope 5655 and memory ordering 5656 stronger than 5657 unordered (this is 5658 termed the 5659 fence-paired-atomic). 5660 - Ensures that all 5661 memory operations 5662 to local have 5663 completed before 5664 performing the 5665 following 5666 fence-paired-atomic. 5667 5668 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5669 - system vmcnt(0) 5670 5671 - If OpenCL and 5672 address space is 5673 not generic, omit 5674 lgkmcnt(0). 5675 - If OpenCL and 5676 address space is 5677 local, omit 5678 vmcnt(0). 5679 - However, since LLVM 5680 currently has no 5681 address space on 5682 the fence need to 5683 conservatively 5684 always generate. If 5685 fence had an 5686 address space then 5687 set to address 5688 space of OpenCL 5689 fence flag, or to 5690 generic if both 5691 local and global 5692 flags are 5693 specified. 5694 - Could be split into 5695 separate s_waitcnt 5696 vmcnt(0) and 5697 s_waitcnt 5698 lgkmcnt(0) to allow 5699 them to be 5700 independently moved 5701 according to the 5702 following rules. 5703 - s_waitcnt vmcnt(0) 5704 must happen after 5705 any preceding 5706 global/generic 5707 load/store/load 5708 atomic/store 5709 atomic/atomicrmw. 5710 - s_waitcnt lgkmcnt(0) 5711 must happen after 5712 any preceding 5713 local/generic 5714 load/store/load 5715 atomic/store 5716 atomic/atomicrmw. 5717 - Must happen before 5718 any following store 5719 atomic/atomicrmw 5720 with an equal or 5721 wider sync scope 5722 and memory ordering 5723 stronger than 5724 unordered (this is 5725 termed the 5726 fence-paired-atomic). 5727 - Ensures that all 5728 memory operations 5729 have 5730 completed before 5731 performing the 5732 following 5733 fence-paired-atomic. 5734 5735 **Acquire-Release Atomic** 5736 ------------------------------------------------------------------------------------ 5737 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5738 - wavefront - local 5739 - generic 5740 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5741 5742 - If OpenCL, omit. 5743 - Must happen after 5744 any preceding 5745 local/generic 5746 load/store/load 5747 atomic/store 5748 atomic/atomicrmw. 5749 - Must happen before 5750 the following 5751 atomicrmw. 5752 - Ensures that all 5753 memory operations 5754 to local have 5755 completed before 5756 performing the 5757 atomicrmw that is 5758 being released. 5759 5760 2. buffer/global_atomic 5761 5762 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5763 2. s_waitcnt lgkmcnt(0) 5764 5765 - If OpenCL, omit. 5766 - Must happen before 5767 any following 5768 global/generic 5769 load/load 5770 atomic/store/store 5771 atomic/atomicrmw. 5772 - Ensures any 5773 following global 5774 data read is no 5775 older than the local load 5776 atomic value being 5777 acquired. 5778 5779 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5780 5781 - If OpenCL, omit. 5782 - Must happen after 5783 any preceding 5784 local/generic 5785 load/store/load 5786 atomic/store 5787 atomic/atomicrmw. 5788 - Must happen before 5789 the following 5790 atomicrmw. 5791 - Ensures that all 5792 memory operations 5793 to local have 5794 completed before 5795 performing the 5796 atomicrmw that is 5797 being released. 5798 5799 2. flat_atomic 5800 3. s_waitcnt lgkmcnt(0) 5801 5802 - If OpenCL, omit. 5803 - Must happen before 5804 any following 5805 global/generic 5806 load/load 5807 atomic/store/store 5808 atomic/atomicrmw. 5809 - Ensures any 5810 following global 5811 data read is no 5812 older than a local load 5813 atomic value being 5814 acquired. 5815 5816 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5817 - system vmcnt(0) 5818 5819 - If OpenCL, omit 5820 lgkmcnt(0). 5821 - Could be split into 5822 separate s_waitcnt 5823 vmcnt(0) and 5824 s_waitcnt 5825 lgkmcnt(0) to allow 5826 them to be 5827 independently moved 5828 according to the 5829 following rules. 5830 - s_waitcnt vmcnt(0) 5831 must happen after 5832 any preceding 5833 global/generic 5834 load/store/load 5835 atomic/store 5836 atomic/atomicrmw. 5837 - s_waitcnt lgkmcnt(0) 5838 must happen after 5839 any preceding 5840 local/generic 5841 load/store/load 5842 atomic/store 5843 atomic/atomicrmw. 5844 - Must happen before 5845 the following 5846 atomicrmw. 5847 - Ensures that all 5848 memory operations 5849 to global have 5850 completed before 5851 performing the 5852 atomicrmw that is 5853 being released. 5854 5855 2. buffer/global_atomic 5856 3. s_waitcnt vmcnt(0) 5857 5858 - Must happen before 5859 following 5860 buffer_wbinvl1_vol. 5861 - Ensures the 5862 atomicrmw has 5863 completed before 5864 invalidating the 5865 cache. 5866 5867 4. buffer_wbinvl1_vol 5868 5869 - Must happen before 5870 any following 5871 global/generic 5872 load/load 5873 atomic/atomicrmw. 5874 - Ensures that 5875 following loads 5876 will not see stale 5877 global data. 5878 5879 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5880 - system vmcnt(0) 5881 5882 - If OpenCL, omit 5883 lgkmcnt(0). 5884 - Could be split into 5885 separate s_waitcnt 5886 vmcnt(0) and 5887 s_waitcnt 5888 lgkmcnt(0) to allow 5889 them to be 5890 independently moved 5891 according to the 5892 following rules. 5893 - s_waitcnt vmcnt(0) 5894 must happen after 5895 any preceding 5896 global/generic 5897 load/store/load 5898 atomic/store 5899 atomic/atomicrmw. 5900 - s_waitcnt lgkmcnt(0) 5901 must happen after 5902 any preceding 5903 local/generic 5904 load/store/load 5905 atomic/store 5906 atomic/atomicrmw. 5907 - Must happen before 5908 the following 5909 atomicrmw. 5910 - Ensures that all 5911 memory operations 5912 to global have 5913 completed before 5914 performing the 5915 atomicrmw that is 5916 being released. 5917 5918 2. flat_atomic 5919 3. s_waitcnt vmcnt(0) & 5920 lgkmcnt(0) 5921 5922 - If OpenCL, omit 5923 lgkmcnt(0). 5924 - Must happen before 5925 following 5926 buffer_wbinvl1_vol. 5927 - Ensures the 5928 atomicrmw has 5929 completed before 5930 invalidating the 5931 cache. 5932 5933 4. buffer_wbinvl1_vol 5934 5935 - Must happen before 5936 any following 5937 global/generic 5938 load/load 5939 atomic/atomicrmw. 5940 - Ensures that 5941 following loads 5942 will not see stale 5943 global data. 5944 5945 fence acq_rel - singlethread *none* *none* 5946 - wavefront 5947 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5948 5949 - If OpenCL and 5950 address space is 5951 not generic, omit. 5952 - However, 5953 since LLVM 5954 currently has no 5955 address space on 5956 the fence need to 5957 conservatively 5958 always generate 5959 (see comment for 5960 previous fence). 5961 - Must happen after 5962 any preceding 5963 local/generic 5964 load/load 5965 atomic/store/store 5966 atomic/atomicrmw. 5967 - Must happen before 5968 any following 5969 global/generic 5970 load/load 5971 atomic/store/store 5972 atomic/atomicrmw. 5973 - Ensures that all 5974 memory operations 5975 to local have 5976 completed before 5977 performing any 5978 following global 5979 memory operations. 5980 - Ensures that the 5981 preceding 5982 local/generic load 5983 atomic/atomicrmw 5984 with an equal or 5985 wider sync scope 5986 and memory ordering 5987 stronger than 5988 unordered (this is 5989 termed the 5990 acquire-fence-paired-atomic) 5991 has completed 5992 before following 5993 global memory 5994 operations. This 5995 satisfies the 5996 requirements of 5997 acquire. 5998 - Ensures that all 5999 previous memory 6000 operations have 6001 completed before a 6002 following 6003 local/generic store 6004 atomic/atomicrmw 6005 with an equal or 6006 wider sync scope 6007 and memory ordering 6008 stronger than 6009 unordered (this is 6010 termed the 6011 release-fence-paired-atomic). 6012 This satisfies the 6013 requirements of 6014 release. 6015 6016 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 6017 - system vmcnt(0) 6018 6019 - If OpenCL and 6020 address space is 6021 not generic, omit 6022 lgkmcnt(0). 6023 - However, since LLVM 6024 currently has no 6025 address space on 6026 the fence need to 6027 conservatively 6028 always generate 6029 (see comment for 6030 previous fence). 6031 - Could be split into 6032 separate s_waitcnt 6033 vmcnt(0) and 6034 s_waitcnt 6035 lgkmcnt(0) to allow 6036 them to be 6037 independently moved 6038 according to the 6039 following rules. 6040 - s_waitcnt vmcnt(0) 6041 must happen after 6042 any preceding 6043 global/generic 6044 load/store/load 6045 atomic/store 6046 atomic/atomicrmw. 6047 - s_waitcnt lgkmcnt(0) 6048 must happen after 6049 any preceding 6050 local/generic 6051 load/store/load 6052 atomic/store 6053 atomic/atomicrmw. 6054 - Must happen before 6055 the following 6056 buffer_wbinvl1_vol. 6057 - Ensures that the 6058 preceding 6059 global/local/generic 6060 load 6061 atomic/atomicrmw 6062 with an equal or 6063 wider sync scope 6064 and memory ordering 6065 stronger than 6066 unordered (this is 6067 termed the 6068 acquire-fence-paired-atomic) 6069 has completed 6070 before invalidating 6071 the cache. This 6072 satisfies the 6073 requirements of 6074 acquire. 6075 - Ensures that all 6076 previous memory 6077 operations have 6078 completed before a 6079 following 6080 global/local/generic 6081 store 6082 atomic/atomicrmw 6083 with an equal or 6084 wider sync scope 6085 and memory ordering 6086 stronger than 6087 unordered (this is 6088 termed the 6089 release-fence-paired-atomic). 6090 This satisfies the 6091 requirements of 6092 release. 6093 6094 2. buffer_wbinvl1_vol 6095 6096 - Must happen before 6097 any following 6098 global/generic 6099 load/load 6100 atomic/store/store 6101 atomic/atomicrmw. 6102 - Ensures that 6103 following loads 6104 will not see stale 6105 global data. This 6106 satisfies the 6107 requirements of 6108 acquire. 6109 6110 **Sequential Consistent Atomic** 6111 ------------------------------------------------------------------------------------ 6112 load atomic seq_cst - singlethread - global *Same as corresponding 6113 - wavefront - local load atomic acquire, 6114 - generic except must generate 6115 all instructions even 6116 for OpenCL.* 6117 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 6118 - generic 6119 6120 - Must 6121 happen after 6122 preceding 6123 local/generic load 6124 atomic/store 6125 atomic/atomicrmw 6126 with memory 6127 ordering of seq_cst 6128 and with equal or 6129 wider sync scope. 6130 (Note that seq_cst 6131 fences have their 6132 own s_waitcnt 6133 lgkmcnt(0) and so do 6134 not need to be 6135 considered.) 6136 - Ensures any 6137 preceding 6138 sequential 6139 consistent local 6140 memory instructions 6141 have completed 6142 before executing 6143 this sequentially 6144 consistent 6145 instruction. This 6146 prevents reordering 6147 a seq_cst store 6148 followed by a 6149 seq_cst load. (Note 6150 that seq_cst is 6151 stronger than 6152 acquire/release as 6153 the reordering of 6154 load acquire 6155 followed by a store 6156 release is 6157 prevented by the 6158 s_waitcnt of 6159 the release, but 6160 there is nothing 6161 preventing a store 6162 release followed by 6163 load acquire from 6164 completing out of 6165 order. The s_waitcnt 6166 could be placed after 6167 seq_store or before 6168 the seq_load. We 6169 choose the load to 6170 make the s_waitcnt be 6171 as late as possible 6172 so that the store 6173 may have already 6174 completed.) 6175 6176 2. *Following 6177 instructions same as 6178 corresponding load 6179 atomic acquire, 6180 except must generate 6181 all instructions even 6182 for OpenCL.* 6183 load atomic seq_cst - workgroup - local *Same as corresponding 6184 load atomic acquire, 6185 except must generate 6186 all instructions even 6187 for OpenCL.* 6188 6189 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 6190 - system - generic vmcnt(0) 6191 6192 - Could be split into 6193 separate s_waitcnt 6194 vmcnt(0) 6195 and s_waitcnt 6196 lgkmcnt(0) to allow 6197 them to be 6198 independently moved 6199 according to the 6200 following rules. 6201 - s_waitcnt lgkmcnt(0) 6202 must happen after 6203 preceding 6204 global/generic load 6205 atomic/store 6206 atomic/atomicrmw 6207 with memory 6208 ordering of seq_cst 6209 and with equal or 6210 wider sync scope. 6211 (Note that seq_cst 6212 fences have their 6213 own s_waitcnt 6214 lgkmcnt(0) and so do 6215 not need to be 6216 considered.) 6217 - s_waitcnt vmcnt(0) 6218 must happen after 6219 preceding 6220 global/generic load 6221 atomic/store 6222 atomic/atomicrmw 6223 with memory 6224 ordering of seq_cst 6225 and with equal or 6226 wider sync scope. 6227 (Note that seq_cst 6228 fences have their 6229 own s_waitcnt 6230 vmcnt(0) and so do 6231 not need to be 6232 considered.) 6233 - Ensures any 6234 preceding 6235 sequential 6236 consistent global 6237 memory instructions 6238 have completed 6239 before executing 6240 this sequentially 6241 consistent 6242 instruction. This 6243 prevents reordering 6244 a seq_cst store 6245 followed by a 6246 seq_cst load. (Note 6247 that seq_cst is 6248 stronger than 6249 acquire/release as 6250 the reordering of 6251 load acquire 6252 followed by a store 6253 release is 6254 prevented by the 6255 s_waitcnt of 6256 the release, but 6257 there is nothing 6258 preventing a store 6259 release followed by 6260 load acquire from 6261 completing out of 6262 order. The s_waitcnt 6263 could be placed after 6264 seq_store or before 6265 the seq_load. We 6266 choose the load to 6267 make the s_waitcnt be 6268 as late as possible 6269 so that the store 6270 may have already 6271 completed.) 6272 6273 2. *Following 6274 instructions same as 6275 corresponding load 6276 atomic acquire, 6277 except must generate 6278 all instructions even 6279 for OpenCL.* 6280 store atomic seq_cst - singlethread - global *Same as corresponding 6281 - wavefront - local store atomic release, 6282 - workgroup - generic except must generate 6283 - agent all instructions even 6284 - system for OpenCL.* 6285 atomicrmw seq_cst - singlethread - global *Same as corresponding 6286 - wavefront - local atomicrmw acq_rel, 6287 - workgroup - generic except must generate 6288 - agent all instructions even 6289 - system for OpenCL.* 6290 fence seq_cst - singlethread *none* *Same as corresponding 6291 - wavefront fence acq_rel, 6292 - workgroup except must generate 6293 - agent all instructions even 6294 - system for OpenCL.* 6295 ============ ============ ============== ========== ================================ 6296 6297.. _amdgpu-amdhsa-memory-model-gfx90a: 6298 6299Memory Model GFX90A 6300+++++++++++++++++++ 6301 6302For GFX90A: 6303 6304* Each agent has multiple shader arrays (SA). 6305* Each SA has multiple compute units (CU). 6306* Each CU has multiple SIMDs that execute wavefronts. 6307* The wavefronts for a single work-group are executed in the same CU but may be 6308 executed by different SIMDs. The exception is when in tgsplit execution mode 6309 when the wavefronts may be executed by different SIMDs in different CUs. 6310* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6311 executing on it. The exception is when in tgsplit execution mode when no LDS 6312 is allocated as wavefronts of the same work-group can be in different CUs. 6313* All LDS operations of a CU are performed as wavefront wide operations in a 6314 global order and involve no caching. Completion is reported to a wavefront in 6315 execution order. 6316* The LDS memory has multiple request queues shared by the SIMDs of a 6317 CU. Therefore, the LDS operations performed by different wavefronts of a 6318 work-group can be reordered relative to each other, which can result in 6319 reordering the visibility of vector memory operations with respect to LDS 6320 operations of other wavefronts in the same work-group. A ``s_waitcnt 6321 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6322 vector memory operations between wavefronts of a work-group, but not between 6323 operations performed by the same wavefront. 6324* The vector memory operations are performed as wavefront wide operations and 6325 completion is reported to a wavefront in execution order. The exception is 6326 that ``flat_load/store/atomic`` instructions can report out of vector memory 6327 order if they access LDS memory, and out of LDS operation order if they access 6328 global memory. 6329* The vector memory operations access a single vector L1 cache shared by all 6330 SIMDs a CU. Therefore: 6331 6332 * No special action is required for coherence between the lanes of a single 6333 wavefront. 6334 6335 * No special action is required for coherence between wavefronts in the same 6336 work-group since they execute on the same CU. The exception is when in 6337 tgsplit execution mode as wavefronts of the same work-group can be in 6338 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6339 the following item. 6340 6341 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6342 executing in different work-groups as they may be executing on different 6343 CUs. 6344 6345* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6346 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6347 scalar operations are used in a restricted way so do not impact the memory 6348 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6349* The vector and scalar memory operations use an L2 cache shared by all CUs on 6350 the same agent. 6351 6352 * The L2 cache has independent channels to service disjoint ranges of virtual 6353 addresses. 6354 * Each CU has a separate request queue per channel. Therefore, the vector and 6355 scalar memory operations performed by wavefronts executing in different 6356 work-groups (which may be executing on different CUs), or the same 6357 work-group if executing in tgsplit mode, of an agent can be reordered 6358 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6359 synchronization between vector memory operations of different CUs. It 6360 ensures a previous vector memory operation has completed before executing a 6361 subsequent vector memory or LDS operation and so can be used to meet the 6362 requirements of acquire and release. 6363 * The L2 cache of one agent can be kept coherent with other agents by: 6364 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6365 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6366 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6367 6368 * Any local memory cache lines will be automatically invalidated by writes 6369 from CUs associated with other L2 caches, or writes from the CPU, due to 6370 the cache probe caused by coherent requests. Coherent requests are caused 6371 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6372 XGMI, and by PCIe requests that are configured to be coherent requests. 6373 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6374 Subsequent access from the GPU will automatically invalidate or writeback 6375 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6376 * Since all work-groups on the same agent share the same L2, no L2 6377 invalidation or writeback is required for coherence. 6378 * To ensure coherence of local and remote memory writes of work-groups in 6379 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6380 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6381 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6382 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6383 remote fine grain memory) bypasses the L2, so both will never result in 6384 dirty L2 cache lines. 6385 * To ensure coherence of local and remote memory reads of work-groups in 6386 different agents a ``buffer_invl2`` is required. It will invalidate L2 6387 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6388 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6389 coarse memory) cause local reads to be invalidated by remote writes with 6390 with the PTE C-bit so these cache lines are not invalidated. Note that 6391 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6392 never result in L2 cache lines that need to be invalidated. 6393 6394 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6395 MTYPE UC (uncached) which bypasses the L2. 6396 6397Scalar memory operations are only used to access memory that is proven to not 6398change during the execution of the kernel dispatch. This includes constant 6399address space and global address space for program scope ``const`` variables. 6400Therefore, the kernel machine code does not have to maintain the scalar cache to 6401ensure it is coherent with the vector caches. The scalar and vector caches are 6402invalidated between kernel dispatches by CP since constant address space data 6403may change between kernel dispatch executions. See 6404:ref:`amdgpu-amdhsa-memory-spaces`. 6405 6406The one exception is if scalar writes are used to spill SGPR registers. In this 6407case the AMDGPU backend ensures the memory location used to spill is never 6408accessed by vector memory operations at the same time. If scalar writes are used 6409then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6410return since the locations may be used for vector memory instructions by a 6411future wavefront that uses the same scratch area, or a function call that 6412creates a frame at the same address, respectively. There is no need for a 6413``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6414 6415For kernarg backing memory: 6416 6417* CP invalidates the L1 cache at the start of each kernel dispatch. 6418* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6419 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6420 cache. This also causes it to be treated as non-volatile and so is not 6421 invalidated by ``*_vol``. 6422* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6423 so the L2 cache will be coherent with the CPU and other agents. 6424 6425Scratch backing memory (which is used for the private address space) is accessed 6426with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6427only accessed by a single thread, and is always write-before-read, there is 6428never a need to invalidate these entries from the L1 cache. Hence all cache 6429invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6430 6431The code sequences used to implement the memory model for GFX90A are defined 6432in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6433 6434 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6435 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6436 6437 ============ ============ ============== ========== ================================ 6438 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6439 Ordering Sync Scope Address GFX90A 6440 Space 6441 ============ ============ ============== ========== ================================ 6442 **Non-Atomic** 6443 ------------------------------------------------------------------------------------ 6444 load *none* *none* - global - !volatile & !nontemporal 6445 - generic 6446 - private 1. buffer/global/flat_load 6447 - constant 6448 - !volatile & nontemporal 6449 6450 1. buffer/global/flat_load 6451 glc=1 slc=1 6452 6453 - volatile 6454 6455 1. buffer/global/flat_load 6456 glc=1 6457 2. s_waitcnt vmcnt(0) 6458 6459 - Must happen before 6460 any following volatile 6461 global/generic 6462 load/store. 6463 - Ensures that 6464 volatile 6465 operations to 6466 different 6467 addresses will not 6468 be reordered by 6469 hardware. 6470 6471 load *none* *none* - local 1. ds_load 6472 store *none* *none* - global - !volatile & !nontemporal 6473 - generic 6474 - private 1. buffer/global/flat_store 6475 - constant 6476 - !volatile & nontemporal 6477 6478 1. buffer/global/flat_store 6479 glc=1 slc=1 6480 6481 - volatile 6482 6483 1. buffer/global/flat_store 6484 2. s_waitcnt vmcnt(0) 6485 6486 - Must happen before 6487 any following volatile 6488 global/generic 6489 load/store. 6490 - Ensures that 6491 volatile 6492 operations to 6493 different 6494 addresses will not 6495 be reordered by 6496 hardware. 6497 6498 store *none* *none* - local 1. ds_store 6499 **Unordered Atomic** 6500 ------------------------------------------------------------------------------------ 6501 load atomic unordered *any* *any* *Same as non-atomic*. 6502 store atomic unordered *any* *any* *Same as non-atomic*. 6503 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6504 **Monotonic Atomic** 6505 ------------------------------------------------------------------------------------ 6506 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6507 - wavefront - generic 6508 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6509 - generic glc=1 6510 6511 - If not TgSplit execution 6512 mode, omit glc=1. 6513 6514 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6515 - wavefront local address space cannot 6516 - workgroup be used.* 6517 6518 1. ds_load 6519 load atomic monotonic - agent - global 1. buffer/global/flat_load 6520 - generic glc=1 6521 load atomic monotonic - system - global 1. buffer/global/flat_load 6522 - generic glc=1 6523 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6524 - wavefront - generic 6525 - workgroup 6526 - agent 6527 store atomic monotonic - system - global 1. buffer/global/flat_store 6528 - generic 6529 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6530 - wavefront local address space cannot 6531 - workgroup be used.* 6532 6533 1. ds_store 6534 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6535 - wavefront - generic 6536 - workgroup 6537 - agent 6538 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6539 - generic 6540 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6541 - wavefront local address space cannot 6542 - workgroup be used.* 6543 6544 1. ds_atomic 6545 **Acquire Atomic** 6546 ------------------------------------------------------------------------------------ 6547 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6548 - wavefront - local 6549 - generic 6550 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6551 6552 - If not TgSplit execution 6553 mode, omit glc=1. 6554 6555 2. s_waitcnt vmcnt(0) 6556 6557 - If not TgSplit execution 6558 mode, omit. 6559 - Must happen before the 6560 following buffer_wbinvl1_vol. 6561 6562 3. buffer_wbinvl1_vol 6563 6564 - If not TgSplit execution 6565 mode, omit. 6566 - Must happen before 6567 any following 6568 global/generic 6569 load/load 6570 atomic/store/store 6571 atomic/atomicrmw. 6572 - Ensures that 6573 following 6574 loads will not see 6575 stale data. 6576 6577 load atomic acquire - workgroup - local *If TgSplit execution mode, 6578 local address space cannot 6579 be used.* 6580 6581 1. ds_load 6582 2. s_waitcnt lgkmcnt(0) 6583 6584 - If OpenCL, omit. 6585 - Must happen before 6586 any following 6587 global/generic 6588 load/load 6589 atomic/store/store 6590 atomic/atomicrmw. 6591 - Ensures any 6592 following global 6593 data read is no 6594 older than the local load 6595 atomic value being 6596 acquired. 6597 6598 load atomic acquire - workgroup - generic 1. flat_load glc=1 6599 6600 - If not TgSplit execution 6601 mode, omit glc=1. 6602 6603 2. s_waitcnt lgkm/vmcnt(0) 6604 6605 - Use lgkmcnt(0) if not 6606 TgSplit execution mode 6607 and vmcnt(0) if TgSplit 6608 execution mode. 6609 - If OpenCL, omit lgkmcnt(0). 6610 - Must happen before 6611 the following 6612 buffer_wbinvl1_vol and any 6613 following global/generic 6614 load/load 6615 atomic/store/store 6616 atomic/atomicrmw. 6617 - Ensures any 6618 following global 6619 data read is no 6620 older than a local load 6621 atomic value being 6622 acquired. 6623 6624 3. buffer_wbinvl1_vol 6625 6626 - If not TgSplit execution 6627 mode, omit. 6628 - Ensures that 6629 following 6630 loads will not see 6631 stale data. 6632 6633 load atomic acquire - agent - global 1. buffer/global_load 6634 glc=1 6635 2. s_waitcnt vmcnt(0) 6636 6637 - Must happen before 6638 following 6639 buffer_wbinvl1_vol. 6640 - Ensures the load 6641 has completed 6642 before invalidating 6643 the cache. 6644 6645 3. buffer_wbinvl1_vol 6646 6647 - Must happen before 6648 any following 6649 global/generic 6650 load/load 6651 atomic/atomicrmw. 6652 - Ensures that 6653 following 6654 loads will not see 6655 stale global data. 6656 6657 load atomic acquire - system - global 1. buffer/global/flat_load 6658 glc=1 6659 2. s_waitcnt vmcnt(0) 6660 6661 - Must happen before 6662 following buffer_invl2 and 6663 buffer_wbinvl1_vol. 6664 - Ensures the load 6665 has completed 6666 before invalidating 6667 the cache. 6668 6669 3. buffer_invl2; 6670 buffer_wbinvl1_vol 6671 6672 - Must happen before 6673 any following 6674 global/generic 6675 load/load 6676 atomic/atomicrmw. 6677 - Ensures that 6678 following 6679 loads will not see 6680 stale L1 global data, 6681 nor see stale L2 MTYPE 6682 NC global data. 6683 MTYPE RW and CC memory will 6684 never be stale in L2 due to 6685 the memory probes. 6686 6687 load atomic acquire - agent - generic 1. flat_load glc=1 6688 2. s_waitcnt vmcnt(0) & 6689 lgkmcnt(0) 6690 6691 - If TgSplit execution mode, 6692 omit lgkmcnt(0). 6693 - If OpenCL omit 6694 lgkmcnt(0). 6695 - Must happen before 6696 following 6697 buffer_wbinvl1_vol. 6698 - Ensures the flat_load 6699 has completed 6700 before invalidating 6701 the cache. 6702 6703 3. buffer_wbinvl1_vol 6704 6705 - Must happen before 6706 any following 6707 global/generic 6708 load/load 6709 atomic/atomicrmw. 6710 - Ensures that 6711 following loads 6712 will not see stale 6713 global data. 6714 6715 load atomic acquire - system - generic 1. flat_load glc=1 6716 2. s_waitcnt vmcnt(0) & 6717 lgkmcnt(0) 6718 6719 - If TgSplit execution mode, 6720 omit lgkmcnt(0). 6721 - If OpenCL omit 6722 lgkmcnt(0). 6723 - Must happen before 6724 following 6725 buffer_invl2 and 6726 buffer_wbinvl1_vol. 6727 - Ensures the flat_load 6728 has completed 6729 before invalidating 6730 the caches. 6731 6732 3. buffer_invl2; 6733 buffer_wbinvl1_vol 6734 6735 - Must happen before 6736 any following 6737 global/generic 6738 load/load 6739 atomic/atomicrmw. 6740 - Ensures that 6741 following 6742 loads will not see 6743 stale L1 global data, 6744 nor see stale L2 MTYPE 6745 NC global data. 6746 MTYPE RW and CC memory will 6747 never be stale in L2 due to 6748 the memory probes. 6749 6750 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6751 - wavefront - generic 6752 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6753 - wavefront local address space cannot 6754 be used.* 6755 6756 1. ds_atomic 6757 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6758 2. s_waitcnt vmcnt(0) 6759 6760 - If not TgSplit execution 6761 mode, omit. 6762 - Must happen before the 6763 following buffer_wbinvl1_vol. 6764 - Ensures the atomicrmw 6765 has completed 6766 before invalidating 6767 the cache. 6768 6769 3. buffer_wbinvl1_vol 6770 6771 - If not TgSplit execution 6772 mode, omit. 6773 - Must happen before 6774 any following 6775 global/generic 6776 load/load 6777 atomic/atomicrmw. 6778 - Ensures that 6779 following loads 6780 will not see stale 6781 global data. 6782 6783 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6784 local address space cannot 6785 be used.* 6786 6787 1. ds_atomic 6788 2. s_waitcnt lgkmcnt(0) 6789 6790 - If OpenCL, omit. 6791 - Must happen before 6792 any following 6793 global/generic 6794 load/load 6795 atomic/store/store 6796 atomic/atomicrmw. 6797 - Ensures any 6798 following global 6799 data read is no 6800 older than the local 6801 atomicrmw value 6802 being acquired. 6803 6804 atomicrmw acquire - workgroup - generic 1. flat_atomic 6805 2. s_waitcnt lgkm/vmcnt(0) 6806 6807 - Use lgkmcnt(0) if not 6808 TgSplit execution mode 6809 and vmcnt(0) if TgSplit 6810 execution mode. 6811 - If OpenCL, omit lgkmcnt(0). 6812 - Must happen before 6813 the following 6814 buffer_wbinvl1_vol and 6815 any following 6816 global/generic 6817 load/load 6818 atomic/store/store 6819 atomic/atomicrmw. 6820 - Ensures any 6821 following global 6822 data read is no 6823 older than a local 6824 atomicrmw value 6825 being acquired. 6826 6827 3. buffer_wbinvl1_vol 6828 6829 - If not TgSplit execution 6830 mode, omit. 6831 - Ensures that 6832 following 6833 loads will not see 6834 stale data. 6835 6836 atomicrmw acquire - agent - global 1. buffer/global_atomic 6837 2. s_waitcnt vmcnt(0) 6838 6839 - Must happen before 6840 following 6841 buffer_wbinvl1_vol. 6842 - Ensures the 6843 atomicrmw has 6844 completed before 6845 invalidating the 6846 cache. 6847 6848 3. buffer_wbinvl1_vol 6849 6850 - Must happen before 6851 any following 6852 global/generic 6853 load/load 6854 atomic/atomicrmw. 6855 - Ensures that 6856 following loads 6857 will not see stale 6858 global data. 6859 6860 atomicrmw acquire - system - global 1. buffer/global_atomic 6861 2. s_waitcnt vmcnt(0) 6862 6863 - Must happen before 6864 following buffer_invl2 and 6865 buffer_wbinvl1_vol. 6866 - Ensures the 6867 atomicrmw has 6868 completed before 6869 invalidating the 6870 caches. 6871 6872 3. buffer_invl2; 6873 buffer_wbinvl1_vol 6874 6875 - Must happen before 6876 any following 6877 global/generic 6878 load/load 6879 atomic/atomicrmw. 6880 - Ensures that 6881 following 6882 loads will not see 6883 stale L1 global data, 6884 nor see stale L2 MTYPE 6885 NC global data. 6886 MTYPE RW and CC memory will 6887 never be stale in L2 due to 6888 the memory probes. 6889 6890 atomicrmw acquire - agent - generic 1. flat_atomic 6891 2. s_waitcnt vmcnt(0) & 6892 lgkmcnt(0) 6893 6894 - If TgSplit execution mode, 6895 omit lgkmcnt(0). 6896 - If OpenCL, omit 6897 lgkmcnt(0). 6898 - Must happen before 6899 following 6900 buffer_wbinvl1_vol. 6901 - Ensures the 6902 atomicrmw has 6903 completed before 6904 invalidating the 6905 cache. 6906 6907 3. buffer_wbinvl1_vol 6908 6909 - Must happen before 6910 any following 6911 global/generic 6912 load/load 6913 atomic/atomicrmw. 6914 - Ensures that 6915 following loads 6916 will not see stale 6917 global data. 6918 6919 atomicrmw acquire - system - generic 1. flat_atomic 6920 2. s_waitcnt vmcnt(0) & 6921 lgkmcnt(0) 6922 6923 - If TgSplit execution mode, 6924 omit lgkmcnt(0). 6925 - If OpenCL, omit 6926 lgkmcnt(0). 6927 - Must happen before 6928 following 6929 buffer_invl2 and 6930 buffer_wbinvl1_vol. 6931 - Ensures the 6932 atomicrmw has 6933 completed before 6934 invalidating the 6935 caches. 6936 6937 3. buffer_invl2; 6938 buffer_wbinvl1_vol 6939 6940 - Must happen before 6941 any following 6942 global/generic 6943 load/load 6944 atomic/atomicrmw. 6945 - Ensures that 6946 following 6947 loads will not see 6948 stale L1 global data, 6949 nor see stale L2 MTYPE 6950 NC global data. 6951 MTYPE RW and CC memory will 6952 never be stale in L2 due to 6953 the memory probes. 6954 6955 fence acquire - singlethread *none* *none* 6956 - wavefront 6957 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6958 6959 - Use lgkmcnt(0) if not 6960 TgSplit execution mode 6961 and vmcnt(0) if TgSplit 6962 execution mode. 6963 - If OpenCL and 6964 address space is 6965 not generic, omit 6966 lgkmcnt(0). 6967 - If OpenCL and 6968 address space is 6969 local, omit 6970 vmcnt(0). 6971 - However, since LLVM 6972 currently has no 6973 address space on 6974 the fence need to 6975 conservatively 6976 always generate. If 6977 fence had an 6978 address space then 6979 set to address 6980 space of OpenCL 6981 fence flag, or to 6982 generic if both 6983 local and global 6984 flags are 6985 specified. 6986 - s_waitcnt vmcnt(0) 6987 must happen after 6988 any preceding 6989 global/generic load 6990 atomic/ 6991 atomicrmw 6992 with an equal or 6993 wider sync scope 6994 and memory ordering 6995 stronger than 6996 unordered (this is 6997 termed the 6998 fence-paired-atomic). 6999 - s_waitcnt lgkmcnt(0) 7000 must happen after 7001 any preceding 7002 local/generic load 7003 atomic/atomicrmw 7004 with an equal or 7005 wider sync scope 7006 and memory ordering 7007 stronger than 7008 unordered (this is 7009 termed the 7010 fence-paired-atomic). 7011 - Must happen before 7012 the following 7013 buffer_wbinvl1_vol and 7014 any following 7015 global/generic 7016 load/load 7017 atomic/store/store 7018 atomic/atomicrmw. 7019 - Ensures any 7020 following global 7021 data read is no 7022 older than the 7023 value read by the 7024 fence-paired-atomic. 7025 7026 2. buffer_wbinvl1_vol 7027 7028 - If not TgSplit execution 7029 mode, omit. 7030 - Ensures that 7031 following 7032 loads will not see 7033 stale data. 7034 7035 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 7036 vmcnt(0) 7037 7038 - If TgSplit execution mode, 7039 omit lgkmcnt(0). 7040 - If OpenCL and 7041 address space is 7042 not generic, omit 7043 lgkmcnt(0). 7044 - However, since LLVM 7045 currently has no 7046 address space on 7047 the fence need to 7048 conservatively 7049 always generate 7050 (see comment for 7051 previous fence). 7052 - Could be split into 7053 separate s_waitcnt 7054 vmcnt(0) and 7055 s_waitcnt 7056 lgkmcnt(0) to allow 7057 them to be 7058 independently moved 7059 according to the 7060 following rules. 7061 - s_waitcnt vmcnt(0) 7062 must happen after 7063 any preceding 7064 global/generic load 7065 atomic/atomicrmw 7066 with an equal or 7067 wider sync scope 7068 and memory ordering 7069 stronger than 7070 unordered (this is 7071 termed the 7072 fence-paired-atomic). 7073 - s_waitcnt lgkmcnt(0) 7074 must happen after 7075 any preceding 7076 local/generic load 7077 atomic/atomicrmw 7078 with an equal or 7079 wider sync scope 7080 and memory ordering 7081 stronger than 7082 unordered (this is 7083 termed the 7084 fence-paired-atomic). 7085 - Must happen before 7086 the following 7087 buffer_wbinvl1_vol. 7088 - Ensures that the 7089 fence-paired atomic 7090 has completed 7091 before invalidating 7092 the 7093 cache. Therefore 7094 any following 7095 locations read must 7096 be no older than 7097 the value read by 7098 the 7099 fence-paired-atomic. 7100 7101 2. buffer_wbinvl1_vol 7102 7103 - Must happen before any 7104 following global/generic 7105 load/load 7106 atomic/store/store 7107 atomic/atomicrmw. 7108 - Ensures that 7109 following loads 7110 will not see stale 7111 global data. 7112 7113 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 7114 vmcnt(0) 7115 7116 - If TgSplit execution mode, 7117 omit lgkmcnt(0). 7118 - If OpenCL and 7119 address space is 7120 not generic, omit 7121 lgkmcnt(0). 7122 - However, since LLVM 7123 currently has no 7124 address space on 7125 the fence need to 7126 conservatively 7127 always generate 7128 (see comment for 7129 previous fence). 7130 - Could be split into 7131 separate s_waitcnt 7132 vmcnt(0) and 7133 s_waitcnt 7134 lgkmcnt(0) to allow 7135 them to be 7136 independently moved 7137 according to the 7138 following rules. 7139 - s_waitcnt vmcnt(0) 7140 must happen after 7141 any preceding 7142 global/generic load 7143 atomic/atomicrmw 7144 with an equal or 7145 wider sync scope 7146 and memory ordering 7147 stronger than 7148 unordered (this is 7149 termed the 7150 fence-paired-atomic). 7151 - s_waitcnt lgkmcnt(0) 7152 must happen after 7153 any preceding 7154 local/generic load 7155 atomic/atomicrmw 7156 with an equal or 7157 wider sync scope 7158 and memory ordering 7159 stronger than 7160 unordered (this is 7161 termed the 7162 fence-paired-atomic). 7163 - Must happen before 7164 the following buffer_invl2 and 7165 buffer_wbinvl1_vol. 7166 - Ensures that the 7167 fence-paired atomic 7168 has completed 7169 before invalidating 7170 the 7171 cache. Therefore 7172 any following 7173 locations read must 7174 be no older than 7175 the value read by 7176 the 7177 fence-paired-atomic. 7178 7179 2. buffer_invl2; 7180 buffer_wbinvl1_vol 7181 7182 - Must happen before any 7183 following global/generic 7184 load/load 7185 atomic/store/store 7186 atomic/atomicrmw. 7187 - Ensures that 7188 following 7189 loads will not see 7190 stale L1 global data, 7191 nor see stale L2 MTYPE 7192 NC global data. 7193 MTYPE RW and CC memory will 7194 never be stale in L2 due to 7195 the memory probes. 7196 **Release Atomic** 7197 ------------------------------------------------------------------------------------ 7198 store atomic release - singlethread - global 1. buffer/global/flat_store 7199 - wavefront - generic 7200 store atomic release - singlethread - local *If TgSplit execution mode, 7201 - wavefront local address space cannot 7202 be used.* 7203 7204 1. ds_store 7205 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7206 - generic 7207 - Use lgkmcnt(0) if not 7208 TgSplit execution mode 7209 and vmcnt(0) if TgSplit 7210 execution mode. 7211 - If OpenCL, omit lgkmcnt(0). 7212 - s_waitcnt vmcnt(0) 7213 must happen after 7214 any preceding 7215 global/generic load/store/ 7216 load atomic/store atomic/ 7217 atomicrmw. 7218 - s_waitcnt lgkmcnt(0) 7219 must happen after 7220 any preceding 7221 local/generic 7222 load/store/load 7223 atomic/store 7224 atomic/atomicrmw. 7225 - Must happen before 7226 the following 7227 store. 7228 - Ensures that all 7229 memory operations 7230 have 7231 completed before 7232 performing the 7233 store that is being 7234 released. 7235 7236 2. buffer/global/flat_store 7237 store atomic release - workgroup - local *If TgSplit execution mode, 7238 local address space cannot 7239 be used.* 7240 7241 1. ds_store 7242 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7243 - generic vmcnt(0) 7244 7245 - If TgSplit execution mode, 7246 omit lgkmcnt(0). 7247 - If OpenCL and 7248 address space is 7249 not generic, omit 7250 lgkmcnt(0). 7251 - Could be split into 7252 separate s_waitcnt 7253 vmcnt(0) and 7254 s_waitcnt 7255 lgkmcnt(0) to allow 7256 them to be 7257 independently moved 7258 according to the 7259 following rules. 7260 - s_waitcnt vmcnt(0) 7261 must happen after 7262 any preceding 7263 global/generic 7264 load/store/load 7265 atomic/store 7266 atomic/atomicrmw. 7267 - s_waitcnt lgkmcnt(0) 7268 must happen after 7269 any preceding 7270 local/generic 7271 load/store/load 7272 atomic/store 7273 atomic/atomicrmw. 7274 - Must happen before 7275 the following 7276 store. 7277 - Ensures that all 7278 memory operations 7279 to memory have 7280 completed before 7281 performing the 7282 store that is being 7283 released. 7284 7285 2. buffer/global/flat_store 7286 store atomic release - system - global 1. buffer_wbl2 7287 - generic 7288 - Must happen before 7289 following s_waitcnt. 7290 - Performs L2 writeback to 7291 ensure previous 7292 global/generic 7293 store/atomicrmw are 7294 visible at system scope. 7295 7296 2. s_waitcnt lgkmcnt(0) & 7297 vmcnt(0) 7298 7299 - If TgSplit execution mode, 7300 omit lgkmcnt(0). 7301 - If OpenCL and 7302 address space is 7303 not generic, omit 7304 lgkmcnt(0). 7305 - Could be split into 7306 separate s_waitcnt 7307 vmcnt(0) and 7308 s_waitcnt 7309 lgkmcnt(0) to allow 7310 them to be 7311 independently moved 7312 according to the 7313 following rules. 7314 - s_waitcnt vmcnt(0) 7315 must happen after any 7316 preceding 7317 global/generic 7318 load/store/load 7319 atomic/store 7320 atomic/atomicrmw. 7321 - s_waitcnt lgkmcnt(0) 7322 must happen after any 7323 preceding 7324 local/generic 7325 load/store/load 7326 atomic/store 7327 atomic/atomicrmw. 7328 - Must happen before 7329 the following 7330 store. 7331 - Ensures that all 7332 memory operations 7333 to memory and the L2 7334 writeback have 7335 completed before 7336 performing the 7337 store that is being 7338 released. 7339 7340 3. buffer/global/flat_store 7341 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7342 - wavefront - generic 7343 atomicrmw release - singlethread - local *If TgSplit execution mode, 7344 - wavefront local address space cannot 7345 be used.* 7346 7347 1. ds_atomic 7348 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7349 - generic 7350 - Use lgkmcnt(0) if not 7351 TgSplit execution mode 7352 and vmcnt(0) if TgSplit 7353 execution mode. 7354 - If OpenCL, omit 7355 lgkmcnt(0). 7356 - s_waitcnt vmcnt(0) 7357 must happen after 7358 any preceding 7359 global/generic load/store/ 7360 load atomic/store atomic/ 7361 atomicrmw. 7362 - s_waitcnt lgkmcnt(0) 7363 must happen after 7364 any preceding 7365 local/generic 7366 load/store/load 7367 atomic/store 7368 atomic/atomicrmw. 7369 - Must happen before 7370 the following 7371 atomicrmw. 7372 - Ensures that all 7373 memory operations 7374 have 7375 completed before 7376 performing the 7377 atomicrmw that is 7378 being released. 7379 7380 2. buffer/global/flat_atomic 7381 atomicrmw release - workgroup - local *If TgSplit execution mode, 7382 local address space cannot 7383 be used.* 7384 7385 1. ds_atomic 7386 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7387 - generic vmcnt(0) 7388 7389 - If TgSplit execution mode, 7390 omit lgkmcnt(0). 7391 - If OpenCL, omit 7392 lgkmcnt(0). 7393 - Could be split into 7394 separate s_waitcnt 7395 vmcnt(0) and 7396 s_waitcnt 7397 lgkmcnt(0) to allow 7398 them to be 7399 independently moved 7400 according to the 7401 following rules. 7402 - s_waitcnt vmcnt(0) 7403 must happen after 7404 any preceding 7405 global/generic 7406 load/store/load 7407 atomic/store 7408 atomic/atomicrmw. 7409 - s_waitcnt lgkmcnt(0) 7410 must happen after 7411 any preceding 7412 local/generic 7413 load/store/load 7414 atomic/store 7415 atomic/atomicrmw. 7416 - Must happen before 7417 the following 7418 atomicrmw. 7419 - Ensures that all 7420 memory operations 7421 to global and local 7422 have completed 7423 before performing 7424 the atomicrmw that 7425 is being released. 7426 7427 2. buffer/global/flat_atomic 7428 atomicrmw release - system - global 1. buffer_wbl2 7429 - generic 7430 - Must happen before 7431 following s_waitcnt. 7432 - Performs L2 writeback to 7433 ensure previous 7434 global/generic 7435 store/atomicrmw are 7436 visible at system scope. 7437 7438 2. s_waitcnt lgkmcnt(0) & 7439 vmcnt(0) 7440 7441 - If TgSplit execution mode, 7442 omit lgkmcnt(0). 7443 - If OpenCL, omit 7444 lgkmcnt(0). 7445 - Could be split into 7446 separate s_waitcnt 7447 vmcnt(0) and 7448 s_waitcnt 7449 lgkmcnt(0) to allow 7450 them to be 7451 independently moved 7452 according to the 7453 following rules. 7454 - s_waitcnt vmcnt(0) 7455 must happen after 7456 any preceding 7457 global/generic 7458 load/store/load 7459 atomic/store 7460 atomic/atomicrmw. 7461 - s_waitcnt lgkmcnt(0) 7462 must happen after 7463 any preceding 7464 local/generic 7465 load/store/load 7466 atomic/store 7467 atomic/atomicrmw. 7468 - Must happen before 7469 the following 7470 atomicrmw. 7471 - Ensures that all 7472 memory operations 7473 to memory and the L2 7474 writeback have 7475 completed before 7476 performing the 7477 store that is being 7478 released. 7479 7480 3. buffer/global/flat_atomic 7481 fence release - singlethread *none* *none* 7482 - wavefront 7483 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7484 7485 - Use lgkmcnt(0) if not 7486 TgSplit execution mode 7487 and vmcnt(0) if TgSplit 7488 execution mode. 7489 - If OpenCL and 7490 address space is 7491 not generic, omit 7492 lgkmcnt(0). 7493 - If OpenCL and 7494 address space is 7495 local, omit 7496 vmcnt(0). 7497 - However, since LLVM 7498 currently has no 7499 address space on 7500 the fence need to 7501 conservatively 7502 always generate. If 7503 fence had an 7504 address space then 7505 set to address 7506 space of OpenCL 7507 fence flag, or to 7508 generic if both 7509 local and global 7510 flags are 7511 specified. 7512 - s_waitcnt vmcnt(0) 7513 must happen after 7514 any preceding 7515 global/generic 7516 load/store/ 7517 load atomic/store atomic/ 7518 atomicrmw. 7519 - s_waitcnt lgkmcnt(0) 7520 must happen after 7521 any preceding 7522 local/generic 7523 load/load 7524 atomic/store/store 7525 atomic/atomicrmw. 7526 - Must happen before 7527 any following store 7528 atomic/atomicrmw 7529 with an equal or 7530 wider sync scope 7531 and memory ordering 7532 stronger than 7533 unordered (this is 7534 termed the 7535 fence-paired-atomic). 7536 - Ensures that all 7537 memory operations 7538 have 7539 completed before 7540 performing the 7541 following 7542 fence-paired-atomic. 7543 7544 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7545 vmcnt(0) 7546 7547 - If TgSplit execution mode, 7548 omit lgkmcnt(0). 7549 - If OpenCL and 7550 address space is 7551 not generic, omit 7552 lgkmcnt(0). 7553 - If OpenCL and 7554 address space is 7555 local, omit 7556 vmcnt(0). 7557 - However, since LLVM 7558 currently has no 7559 address space on 7560 the fence need to 7561 conservatively 7562 always generate. If 7563 fence had an 7564 address space then 7565 set to address 7566 space of OpenCL 7567 fence flag, or to 7568 generic if both 7569 local and global 7570 flags are 7571 specified. 7572 - Could be split into 7573 separate s_waitcnt 7574 vmcnt(0) and 7575 s_waitcnt 7576 lgkmcnt(0) to allow 7577 them to be 7578 independently moved 7579 according to the 7580 following rules. 7581 - s_waitcnt vmcnt(0) 7582 must happen after 7583 any preceding 7584 global/generic 7585 load/store/load 7586 atomic/store 7587 atomic/atomicrmw. 7588 - s_waitcnt lgkmcnt(0) 7589 must happen after 7590 any preceding 7591 local/generic 7592 load/store/load 7593 atomic/store 7594 atomic/atomicrmw. 7595 - Must happen before 7596 any following store 7597 atomic/atomicrmw 7598 with an equal or 7599 wider sync scope 7600 and memory ordering 7601 stronger than 7602 unordered (this is 7603 termed the 7604 fence-paired-atomic). 7605 - Ensures that all 7606 memory operations 7607 have 7608 completed before 7609 performing the 7610 following 7611 fence-paired-atomic. 7612 7613 fence release - system *none* 1. buffer_wbl2 7614 7615 - If OpenCL and 7616 address space is 7617 local, omit. 7618 - Must happen before 7619 following s_waitcnt. 7620 - Performs L2 writeback to 7621 ensure previous 7622 global/generic 7623 store/atomicrmw are 7624 visible at system scope. 7625 7626 2. s_waitcnt lgkmcnt(0) & 7627 vmcnt(0) 7628 7629 - If TgSplit execution mode, 7630 omit lgkmcnt(0). 7631 - If OpenCL and 7632 address space is 7633 not generic, omit 7634 lgkmcnt(0). 7635 - If OpenCL and 7636 address space is 7637 local, omit 7638 vmcnt(0). 7639 - However, since LLVM 7640 currently has no 7641 address space on 7642 the fence need to 7643 conservatively 7644 always generate. If 7645 fence had an 7646 address space then 7647 set to address 7648 space of OpenCL 7649 fence flag, or to 7650 generic if both 7651 local and global 7652 flags are 7653 specified. 7654 - Could be split into 7655 separate s_waitcnt 7656 vmcnt(0) and 7657 s_waitcnt 7658 lgkmcnt(0) to allow 7659 them to be 7660 independently moved 7661 according to the 7662 following rules. 7663 - s_waitcnt vmcnt(0) 7664 must happen after 7665 any preceding 7666 global/generic 7667 load/store/load 7668 atomic/store 7669 atomic/atomicrmw. 7670 - s_waitcnt lgkmcnt(0) 7671 must happen after 7672 any preceding 7673 local/generic 7674 load/store/load 7675 atomic/store 7676 atomic/atomicrmw. 7677 - Must happen before 7678 any following store 7679 atomic/atomicrmw 7680 with an equal or 7681 wider sync scope 7682 and memory ordering 7683 stronger than 7684 unordered (this is 7685 termed the 7686 fence-paired-atomic). 7687 - Ensures that all 7688 memory operations 7689 have 7690 completed before 7691 performing the 7692 following 7693 fence-paired-atomic. 7694 7695 **Acquire-Release Atomic** 7696 ------------------------------------------------------------------------------------ 7697 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7698 - wavefront - generic 7699 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7700 - wavefront local address space cannot 7701 be used.* 7702 7703 1. ds_atomic 7704 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7705 7706 - Use lgkmcnt(0) if not 7707 TgSplit execution mode 7708 and vmcnt(0) if TgSplit 7709 execution mode. 7710 - If OpenCL, omit 7711 lgkmcnt(0). 7712 - Must happen after 7713 any preceding 7714 local/generic 7715 load/store/load 7716 atomic/store 7717 atomic/atomicrmw. 7718 - s_waitcnt vmcnt(0) 7719 must happen after 7720 any preceding 7721 global/generic load/store/ 7722 load atomic/store atomic/ 7723 atomicrmw. 7724 - s_waitcnt lgkmcnt(0) 7725 must happen after 7726 any preceding 7727 local/generic 7728 load/store/load 7729 atomic/store 7730 atomic/atomicrmw. 7731 - Must happen before 7732 the following 7733 atomicrmw. 7734 - Ensures that all 7735 memory operations 7736 have 7737 completed before 7738 performing the 7739 atomicrmw that is 7740 being released. 7741 7742 2. buffer/global_atomic 7743 3. s_waitcnt vmcnt(0) 7744 7745 - If not TgSplit execution 7746 mode, omit. 7747 - Must happen before 7748 the following 7749 buffer_wbinvl1_vol. 7750 - Ensures any 7751 following global 7752 data read is no 7753 older than the 7754 atomicrmw value 7755 being acquired. 7756 7757 4. buffer_wbinvl1_vol 7758 7759 - If not TgSplit execution 7760 mode, omit. 7761 - Ensures that 7762 following 7763 loads will not see 7764 stale data. 7765 7766 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7767 local address space cannot 7768 be used.* 7769 7770 1. ds_atomic 7771 2. s_waitcnt lgkmcnt(0) 7772 7773 - If OpenCL, omit. 7774 - Must happen before 7775 any following 7776 global/generic 7777 load/load 7778 atomic/store/store 7779 atomic/atomicrmw. 7780 - Ensures any 7781 following global 7782 data read is no 7783 older than the local load 7784 atomic value being 7785 acquired. 7786 7787 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7788 7789 - Use lgkmcnt(0) if not 7790 TgSplit execution mode 7791 and vmcnt(0) if TgSplit 7792 execution mode. 7793 - If OpenCL, omit 7794 lgkmcnt(0). 7795 - s_waitcnt vmcnt(0) 7796 must happen after 7797 any preceding 7798 global/generic load/store/ 7799 load atomic/store atomic/ 7800 atomicrmw. 7801 - s_waitcnt lgkmcnt(0) 7802 must happen after 7803 any preceding 7804 local/generic 7805 load/store/load 7806 atomic/store 7807 atomic/atomicrmw. 7808 - Must happen before 7809 the following 7810 atomicrmw. 7811 - Ensures that all 7812 memory operations 7813 have 7814 completed before 7815 performing the 7816 atomicrmw that is 7817 being released. 7818 7819 2. flat_atomic 7820 3. s_waitcnt lgkmcnt(0) & 7821 vmcnt(0) 7822 7823 - If not TgSplit execution 7824 mode, omit vmcnt(0). 7825 - If OpenCL, omit 7826 lgkmcnt(0). 7827 - Must happen before 7828 the following 7829 buffer_wbinvl1_vol and 7830 any following 7831 global/generic 7832 load/load 7833 atomic/store/store 7834 atomic/atomicrmw. 7835 - Ensures any 7836 following global 7837 data read is no 7838 older than a local load 7839 atomic value being 7840 acquired. 7841 7842 3. buffer_wbinvl1_vol 7843 7844 - If not TgSplit execution 7845 mode, omit. 7846 - Ensures that 7847 following 7848 loads will not see 7849 stale data. 7850 7851 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7852 vmcnt(0) 7853 7854 - If TgSplit execution mode, 7855 omit lgkmcnt(0). 7856 - If OpenCL, omit 7857 lgkmcnt(0). 7858 - Could be split into 7859 separate s_waitcnt 7860 vmcnt(0) and 7861 s_waitcnt 7862 lgkmcnt(0) to allow 7863 them to be 7864 independently moved 7865 according to the 7866 following rules. 7867 - s_waitcnt vmcnt(0) 7868 must happen after 7869 any preceding 7870 global/generic 7871 load/store/load 7872 atomic/store 7873 atomic/atomicrmw. 7874 - s_waitcnt lgkmcnt(0) 7875 must happen after 7876 any preceding 7877 local/generic 7878 load/store/load 7879 atomic/store 7880 atomic/atomicrmw. 7881 - Must happen before 7882 the following 7883 atomicrmw. 7884 - Ensures that all 7885 memory operations 7886 to global have 7887 completed before 7888 performing the 7889 atomicrmw that is 7890 being released. 7891 7892 2. buffer/global_atomic 7893 3. s_waitcnt vmcnt(0) 7894 7895 - Must happen before 7896 following 7897 buffer_wbinvl1_vol. 7898 - Ensures the 7899 atomicrmw has 7900 completed before 7901 invalidating the 7902 cache. 7903 7904 4. buffer_wbinvl1_vol 7905 7906 - Must happen before 7907 any following 7908 global/generic 7909 load/load 7910 atomic/atomicrmw. 7911 - Ensures that 7912 following loads 7913 will not see stale 7914 global data. 7915 7916 atomicrmw acq_rel - system - global 1. buffer_wbl2 7917 7918 - Must happen before 7919 following s_waitcnt. 7920 - Performs L2 writeback to 7921 ensure previous 7922 global/generic 7923 store/atomicrmw are 7924 visible at system scope. 7925 7926 2. s_waitcnt lgkmcnt(0) & 7927 vmcnt(0) 7928 7929 - If TgSplit execution mode, 7930 omit lgkmcnt(0). 7931 - If OpenCL, omit 7932 lgkmcnt(0). 7933 - Could be split into 7934 separate s_waitcnt 7935 vmcnt(0) and 7936 s_waitcnt 7937 lgkmcnt(0) to allow 7938 them to be 7939 independently moved 7940 according to the 7941 following rules. 7942 - s_waitcnt vmcnt(0) 7943 must happen after 7944 any preceding 7945 global/generic 7946 load/store/load 7947 atomic/store 7948 atomic/atomicrmw. 7949 - s_waitcnt lgkmcnt(0) 7950 must happen after 7951 any preceding 7952 local/generic 7953 load/store/load 7954 atomic/store 7955 atomic/atomicrmw. 7956 - Must happen before 7957 the following 7958 atomicrmw. 7959 - Ensures that all 7960 memory operations 7961 to global and L2 writeback 7962 have completed before 7963 performing the 7964 atomicrmw that is 7965 being released. 7966 7967 3. buffer/global_atomic 7968 4. s_waitcnt vmcnt(0) 7969 7970 - Must happen before 7971 following buffer_invl2 and 7972 buffer_wbinvl1_vol. 7973 - Ensures the 7974 atomicrmw has 7975 completed before 7976 invalidating the 7977 caches. 7978 7979 5. buffer_invl2; 7980 buffer_wbinvl1_vol 7981 7982 - Must happen before 7983 any following 7984 global/generic 7985 load/load 7986 atomic/atomicrmw. 7987 - Ensures that 7988 following 7989 loads will not see 7990 stale L1 global data, 7991 nor see stale L2 MTYPE 7992 NC global data. 7993 MTYPE RW and CC memory will 7994 never be stale in L2 due to 7995 the memory probes. 7996 7997 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7998 vmcnt(0) 7999 8000 - If TgSplit execution mode, 8001 omit lgkmcnt(0). 8002 - If OpenCL, omit 8003 lgkmcnt(0). 8004 - Could be split into 8005 separate s_waitcnt 8006 vmcnt(0) and 8007 s_waitcnt 8008 lgkmcnt(0) to allow 8009 them to be 8010 independently moved 8011 according to the 8012 following rules. 8013 - s_waitcnt vmcnt(0) 8014 must happen after 8015 any preceding 8016 global/generic 8017 load/store/load 8018 atomic/store 8019 atomic/atomicrmw. 8020 - s_waitcnt lgkmcnt(0) 8021 must happen after 8022 any preceding 8023 local/generic 8024 load/store/load 8025 atomic/store 8026 atomic/atomicrmw. 8027 - Must happen before 8028 the following 8029 atomicrmw. 8030 - Ensures that all 8031 memory operations 8032 to global have 8033 completed before 8034 performing the 8035 atomicrmw that is 8036 being released. 8037 8038 2. flat_atomic 8039 3. s_waitcnt vmcnt(0) & 8040 lgkmcnt(0) 8041 8042 - If TgSplit execution mode, 8043 omit lgkmcnt(0). 8044 - If OpenCL, omit 8045 lgkmcnt(0). 8046 - Must happen before 8047 following 8048 buffer_wbinvl1_vol. 8049 - Ensures the 8050 atomicrmw has 8051 completed before 8052 invalidating the 8053 cache. 8054 8055 4. buffer_wbinvl1_vol 8056 8057 - Must happen before 8058 any following 8059 global/generic 8060 load/load 8061 atomic/atomicrmw. 8062 - Ensures that 8063 following loads 8064 will not see stale 8065 global data. 8066 8067 atomicrmw acq_rel - system - generic 1. buffer_wbl2 8068 8069 - Must happen before 8070 following s_waitcnt. 8071 - Performs L2 writeback to 8072 ensure previous 8073 global/generic 8074 store/atomicrmw are 8075 visible at system scope. 8076 8077 2. s_waitcnt lgkmcnt(0) & 8078 vmcnt(0) 8079 8080 - If TgSplit execution mode, 8081 omit lgkmcnt(0). 8082 - If OpenCL, omit 8083 lgkmcnt(0). 8084 - Could be split into 8085 separate s_waitcnt 8086 vmcnt(0) and 8087 s_waitcnt 8088 lgkmcnt(0) to allow 8089 them to be 8090 independently moved 8091 according to the 8092 following rules. 8093 - s_waitcnt vmcnt(0) 8094 must happen after 8095 any preceding 8096 global/generic 8097 load/store/load 8098 atomic/store 8099 atomic/atomicrmw. 8100 - s_waitcnt lgkmcnt(0) 8101 must happen after 8102 any preceding 8103 local/generic 8104 load/store/load 8105 atomic/store 8106 atomic/atomicrmw. 8107 - Must happen before 8108 the following 8109 atomicrmw. 8110 - Ensures that all 8111 memory operations 8112 to global and L2 writeback 8113 have completed before 8114 performing the 8115 atomicrmw that is 8116 being released. 8117 8118 3. flat_atomic 8119 4. s_waitcnt vmcnt(0) & 8120 lgkmcnt(0) 8121 8122 - If TgSplit execution mode, 8123 omit lgkmcnt(0). 8124 - If OpenCL, omit 8125 lgkmcnt(0). 8126 - Must happen before 8127 following buffer_invl2 and 8128 buffer_wbinvl1_vol. 8129 - Ensures the 8130 atomicrmw has 8131 completed before 8132 invalidating the 8133 caches. 8134 8135 5. buffer_invl2; 8136 buffer_wbinvl1_vol 8137 8138 - Must happen before 8139 any following 8140 global/generic 8141 load/load 8142 atomic/atomicrmw. 8143 - Ensures that 8144 following 8145 loads will not see 8146 stale L1 global data, 8147 nor see stale L2 MTYPE 8148 NC global data. 8149 MTYPE RW and CC memory will 8150 never be stale in L2 due to 8151 the memory probes. 8152 8153 fence acq_rel - singlethread *none* *none* 8154 - wavefront 8155 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 8156 8157 - Use lgkmcnt(0) if not 8158 TgSplit execution mode 8159 and vmcnt(0) if TgSplit 8160 execution mode. 8161 - If OpenCL and 8162 address space is 8163 not generic, omit 8164 lgkmcnt(0). 8165 - If OpenCL and 8166 address space is 8167 local, omit 8168 vmcnt(0). 8169 - However, 8170 since LLVM 8171 currently has no 8172 address space on 8173 the fence need to 8174 conservatively 8175 always generate 8176 (see comment for 8177 previous fence). 8178 - s_waitcnt vmcnt(0) 8179 must happen after 8180 any preceding 8181 global/generic 8182 load/store/ 8183 load atomic/store atomic/ 8184 atomicrmw. 8185 - s_waitcnt lgkmcnt(0) 8186 must happen after 8187 any preceding 8188 local/generic 8189 load/load 8190 atomic/store/store 8191 atomic/atomicrmw. 8192 - Must happen before 8193 any following 8194 global/generic 8195 load/load 8196 atomic/store/store 8197 atomic/atomicrmw. 8198 - Ensures that all 8199 memory operations 8200 have 8201 completed before 8202 performing any 8203 following global 8204 memory operations. 8205 - Ensures that the 8206 preceding 8207 local/generic load 8208 atomic/atomicrmw 8209 with an equal or 8210 wider sync scope 8211 and memory ordering 8212 stronger than 8213 unordered (this is 8214 termed the 8215 acquire-fence-paired-atomic) 8216 has completed 8217 before following 8218 global memory 8219 operations. This 8220 satisfies the 8221 requirements of 8222 acquire. 8223 - Ensures that all 8224 previous memory 8225 operations have 8226 completed before a 8227 following 8228 local/generic store 8229 atomic/atomicrmw 8230 with an equal or 8231 wider sync scope 8232 and memory ordering 8233 stronger than 8234 unordered (this is 8235 termed the 8236 release-fence-paired-atomic). 8237 This satisfies the 8238 requirements of 8239 release. 8240 - Must happen before 8241 the following 8242 buffer_wbinvl1_vol. 8243 - Ensures that the 8244 acquire-fence-paired 8245 atomic has completed 8246 before invalidating 8247 the 8248 cache. Therefore 8249 any following 8250 locations read must 8251 be no older than 8252 the value read by 8253 the 8254 acquire-fence-paired-atomic. 8255 8256 2. buffer_wbinvl1_vol 8257 8258 - If not TgSplit execution 8259 mode, omit. 8260 - Ensures that 8261 following 8262 loads will not see 8263 stale data. 8264 8265 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8266 vmcnt(0) 8267 8268 - If TgSplit execution mode, 8269 omit lgkmcnt(0). 8270 - If OpenCL and 8271 address space is 8272 not generic, omit 8273 lgkmcnt(0). 8274 - However, since LLVM 8275 currently has no 8276 address space on 8277 the fence need to 8278 conservatively 8279 always generate 8280 (see comment for 8281 previous fence). 8282 - Could be split into 8283 separate s_waitcnt 8284 vmcnt(0) and 8285 s_waitcnt 8286 lgkmcnt(0) to allow 8287 them to be 8288 independently moved 8289 according to the 8290 following rules. 8291 - s_waitcnt vmcnt(0) 8292 must happen after 8293 any preceding 8294 global/generic 8295 load/store/load 8296 atomic/store 8297 atomic/atomicrmw. 8298 - s_waitcnt lgkmcnt(0) 8299 must happen after 8300 any preceding 8301 local/generic 8302 load/store/load 8303 atomic/store 8304 atomic/atomicrmw. 8305 - Must happen before 8306 the following 8307 buffer_wbinvl1_vol. 8308 - Ensures that the 8309 preceding 8310 global/local/generic 8311 load 8312 atomic/atomicrmw 8313 with an equal or 8314 wider sync scope 8315 and memory ordering 8316 stronger than 8317 unordered (this is 8318 termed the 8319 acquire-fence-paired-atomic) 8320 has completed 8321 before invalidating 8322 the cache. This 8323 satisfies the 8324 requirements of 8325 acquire. 8326 - Ensures that all 8327 previous memory 8328 operations have 8329 completed before a 8330 following 8331 global/local/generic 8332 store 8333 atomic/atomicrmw 8334 with an equal or 8335 wider sync scope 8336 and memory ordering 8337 stronger than 8338 unordered (this is 8339 termed the 8340 release-fence-paired-atomic). 8341 This satisfies the 8342 requirements of 8343 release. 8344 8345 2. buffer_wbinvl1_vol 8346 8347 - Must happen before 8348 any following 8349 global/generic 8350 load/load 8351 atomic/store/store 8352 atomic/atomicrmw. 8353 - Ensures that 8354 following loads 8355 will not see stale 8356 global data. This 8357 satisfies the 8358 requirements of 8359 acquire. 8360 8361 fence acq_rel - system *none* 1. buffer_wbl2 8362 8363 - If OpenCL and 8364 address space is 8365 local, omit. 8366 - Must happen before 8367 following s_waitcnt. 8368 - Performs L2 writeback to 8369 ensure previous 8370 global/generic 8371 store/atomicrmw are 8372 visible at system scope. 8373 8374 2. s_waitcnt lgkmcnt(0) & 8375 vmcnt(0) 8376 8377 - If TgSplit execution mode, 8378 omit lgkmcnt(0). 8379 - If OpenCL and 8380 address space is 8381 not generic, omit 8382 lgkmcnt(0). 8383 - However, since LLVM 8384 currently has no 8385 address space on 8386 the fence need to 8387 conservatively 8388 always generate 8389 (see comment for 8390 previous fence). 8391 - Could be split into 8392 separate s_waitcnt 8393 vmcnt(0) and 8394 s_waitcnt 8395 lgkmcnt(0) to allow 8396 them to be 8397 independently moved 8398 according to the 8399 following rules. 8400 - s_waitcnt vmcnt(0) 8401 must happen after 8402 any preceding 8403 global/generic 8404 load/store/load 8405 atomic/store 8406 atomic/atomicrmw. 8407 - s_waitcnt lgkmcnt(0) 8408 must happen after 8409 any preceding 8410 local/generic 8411 load/store/load 8412 atomic/store 8413 atomic/atomicrmw. 8414 - Must happen before 8415 the following buffer_invl2 and 8416 buffer_wbinvl1_vol. 8417 - Ensures that the 8418 preceding 8419 global/local/generic 8420 load 8421 atomic/atomicrmw 8422 with an equal or 8423 wider sync scope 8424 and memory ordering 8425 stronger than 8426 unordered (this is 8427 termed the 8428 acquire-fence-paired-atomic) 8429 has completed 8430 before invalidating 8431 the cache. This 8432 satisfies the 8433 requirements of 8434 acquire. 8435 - Ensures that all 8436 previous memory 8437 operations have 8438 completed before a 8439 following 8440 global/local/generic 8441 store 8442 atomic/atomicrmw 8443 with an equal or 8444 wider sync scope 8445 and memory ordering 8446 stronger than 8447 unordered (this is 8448 termed the 8449 release-fence-paired-atomic). 8450 This satisfies the 8451 requirements of 8452 release. 8453 8454 3. buffer_invl2; 8455 buffer_wbinvl1_vol 8456 8457 - Must happen before 8458 any following 8459 global/generic 8460 load/load 8461 atomic/store/store 8462 atomic/atomicrmw. 8463 - Ensures that 8464 following 8465 loads will not see 8466 stale L1 global data, 8467 nor see stale L2 MTYPE 8468 NC global data. 8469 MTYPE RW and CC memory will 8470 never be stale in L2 due to 8471 the memory probes. 8472 8473 **Sequential Consistent Atomic** 8474 ------------------------------------------------------------------------------------ 8475 load atomic seq_cst - singlethread - global *Same as corresponding 8476 - wavefront - local load atomic acquire, 8477 - generic except must generate 8478 all instructions even 8479 for OpenCL.* 8480 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8481 - generic 8482 - Use lgkmcnt(0) if not 8483 TgSplit execution mode 8484 and vmcnt(0) if TgSplit 8485 execution mode. 8486 - s_waitcnt lgkmcnt(0) must 8487 happen after 8488 preceding 8489 local/generic load 8490 atomic/store 8491 atomic/atomicrmw 8492 with memory 8493 ordering of seq_cst 8494 and with equal or 8495 wider sync scope. 8496 (Note that seq_cst 8497 fences have their 8498 own s_waitcnt 8499 lgkmcnt(0) and so do 8500 not need to be 8501 considered.) 8502 - s_waitcnt vmcnt(0) 8503 must happen after 8504 preceding 8505 global/generic load 8506 atomic/store 8507 atomic/atomicrmw 8508 with memory 8509 ordering of seq_cst 8510 and with equal or 8511 wider sync scope. 8512 (Note that seq_cst 8513 fences have their 8514 own s_waitcnt 8515 vmcnt(0) and so do 8516 not need to be 8517 considered.) 8518 - Ensures any 8519 preceding 8520 sequential 8521 consistent global/local 8522 memory instructions 8523 have completed 8524 before executing 8525 this sequentially 8526 consistent 8527 instruction. This 8528 prevents reordering 8529 a seq_cst store 8530 followed by a 8531 seq_cst load. (Note 8532 that seq_cst is 8533 stronger than 8534 acquire/release as 8535 the reordering of 8536 load acquire 8537 followed by a store 8538 release is 8539 prevented by the 8540 s_waitcnt of 8541 the release, but 8542 there is nothing 8543 preventing a store 8544 release followed by 8545 load acquire from 8546 completing out of 8547 order. The s_waitcnt 8548 could be placed after 8549 seq_store or before 8550 the seq_load. We 8551 choose the load to 8552 make the s_waitcnt be 8553 as late as possible 8554 so that the store 8555 may have already 8556 completed.) 8557 8558 2. *Following 8559 instructions same as 8560 corresponding load 8561 atomic acquire, 8562 except must generate 8563 all instructions even 8564 for OpenCL.* 8565 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8566 local address space cannot 8567 be used.* 8568 8569 *Same as corresponding 8570 load atomic acquire, 8571 except must generate 8572 all instructions even 8573 for OpenCL.* 8574 8575 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8576 - system - generic vmcnt(0) 8577 8578 - If TgSplit execution mode, 8579 omit lgkmcnt(0). 8580 - Could be split into 8581 separate s_waitcnt 8582 vmcnt(0) 8583 and s_waitcnt 8584 lgkmcnt(0) to allow 8585 them to be 8586 independently moved 8587 according to the 8588 following rules. 8589 - s_waitcnt lgkmcnt(0) 8590 must happen after 8591 preceding 8592 global/generic load 8593 atomic/store 8594 atomic/atomicrmw 8595 with memory 8596 ordering of seq_cst 8597 and with equal or 8598 wider sync scope. 8599 (Note that seq_cst 8600 fences have their 8601 own s_waitcnt 8602 lgkmcnt(0) and so do 8603 not need to be 8604 considered.) 8605 - s_waitcnt vmcnt(0) 8606 must happen after 8607 preceding 8608 global/generic load 8609 atomic/store 8610 atomic/atomicrmw 8611 with memory 8612 ordering of seq_cst 8613 and with equal or 8614 wider sync scope. 8615 (Note that seq_cst 8616 fences have their 8617 own s_waitcnt 8618 vmcnt(0) and so do 8619 not need to be 8620 considered.) 8621 - Ensures any 8622 preceding 8623 sequential 8624 consistent global 8625 memory instructions 8626 have completed 8627 before executing 8628 this sequentially 8629 consistent 8630 instruction. This 8631 prevents reordering 8632 a seq_cst store 8633 followed by a 8634 seq_cst load. (Note 8635 that seq_cst is 8636 stronger than 8637 acquire/release as 8638 the reordering of 8639 load acquire 8640 followed by a store 8641 release is 8642 prevented by the 8643 s_waitcnt of 8644 the release, but 8645 there is nothing 8646 preventing a store 8647 release followed by 8648 load acquire from 8649 completing out of 8650 order. The s_waitcnt 8651 could be placed after 8652 seq_store or before 8653 the seq_load. We 8654 choose the load to 8655 make the s_waitcnt be 8656 as late as possible 8657 so that the store 8658 may have already 8659 completed.) 8660 8661 2. *Following 8662 instructions same as 8663 corresponding load 8664 atomic acquire, 8665 except must generate 8666 all instructions even 8667 for OpenCL.* 8668 store atomic seq_cst - singlethread - global *Same as corresponding 8669 - wavefront - local store atomic release, 8670 - workgroup - generic except must generate 8671 - agent all instructions even 8672 - system for OpenCL.* 8673 atomicrmw seq_cst - singlethread - global *Same as corresponding 8674 - wavefront - local atomicrmw acq_rel, 8675 - workgroup - generic except must generate 8676 - agent all instructions even 8677 - system for OpenCL.* 8678 fence seq_cst - singlethread *none* *Same as corresponding 8679 - wavefront fence acq_rel, 8680 - workgroup except must generate 8681 - agent all instructions even 8682 - system for OpenCL.* 8683 ============ ============ ============== ========== ================================ 8684 8685.. _amdgpu-amdhsa-memory-model-gfx940: 8686 8687Memory Model GFX940 8688+++++++++++++++++++ 8689 8690For GFX940: 8691 8692* Each agent has multiple shader arrays (SA). 8693* Each SA has multiple compute units (CU). 8694* Each CU has multiple SIMDs that execute wavefronts. 8695* The wavefronts for a single work-group are executed in the same CU but may be 8696 executed by different SIMDs. The exception is when in tgsplit execution mode 8697 when the wavefronts may be executed by different SIMDs in different CUs. 8698* Each CU has a single LDS memory shared by the wavefronts of the work-groups 8699 executing on it. The exception is when in tgsplit execution mode when no LDS 8700 is allocated as wavefronts of the same work-group can be in different CUs. 8701* All LDS operations of a CU are performed as wavefront wide operations in a 8702 global order and involve no caching. Completion is reported to a wavefront in 8703 execution order. 8704* The LDS memory has multiple request queues shared by the SIMDs of a 8705 CU. Therefore, the LDS operations performed by different wavefronts of a 8706 work-group can be reordered relative to each other, which can result in 8707 reordering the visibility of vector memory operations with respect to LDS 8708 operations of other wavefronts in the same work-group. A ``s_waitcnt 8709 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8710 vector memory operations between wavefronts of a work-group, but not between 8711 operations performed by the same wavefront. 8712* The vector memory operations are performed as wavefront wide operations and 8713 completion is reported to a wavefront in execution order. The exception is 8714 that ``flat_load/store/atomic`` instructions can report out of vector memory 8715 order if they access LDS memory, and out of LDS operation order if they access 8716 global memory. 8717* The vector memory operations access a single vector L1 cache shared by all 8718 SIMDs a CU. Therefore: 8719 8720 * No special action is required for coherence between the lanes of a single 8721 wavefront. 8722 8723 * No special action is required for coherence between wavefronts in the same 8724 work-group since they execute on the same CU. The exception is when in 8725 tgsplit execution mode as wavefronts of the same work-group can be in 8726 different CUs and so a ``buffer_inv sc0`` is required which will invalidate 8727 the L1 cache. 8728 8729 * A ``buffer_inv sc0`` is required to invalidate the L1 cache for coherence 8730 between wavefronts executing in different work-groups as they may be 8731 executing on different CUs. 8732 8733 * Atomic read-modify-write instructions implicitly bypass the L1 cache. 8734 Therefore, they do not use the sc0 bit for coherence and instead use it to 8735 indicate if the instruction returns the original value being updated. They 8736 do use sc1 to indicate system or agent scope coherence. 8737 8738* The scalar memory operations access a scalar L1 cache shared by all wavefronts 8739 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 8740 scalar operations are used in a restricted way so do not impact the memory 8741 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 8742* The vector and scalar memory operations use an L2 cache. 8743 8744 * The gfx940 can be configured as a number of smaller agents with each having 8745 a single L2 shared by all CUs on the same agent, or as fewer (possibly one) 8746 larger agents with groups of CUs on each agent each sharing separate L2 8747 caches. 8748 * The L2 cache has independent channels to service disjoint ranges of virtual 8749 addresses. 8750 * Each CU has a separate request queue per channel for its associated L2. 8751 Therefore, the vector and scalar memory operations performed by wavefronts 8752 executing with different L1 caches and the same L2 cache can be reordered 8753 relative to each other. 8754 * A ``s_waitcnt vmcnt(0)`` is required to ensure synchronization between 8755 vector memory operations of different CUs. It ensures a previous vector 8756 memory operation has completed before executing a subsequent vector memory 8757 or LDS operation and so can be used to meet the requirements of acquire and 8758 release. 8759 * An L2 cache can be kept coherent with other L2 caches by using the MTYPE RW 8760 (read-write) for memory local to the L2, and MTYPE NC (non-coherent) with 8761 the PTE C-bit set for memory not local to the L2. 8762 8763 * Any local memory cache lines will be automatically invalidated by writes 8764 from CUs associated with other L2 caches, or writes from the CPU, due to 8765 the cache probe caused by the PTE C-bit. 8766 * XGMI accesses from the CPU to local memory may be cached on the CPU. 8767 Subsequent access from the GPU will automatically invalidate or writeback 8768 the CPU cache due to the L2 probe filter. 8769 * To ensure coherence of local memory writes of CUs with different L1 caches 8770 in the same agent a ``buffer_wbl2`` is required. It does nothing if the 8771 agent is configured to have a single L2, or will writeback dirty L2 cache 8772 lines if configured to have multiple L2 caches. 8773 * To ensure coherence of local memory writes of CUs in different agents a 8774 ``buffer_wbl2 sc1`` is required. It will writeback dirty L2 cache lines. 8775 * To ensure coherence of local memory reads of CUs with different L1 caches 8776 in the same agent a ``buffer_inv sc1`` is required. It does nothing if the 8777 agent is configured to have a single L2, or will invalidate non-local L2 8778 cache lines if configured to have multiple L2 caches. 8779 * To ensure coherence of local memory reads of CUs in different agents a 8780 ``buffer_inv sc0 sc1`` is required. It will invalidate non-local L2 cache 8781 lines if configured to have multiple L2 caches. 8782 8783 * PCIe access from the GPU to the CPU can be kept coherent by using the MTYPE 8784 UC (uncached) which bypasses the L2. 8785 8786Scalar memory operations are only used to access memory that is proven to not 8787change during the execution of the kernel dispatch. This includes constant 8788address space and global address space for program scope ``const`` variables. 8789Therefore, the kernel machine code does not have to maintain the scalar cache to 8790ensure it is coherent with the vector caches. The scalar and vector caches are 8791invalidated between kernel dispatches by CP since constant address space data 8792may change between kernel dispatch executions. See 8793:ref:`amdgpu-amdhsa-memory-spaces`. 8794 8795The one exception is if scalar writes are used to spill SGPR registers. In this 8796case the AMDGPU backend ensures the memory location used to spill is never 8797accessed by vector memory operations at the same time. If scalar writes are used 8798then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8799return since the locations may be used for vector memory instructions by a 8800future wavefront that uses the same scratch area, or a function call that 8801creates a frame at the same address, respectively. There is no need for a 8802``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8803 8804For kernarg backing memory: 8805 8806* CP invalidates the L1 cache at the start of each kernel dispatch. 8807* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 8808 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 8809 cache. This also causes it to be treated as non-volatile and so is not 8810 invalidated by ``*_vol``. 8811* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8812 so the L2 cache will be coherent with the CPU and other agents. 8813 8814Scratch backing memory (which is used for the private address space) is accessed 8815with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 8816only accessed by a single thread, and is always write-before-read, there is 8817never a need to invalidate these entries from the L1 cache. Hence all cache 8818invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 8819 8820The code sequences used to implement the memory model for GFX940 are defined 8821in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx940-table`. 8822 8823 .. table:: AMDHSA Memory Model Code Sequences GFX940 8824 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx940-table 8825 8826 ============ ============ ============== ========== ================================ 8827 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8828 Ordering Sync Scope Address GFX940 8829 Space 8830 ============ ============ ============== ========== ================================ 8831 **Non-Atomic** 8832 ------------------------------------------------------------------------------------ 8833 load *none* *none* - global - !volatile & !nontemporal 8834 - generic 8835 - private 1. buffer/global/flat_load 8836 - constant 8837 - !volatile & nontemporal 8838 8839 1. buffer/global/flat_load 8840 nt=1 8841 8842 - volatile 8843 8844 1. buffer/global/flat_load 8845 sc0=1 sc1=1 8846 2. s_waitcnt vmcnt(0) 8847 8848 - Must happen before 8849 any following volatile 8850 global/generic 8851 load/store. 8852 - Ensures that 8853 volatile 8854 operations to 8855 different 8856 addresses will not 8857 be reordered by 8858 hardware. 8859 8860 load *none* *none* - local 1. ds_load 8861 store *none* *none* - global - !volatile & !nontemporal 8862 - generic 8863 - private 1. buffer/global/flat_store 8864 - constant 8865 - !volatile & nontemporal 8866 8867 1. buffer/global/flat_store 8868 nt=1 8869 8870 - volatile 8871 8872 1. buffer/global/flat_store 8873 sc0=1 sc1=1 8874 2. s_waitcnt vmcnt(0) 8875 8876 - Must happen before 8877 any following volatile 8878 global/generic 8879 load/store. 8880 - Ensures that 8881 volatile 8882 operations to 8883 different 8884 addresses will not 8885 be reordered by 8886 hardware. 8887 8888 store *none* *none* - local 1. ds_store 8889 **Unordered Atomic** 8890 ------------------------------------------------------------------------------------ 8891 load atomic unordered *any* *any* *Same as non-atomic*. 8892 store atomic unordered *any* *any* *Same as non-atomic*. 8893 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8894 **Monotonic Atomic** 8895 ------------------------------------------------------------------------------------ 8896 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8897 - wavefront - generic 8898 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8899 - generic sc0=1 8900 load atomic monotonic - singlethread - local *If TgSplit execution mode, 8901 - wavefront local address space cannot 8902 - workgroup be used.* 8903 8904 1. ds_load 8905 load atomic monotonic - agent - global 1. buffer/global/flat_load 8906 - generic sc1=1 8907 load atomic monotonic - system - global 1. buffer/global/flat_load 8908 - generic sc0=1 sc1=1 8909 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8910 - wavefront - generic 8911 store atomic monotonic - workgroup - global 1. buffer/global/flat_store 8912 - generic sc0=1 8913 store atomic monotonic - agent - global 1. buffer/global/flat_store 8914 - generic sc1=1 8915 store atomic monotonic - system - global 1. buffer/global/flat_store 8916 - generic sc0=1 sc1=1 8917 store atomic monotonic - singlethread - local *If TgSplit execution mode, 8918 - wavefront local address space cannot 8919 - workgroup be used.* 8920 8921 1. ds_store 8922 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8923 - wavefront - generic 8924 - workgroup 8925 - agent 8926 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 8927 - generic sc1=1 8928 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 8929 - wavefront local address space cannot 8930 - workgroup be used.* 8931 8932 1. ds_atomic 8933 **Acquire Atomic** 8934 ------------------------------------------------------------------------------------ 8935 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8936 - wavefront - local 8937 - generic 8938 load atomic acquire - workgroup - global 1. buffer/global_load sc0=1 8939 2. s_waitcnt vmcnt(0) 8940 8941 - If not TgSplit execution 8942 mode, omit. 8943 - Must happen before the 8944 following buffer_inv. 8945 8946 3. buffer_inv sc0=1 8947 8948 - If not TgSplit execution 8949 mode, omit. 8950 - Must happen before 8951 any following 8952 global/generic 8953 load/load 8954 atomic/store/store 8955 atomic/atomicrmw. 8956 - Ensures that 8957 following 8958 loads will not see 8959 stale data. 8960 8961 load atomic acquire - workgroup - local *If TgSplit execution mode, 8962 local address space cannot 8963 be used.* 8964 8965 1. ds_load 8966 2. s_waitcnt lgkmcnt(0) 8967 8968 - If OpenCL, omit. 8969 - Must happen before 8970 any following 8971 global/generic 8972 load/load 8973 atomic/store/store 8974 atomic/atomicrmw. 8975 - Ensures any 8976 following global 8977 data read is no 8978 older than the local load 8979 atomic value being 8980 acquired. 8981 8982 load atomic acquire - workgroup - generic 1. flat_load sc0=1 8983 2. s_waitcnt lgkm/vmcnt(0) 8984 8985 - Use lgkmcnt(0) if not 8986 TgSplit execution mode 8987 and vmcnt(0) if TgSplit 8988 execution mode. 8989 - If OpenCL, omit lgkmcnt(0). 8990 - Must happen before 8991 the following 8992 buffer_inv and any 8993 following global/generic 8994 load/load 8995 atomic/store/store 8996 atomic/atomicrmw. 8997 - Ensures any 8998 following global 8999 data read is no 9000 older than a local load 9001 atomic value being 9002 acquired. 9003 9004 3. buffer_inv sc0=1 9005 9006 - If not TgSplit execution 9007 mode, omit. 9008 - Ensures that 9009 following 9010 loads will not see 9011 stale data. 9012 9013 load atomic acquire - agent - global 1. buffer/global_load 9014 sc1=1 9015 2. s_waitcnt vmcnt(0) 9016 9017 - Must happen before 9018 following 9019 buffer_inv. 9020 - Ensures the load 9021 has completed 9022 before invalidating 9023 the cache. 9024 9025 3. buffer_inv sc1=1 9026 9027 - Must happen before 9028 any following 9029 global/generic 9030 load/load 9031 atomic/atomicrmw. 9032 - Ensures that 9033 following 9034 loads will not see 9035 stale global data. 9036 9037 load atomic acquire - system - global 1. buffer/global/flat_load 9038 sc0=1 sc1=1 9039 2. s_waitcnt vmcnt(0) 9040 9041 - Must happen before 9042 following 9043 buffer_inv. 9044 - Ensures the load 9045 has completed 9046 before invalidating 9047 the cache. 9048 9049 3. buffer_inv sc0=1 sc1=1 9050 9051 - Must happen before 9052 any following 9053 global/generic 9054 load/load 9055 atomic/atomicrmw. 9056 - Ensures that 9057 following 9058 loads will not see 9059 stale MTYPE NC global data. 9060 MTYPE RW and CC memory will 9061 never be stale due to the 9062 memory probes. 9063 9064 load atomic acquire - agent - generic 1. flat_load sc1=1 9065 2. s_waitcnt vmcnt(0) & 9066 lgkmcnt(0) 9067 9068 - If TgSplit execution mode, 9069 omit lgkmcnt(0). 9070 - If OpenCL omit 9071 lgkmcnt(0). 9072 - Must happen before 9073 following 9074 buffer_inv. 9075 - Ensures the flat_load 9076 has completed 9077 before invalidating 9078 the cache. 9079 9080 3. buffer_inv sc1=1 9081 9082 - Must happen before 9083 any following 9084 global/generic 9085 load/load 9086 atomic/atomicrmw. 9087 - Ensures that 9088 following loads 9089 will not see stale 9090 global data. 9091 9092 load atomic acquire - system - generic 1. flat_load sc0=1 sc1=1 9093 2. s_waitcnt vmcnt(0) & 9094 lgkmcnt(0) 9095 9096 - If TgSplit execution mode, 9097 omit lgkmcnt(0). 9098 - If OpenCL omit 9099 lgkmcnt(0). 9100 - Must happen before 9101 the following 9102 buffer_inv. 9103 - Ensures the flat_load 9104 has completed 9105 before invalidating 9106 the caches. 9107 9108 3. buffer_inv sc0=1 sc1=1 9109 9110 - Must happen before 9111 any following 9112 global/generic 9113 load/load 9114 atomic/atomicrmw. 9115 - Ensures that 9116 following 9117 loads will not see 9118 stale MTYPE NC global data. 9119 MTYPE RW and CC memory will 9120 never be stale due to the 9121 memory probes. 9122 9123 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 9124 - wavefront - generic 9125 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 9126 - wavefront local address space cannot 9127 be used.* 9128 9129 1. ds_atomic 9130 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 9131 2. s_waitcnt vmcnt(0) 9132 9133 - If not TgSplit execution 9134 mode, omit. 9135 - Must happen before the 9136 following buffer_inv. 9137 - Ensures the atomicrmw 9138 has completed 9139 before invalidating 9140 the cache. 9141 9142 3. buffer_inv sc0=1 9143 9144 - If not TgSplit execution 9145 mode, omit. 9146 - Must happen before 9147 any following 9148 global/generic 9149 load/load 9150 atomic/atomicrmw. 9151 - Ensures that 9152 following loads 9153 will not see stale 9154 global data. 9155 9156 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 9157 local address space cannot 9158 be used.* 9159 9160 1. ds_atomic 9161 2. s_waitcnt lgkmcnt(0) 9162 9163 - If OpenCL, omit. 9164 - Must happen before 9165 any following 9166 global/generic 9167 load/load 9168 atomic/store/store 9169 atomic/atomicrmw. 9170 - Ensures any 9171 following global 9172 data read is no 9173 older than the local 9174 atomicrmw value 9175 being acquired. 9176 9177 atomicrmw acquire - workgroup - generic 1. flat_atomic 9178 2. s_waitcnt lgkm/vmcnt(0) 9179 9180 - Use lgkmcnt(0) if not 9181 TgSplit execution mode 9182 and vmcnt(0) if TgSplit 9183 execution mode. 9184 - If OpenCL, omit lgkmcnt(0). 9185 - Must happen before 9186 the following 9187 buffer_inv and 9188 any following 9189 global/generic 9190 load/load 9191 atomic/store/store 9192 atomic/atomicrmw. 9193 - Ensures any 9194 following global 9195 data read is no 9196 older than a local 9197 atomicrmw value 9198 being acquired. 9199 9200 3. buffer_inv sc0=1 9201 9202 - If not TgSplit execution 9203 mode, omit. 9204 - Ensures that 9205 following 9206 loads will not see 9207 stale data. 9208 9209 atomicrmw acquire - agent - global 1. buffer/global_atomic 9210 2. s_waitcnt vmcnt(0) 9211 9212 - Must happen before 9213 following 9214 buffer_inv. 9215 - Ensures the 9216 atomicrmw has 9217 completed before 9218 invalidating the 9219 cache. 9220 9221 3. buffer_inv sc1=1 9222 9223 - Must happen before 9224 any following 9225 global/generic 9226 load/load 9227 atomic/atomicrmw. 9228 - Ensures that 9229 following loads 9230 will not see stale 9231 global data. 9232 9233 atomicrmw acquire - system - global 1. buffer/global_atomic 9234 sc1=1 9235 2. s_waitcnt vmcnt(0) 9236 9237 - Must happen before 9238 following 9239 buffer_inv. 9240 - Ensures the 9241 atomicrmw has 9242 completed before 9243 invalidating the 9244 caches. 9245 9246 3. buffer_inv sc0=1 sc1=1 9247 9248 - Must happen before 9249 any following 9250 global/generic 9251 load/load 9252 atomic/atomicrmw. 9253 - Ensures that 9254 following 9255 loads will not see 9256 stale MTYPE NC global data. 9257 MTYPE RW and CC memory will 9258 never be stale due to the 9259 memory probes. 9260 9261 atomicrmw acquire - agent - generic 1. flat_atomic 9262 2. s_waitcnt vmcnt(0) & 9263 lgkmcnt(0) 9264 9265 - If TgSplit execution mode, 9266 omit lgkmcnt(0). 9267 - If OpenCL, omit 9268 lgkmcnt(0). 9269 - Must happen before 9270 following 9271 buffer_inv. 9272 - Ensures the 9273 atomicrmw has 9274 completed before 9275 invalidating the 9276 cache. 9277 9278 3. buffer_inv sc1=1 9279 9280 - Must happen before 9281 any following 9282 global/generic 9283 load/load 9284 atomic/atomicrmw. 9285 - Ensures that 9286 following loads 9287 will not see stale 9288 global data. 9289 9290 atomicrmw acquire - system - generic 1. flat_atomic sc1=1 9291 2. s_waitcnt vmcnt(0) & 9292 lgkmcnt(0) 9293 9294 - If TgSplit execution mode, 9295 omit lgkmcnt(0). 9296 - If OpenCL, omit 9297 lgkmcnt(0). 9298 - Must happen before 9299 following 9300 buffer_inv. 9301 - Ensures the 9302 atomicrmw has 9303 completed before 9304 invalidating the 9305 caches. 9306 9307 3. buffer_inv sc0=1 sc1=1 9308 9309 - Must happen before 9310 any following 9311 global/generic 9312 load/load 9313 atomic/atomicrmw. 9314 - Ensures that 9315 following 9316 loads will not see 9317 stale MTYPE NC global data. 9318 MTYPE RW and CC memory will 9319 never be stale due to the 9320 memory probes. 9321 9322 fence acquire - singlethread *none* *none* 9323 - wavefront 9324 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9325 9326 - Use lgkmcnt(0) if not 9327 TgSplit execution mode 9328 and vmcnt(0) if TgSplit 9329 execution mode. 9330 - If OpenCL and 9331 address space is 9332 not generic, omit 9333 lgkmcnt(0). 9334 - If OpenCL and 9335 address space is 9336 local, omit 9337 vmcnt(0). 9338 - However, since LLVM 9339 currently has no 9340 address space on 9341 the fence need to 9342 conservatively 9343 always generate. If 9344 fence had an 9345 address space then 9346 set to address 9347 space of OpenCL 9348 fence flag, or to 9349 generic if both 9350 local and global 9351 flags are 9352 specified. 9353 - s_waitcnt vmcnt(0) 9354 must happen after 9355 any preceding 9356 global/generic load 9357 atomic/ 9358 atomicrmw 9359 with an equal or 9360 wider sync scope 9361 and memory ordering 9362 stronger than 9363 unordered (this is 9364 termed the 9365 fence-paired-atomic). 9366 - s_waitcnt lgkmcnt(0) 9367 must happen after 9368 any preceding 9369 local/generic load 9370 atomic/atomicrmw 9371 with an equal or 9372 wider sync scope 9373 and memory ordering 9374 stronger than 9375 unordered (this is 9376 termed the 9377 fence-paired-atomic). 9378 - Must happen before 9379 the following 9380 buffer_inv and 9381 any following 9382 global/generic 9383 load/load 9384 atomic/store/store 9385 atomic/atomicrmw. 9386 - Ensures any 9387 following global 9388 data read is no 9389 older than the 9390 value read by the 9391 fence-paired-atomic. 9392 9393 3. buffer_inv sc0=1 9394 9395 - If not TgSplit execution 9396 mode, omit. 9397 - Ensures that 9398 following 9399 loads will not see 9400 stale data. 9401 9402 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9403 vmcnt(0) 9404 9405 - If TgSplit execution mode, 9406 omit lgkmcnt(0). 9407 - If OpenCL and 9408 address space is 9409 not generic, omit 9410 lgkmcnt(0). 9411 - However, since LLVM 9412 currently has no 9413 address space on 9414 the fence need to 9415 conservatively 9416 always generate 9417 (see comment for 9418 previous fence). 9419 - Could be split into 9420 separate s_waitcnt 9421 vmcnt(0) and 9422 s_waitcnt 9423 lgkmcnt(0) to allow 9424 them to be 9425 independently moved 9426 according to the 9427 following rules. 9428 - s_waitcnt vmcnt(0) 9429 must happen after 9430 any preceding 9431 global/generic load 9432 atomic/atomicrmw 9433 with an equal or 9434 wider sync scope 9435 and memory ordering 9436 stronger than 9437 unordered (this is 9438 termed the 9439 fence-paired-atomic). 9440 - s_waitcnt lgkmcnt(0) 9441 must happen after 9442 any preceding 9443 local/generic load 9444 atomic/atomicrmw 9445 with an equal or 9446 wider sync scope 9447 and memory ordering 9448 stronger than 9449 unordered (this is 9450 termed the 9451 fence-paired-atomic). 9452 - Must happen before 9453 the following 9454 buffer_inv. 9455 - Ensures that the 9456 fence-paired atomic 9457 has completed 9458 before invalidating 9459 the 9460 cache. Therefore 9461 any following 9462 locations read must 9463 be no older than 9464 the value read by 9465 the 9466 fence-paired-atomic. 9467 9468 2. buffer_inv sc1=1 9469 9470 - Must happen before any 9471 following global/generic 9472 load/load 9473 atomic/store/store 9474 atomic/atomicrmw. 9475 - Ensures that 9476 following loads 9477 will not see stale 9478 global data. 9479 9480 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 9481 vmcnt(0) 9482 9483 - If TgSplit execution mode, 9484 omit lgkmcnt(0). 9485 - If OpenCL and 9486 address space is 9487 not generic, omit 9488 lgkmcnt(0). 9489 - However, since LLVM 9490 currently has no 9491 address space on 9492 the fence need to 9493 conservatively 9494 always generate 9495 (see comment for 9496 previous fence). 9497 - Could be split into 9498 separate s_waitcnt 9499 vmcnt(0) and 9500 s_waitcnt 9501 lgkmcnt(0) to allow 9502 them to be 9503 independently moved 9504 according to the 9505 following rules. 9506 - s_waitcnt vmcnt(0) 9507 must happen after 9508 any preceding 9509 global/generic load 9510 atomic/atomicrmw 9511 with an equal or 9512 wider sync scope 9513 and memory ordering 9514 stronger than 9515 unordered (this is 9516 termed the 9517 fence-paired-atomic). 9518 - s_waitcnt lgkmcnt(0) 9519 must happen after 9520 any preceding 9521 local/generic load 9522 atomic/atomicrmw 9523 with an equal or 9524 wider sync scope 9525 and memory ordering 9526 stronger than 9527 unordered (this is 9528 termed the 9529 fence-paired-atomic). 9530 - Must happen before 9531 the following 9532 buffer_inv. 9533 - Ensures that the 9534 fence-paired atomic 9535 has completed 9536 before invalidating 9537 the 9538 cache. Therefore 9539 any following 9540 locations read must 9541 be no older than 9542 the value read by 9543 the 9544 fence-paired-atomic. 9545 9546 2. buffer_inv sc0=1 sc1=1 9547 9548 - Must happen before any 9549 following global/generic 9550 load/load 9551 atomic/store/store 9552 atomic/atomicrmw. 9553 - Ensures that 9554 following loads 9555 will not see stale 9556 global data. 9557 9558 **Release Atomic** 9559 ------------------------------------------------------------------------------------ 9560 store atomic release - singlethread - global 1. buffer/global/flat_store 9561 - wavefront - generic 9562 store atomic release - singlethread - local *If TgSplit execution mode, 9563 - wavefront local address space cannot 9564 be used.* 9565 9566 1. ds_store 9567 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9568 - generic 9569 - Use lgkmcnt(0) if not 9570 TgSplit execution mode 9571 and vmcnt(0) if TgSplit 9572 execution mode. 9573 - If OpenCL, omit lgkmcnt(0). 9574 - s_waitcnt vmcnt(0) 9575 must happen after 9576 any preceding 9577 global/generic load/store/ 9578 load atomic/store atomic/ 9579 atomicrmw. 9580 - s_waitcnt lgkmcnt(0) 9581 must happen after 9582 any preceding 9583 local/generic 9584 load/store/load 9585 atomic/store 9586 atomic/atomicrmw. 9587 - Must happen before 9588 the following 9589 store. 9590 - Ensures that all 9591 memory operations 9592 have 9593 completed before 9594 performing the 9595 store that is being 9596 released. 9597 9598 2. buffer/global/flat_store sc0=1 9599 store atomic release - workgroup - local *If TgSplit execution mode, 9600 local address space cannot 9601 be used.* 9602 9603 1. ds_store 9604 store atomic release - agent - global 1. buffer_wbl2 sc1=1 9605 - generic 9606 - Must happen before 9607 following s_waitcnt. 9608 - Performs L2 writeback to 9609 ensure previous 9610 global/generic 9611 store/atomicrmw are 9612 visible at agent scope. 9613 9614 2. s_waitcnt lgkmcnt(0) & 9615 vmcnt(0) 9616 9617 - If TgSplit execution mode, 9618 omit lgkmcnt(0). 9619 - If OpenCL and 9620 address space is 9621 not generic, omit 9622 lgkmcnt(0). 9623 - Could be split into 9624 separate s_waitcnt 9625 vmcnt(0) and 9626 s_waitcnt 9627 lgkmcnt(0) to allow 9628 them to be 9629 independently moved 9630 according to the 9631 following rules. 9632 - s_waitcnt vmcnt(0) 9633 must happen after 9634 any preceding 9635 global/generic 9636 load/store/load 9637 atomic/store 9638 atomic/atomicrmw. 9639 - s_waitcnt lgkmcnt(0) 9640 must happen after 9641 any preceding 9642 local/generic 9643 load/store/load 9644 atomic/store 9645 atomic/atomicrmw. 9646 - Must happen before 9647 the following 9648 store. 9649 - Ensures that all 9650 memory operations 9651 to memory have 9652 completed before 9653 performing the 9654 store that is being 9655 released. 9656 9657 3. buffer/global/flat_store sc1=1 9658 store atomic release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9659 - generic 9660 - Must happen before 9661 following s_waitcnt. 9662 - Performs L2 writeback to 9663 ensure previous 9664 global/generic 9665 store/atomicrmw are 9666 visible at system scope. 9667 9668 2. s_waitcnt lgkmcnt(0) & 9669 vmcnt(0) 9670 9671 - If TgSplit execution mode, 9672 omit lgkmcnt(0). 9673 - If OpenCL and 9674 address space is 9675 not generic, omit 9676 lgkmcnt(0). 9677 - Could be split into 9678 separate s_waitcnt 9679 vmcnt(0) and 9680 s_waitcnt 9681 lgkmcnt(0) to allow 9682 them to be 9683 independently moved 9684 according to the 9685 following rules. 9686 - s_waitcnt vmcnt(0) 9687 must happen after any 9688 preceding 9689 global/generic 9690 load/store/load 9691 atomic/store 9692 atomic/atomicrmw. 9693 - s_waitcnt lgkmcnt(0) 9694 must happen after any 9695 preceding 9696 local/generic 9697 load/store/load 9698 atomic/store 9699 atomic/atomicrmw. 9700 - Must happen before 9701 the following 9702 store. 9703 - Ensures that all 9704 memory operations 9705 to memory and the L2 9706 writeback have 9707 completed before 9708 performing the 9709 store that is being 9710 released. 9711 9712 3. buffer/global/flat_store 9713 sc0=1 sc1=1 9714 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 9715 - wavefront - generic 9716 atomicrmw release - singlethread - local *If TgSplit execution mode, 9717 - wavefront local address space cannot 9718 be used.* 9719 9720 1. ds_atomic 9721 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9722 - generic 9723 - Use lgkmcnt(0) if not 9724 TgSplit execution mode 9725 and vmcnt(0) if TgSplit 9726 execution mode. 9727 - If OpenCL, omit 9728 lgkmcnt(0). 9729 - s_waitcnt vmcnt(0) 9730 must happen after 9731 any preceding 9732 global/generic load/store/ 9733 load atomic/store atomic/ 9734 atomicrmw. 9735 - s_waitcnt lgkmcnt(0) 9736 must happen after 9737 any preceding 9738 local/generic 9739 load/store/load 9740 atomic/store 9741 atomic/atomicrmw. 9742 - Must happen before 9743 the following 9744 atomicrmw. 9745 - Ensures that all 9746 memory operations 9747 have 9748 completed before 9749 performing the 9750 atomicrmw that is 9751 being released. 9752 9753 2. buffer/global/flat_atomic sc0=1 9754 atomicrmw release - workgroup - local *If TgSplit execution mode, 9755 local address space cannot 9756 be used.* 9757 9758 1. ds_atomic 9759 atomicrmw release - agent - global 1. buffer_wbl2 sc1=1 9760 - generic 9761 - Must happen before 9762 following s_waitcnt. 9763 - Performs L2 writeback to 9764 ensure previous 9765 global/generic 9766 store/atomicrmw are 9767 visible at agent scope. 9768 9769 2. s_waitcnt lgkmcnt(0) & 9770 vmcnt(0) 9771 9772 - If TgSplit execution mode, 9773 omit lgkmcnt(0). 9774 - If OpenCL, omit 9775 lgkmcnt(0). 9776 - Could be split into 9777 separate s_waitcnt 9778 vmcnt(0) and 9779 s_waitcnt 9780 lgkmcnt(0) to allow 9781 them to be 9782 independently moved 9783 according to the 9784 following rules. 9785 - s_waitcnt vmcnt(0) 9786 must happen after 9787 any preceding 9788 global/generic 9789 load/store/load 9790 atomic/store 9791 atomic/atomicrmw. 9792 - s_waitcnt lgkmcnt(0) 9793 must happen after 9794 any preceding 9795 local/generic 9796 load/store/load 9797 atomic/store 9798 atomic/atomicrmw. 9799 - Must happen before 9800 the following 9801 atomicrmw. 9802 - Ensures that all 9803 memory operations 9804 to global and local 9805 have completed 9806 before performing 9807 the atomicrmw that 9808 is being released. 9809 9810 3. buffer/global/flat_atomic sc1=1 9811 atomicrmw release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9812 - generic 9813 - Must happen before 9814 following s_waitcnt. 9815 - Performs L2 writeback to 9816 ensure previous 9817 global/generic 9818 store/atomicrmw are 9819 visible at system scope. 9820 9821 2. s_waitcnt lgkmcnt(0) & 9822 vmcnt(0) 9823 9824 - If TgSplit execution mode, 9825 omit lgkmcnt(0). 9826 - If OpenCL, omit 9827 lgkmcnt(0). 9828 - Could be split into 9829 separate s_waitcnt 9830 vmcnt(0) and 9831 s_waitcnt 9832 lgkmcnt(0) to allow 9833 them to be 9834 independently moved 9835 according to the 9836 following rules. 9837 - s_waitcnt vmcnt(0) 9838 must happen after 9839 any preceding 9840 global/generic 9841 load/store/load 9842 atomic/store 9843 atomic/atomicrmw. 9844 - s_waitcnt lgkmcnt(0) 9845 must happen after 9846 any preceding 9847 local/generic 9848 load/store/load 9849 atomic/store 9850 atomic/atomicrmw. 9851 - Must happen before 9852 the following 9853 atomicrmw. 9854 - Ensures that all 9855 memory operations 9856 to memory and the L2 9857 writeback have 9858 completed before 9859 performing the 9860 store that is being 9861 released. 9862 9863 3. buffer/global/flat_atomic 9864 sc0=1 sc1=1 9865 fence release - singlethread *none* *none* 9866 - wavefront 9867 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9868 9869 - Use lgkmcnt(0) if not 9870 TgSplit execution mode 9871 and vmcnt(0) if TgSplit 9872 execution mode. 9873 - If OpenCL and 9874 address space is 9875 not generic, omit 9876 lgkmcnt(0). 9877 - If OpenCL and 9878 address space is 9879 local, omit 9880 vmcnt(0). 9881 - However, since LLVM 9882 currently has no 9883 address space on 9884 the fence need to 9885 conservatively 9886 always generate. If 9887 fence had an 9888 address space then 9889 set to address 9890 space of OpenCL 9891 fence flag, or to 9892 generic if both 9893 local and global 9894 flags are 9895 specified. 9896 - s_waitcnt vmcnt(0) 9897 must happen after 9898 any preceding 9899 global/generic 9900 load/store/ 9901 load atomic/store atomic/ 9902 atomicrmw. 9903 - s_waitcnt lgkmcnt(0) 9904 must happen after 9905 any preceding 9906 local/generic 9907 load/load 9908 atomic/store/store 9909 atomic/atomicrmw. 9910 - Must happen before 9911 any following store 9912 atomic/atomicrmw 9913 with an equal or 9914 wider sync scope 9915 and memory ordering 9916 stronger than 9917 unordered (this is 9918 termed the 9919 fence-paired-atomic). 9920 - Ensures that all 9921 memory operations 9922 have 9923 completed before 9924 performing the 9925 following 9926 fence-paired-atomic. 9927 9928 fence release - agent *none* 1. buffer_wbl2 sc1=1 9929 9930 - If OpenCL and 9931 address space is 9932 local, omit. 9933 - Must happen before 9934 following s_waitcnt. 9935 - Performs L2 writeback to 9936 ensure previous 9937 global/generic 9938 store/atomicrmw are 9939 visible at agent scope. 9940 9941 2. s_waitcnt lgkmcnt(0) & 9942 vmcnt(0) 9943 9944 - If TgSplit execution mode, 9945 omit lgkmcnt(0). 9946 - If OpenCL and 9947 address space is 9948 not generic, omit 9949 lgkmcnt(0). 9950 - If OpenCL and 9951 address space is 9952 local, omit 9953 vmcnt(0). 9954 - However, since LLVM 9955 currently has no 9956 address space on 9957 the fence need to 9958 conservatively 9959 always generate. If 9960 fence had an 9961 address space then 9962 set to address 9963 space of OpenCL 9964 fence flag, or to 9965 generic if both 9966 local and global 9967 flags are 9968 specified. 9969 - Could be split into 9970 separate s_waitcnt 9971 vmcnt(0) and 9972 s_waitcnt 9973 lgkmcnt(0) to allow 9974 them to be 9975 independently moved 9976 according to the 9977 following rules. 9978 - s_waitcnt vmcnt(0) 9979 must happen after 9980 any preceding 9981 global/generic 9982 load/store/load 9983 atomic/store 9984 atomic/atomicrmw. 9985 - s_waitcnt lgkmcnt(0) 9986 must happen after 9987 any preceding 9988 local/generic 9989 load/store/load 9990 atomic/store 9991 atomic/atomicrmw. 9992 - Must happen before 9993 any following store 9994 atomic/atomicrmw 9995 with an equal or 9996 wider sync scope 9997 and memory ordering 9998 stronger than 9999 unordered (this is 10000 termed the 10001 fence-paired-atomic). 10002 - Ensures that all 10003 memory operations 10004 have 10005 completed before 10006 performing the 10007 following 10008 fence-paired-atomic. 10009 10010 fence release - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10011 10012 - Must happen before 10013 following s_waitcnt. 10014 - Performs L2 writeback to 10015 ensure previous 10016 global/generic 10017 store/atomicrmw are 10018 visible at system scope. 10019 10020 2. s_waitcnt lgkmcnt(0) & 10021 vmcnt(0) 10022 10023 - If TgSplit execution mode, 10024 omit lgkmcnt(0). 10025 - If OpenCL and 10026 address space is 10027 not generic, omit 10028 lgkmcnt(0). 10029 - If OpenCL and 10030 address space is 10031 local, omit 10032 vmcnt(0). 10033 - However, since LLVM 10034 currently has no 10035 address space on 10036 the fence need to 10037 conservatively 10038 always generate. If 10039 fence had an 10040 address space then 10041 set to address 10042 space of OpenCL 10043 fence flag, or to 10044 generic if both 10045 local and global 10046 flags are 10047 specified. 10048 - Could be split into 10049 separate s_waitcnt 10050 vmcnt(0) and 10051 s_waitcnt 10052 lgkmcnt(0) to allow 10053 them to be 10054 independently moved 10055 according to the 10056 following rules. 10057 - s_waitcnt vmcnt(0) 10058 must happen after 10059 any preceding 10060 global/generic 10061 load/store/load 10062 atomic/store 10063 atomic/atomicrmw. 10064 - s_waitcnt lgkmcnt(0) 10065 must happen after 10066 any preceding 10067 local/generic 10068 load/store/load 10069 atomic/store 10070 atomic/atomicrmw. 10071 - Must happen before 10072 any following store 10073 atomic/atomicrmw 10074 with an equal or 10075 wider sync scope 10076 and memory ordering 10077 stronger than 10078 unordered (this is 10079 termed the 10080 fence-paired-atomic). 10081 - Ensures that all 10082 memory operations 10083 have 10084 completed before 10085 performing the 10086 following 10087 fence-paired-atomic. 10088 10089 **Acquire-Release Atomic** 10090 ------------------------------------------------------------------------------------ 10091 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 10092 - wavefront - generic 10093 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 10094 - wavefront local address space cannot 10095 be used.* 10096 10097 1. ds_atomic 10098 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10099 10100 - Use lgkmcnt(0) if not 10101 TgSplit execution mode 10102 and vmcnt(0) if TgSplit 10103 execution mode. 10104 - If OpenCL, omit 10105 lgkmcnt(0). 10106 - Must happen after 10107 any preceding 10108 local/generic 10109 load/store/load 10110 atomic/store 10111 atomic/atomicrmw. 10112 - s_waitcnt vmcnt(0) 10113 must happen after 10114 any preceding 10115 global/generic load/store/ 10116 load atomic/store atomic/ 10117 atomicrmw. 10118 - s_waitcnt lgkmcnt(0) 10119 must happen after 10120 any preceding 10121 local/generic 10122 load/store/load 10123 atomic/store 10124 atomic/atomicrmw. 10125 - Must happen before 10126 the following 10127 atomicrmw. 10128 - Ensures that all 10129 memory operations 10130 have 10131 completed before 10132 performing the 10133 atomicrmw that is 10134 being released. 10135 10136 2. buffer/global_atomic 10137 3. s_waitcnt vmcnt(0) 10138 10139 - If not TgSplit execution 10140 mode, omit. 10141 - Must happen before 10142 the following 10143 buffer_inv. 10144 - Ensures any 10145 following global 10146 data read is no 10147 older than the 10148 atomicrmw value 10149 being acquired. 10150 10151 4. buffer_inv sc0=1 10152 10153 - If not TgSplit execution 10154 mode, omit. 10155 - Ensures that 10156 following 10157 loads will not see 10158 stale data. 10159 10160 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 10161 local address space cannot 10162 be used.* 10163 10164 1. ds_atomic 10165 2. s_waitcnt lgkmcnt(0) 10166 10167 - If OpenCL, omit. 10168 - Must happen before 10169 any following 10170 global/generic 10171 load/load 10172 atomic/store/store 10173 atomic/atomicrmw. 10174 - Ensures any 10175 following global 10176 data read is no 10177 older than the local load 10178 atomic value being 10179 acquired. 10180 10181 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 10182 10183 - Use lgkmcnt(0) if not 10184 TgSplit execution mode 10185 and vmcnt(0) if TgSplit 10186 execution mode. 10187 - If OpenCL, omit 10188 lgkmcnt(0). 10189 - s_waitcnt vmcnt(0) 10190 must happen after 10191 any preceding 10192 global/generic load/store/ 10193 load atomic/store atomic/ 10194 atomicrmw. 10195 - s_waitcnt lgkmcnt(0) 10196 must happen after 10197 any preceding 10198 local/generic 10199 load/store/load 10200 atomic/store 10201 atomic/atomicrmw. 10202 - Must happen before 10203 the following 10204 atomicrmw. 10205 - Ensures that all 10206 memory operations 10207 have 10208 completed before 10209 performing the 10210 atomicrmw that is 10211 being released. 10212 10213 2. flat_atomic 10214 3. s_waitcnt lgkmcnt(0) & 10215 vmcnt(0) 10216 10217 - If not TgSplit execution 10218 mode, omit vmcnt(0). 10219 - If OpenCL, omit 10220 lgkmcnt(0). 10221 - Must happen before 10222 the following 10223 buffer_inv and 10224 any following 10225 global/generic 10226 load/load 10227 atomic/store/store 10228 atomic/atomicrmw. 10229 - Ensures any 10230 following global 10231 data read is no 10232 older than a local load 10233 atomic value being 10234 acquired. 10235 10236 3. buffer_inv sc0=1 10237 10238 - If not TgSplit execution 10239 mode, omit. 10240 - Ensures that 10241 following 10242 loads will not see 10243 stale data. 10244 10245 atomicrmw acq_rel - agent - global 1. buffer_wbl2 sc1=1 10246 10247 - Must happen before 10248 following s_waitcnt. 10249 - Performs L2 writeback to 10250 ensure previous 10251 global/generic 10252 store/atomicrmw are 10253 visible at agent scope. 10254 10255 2. s_waitcnt lgkmcnt(0) & 10256 vmcnt(0) 10257 10258 - If TgSplit execution mode, 10259 omit lgkmcnt(0). 10260 - If OpenCL, omit 10261 lgkmcnt(0). 10262 - Could be split into 10263 separate s_waitcnt 10264 vmcnt(0) and 10265 s_waitcnt 10266 lgkmcnt(0) to allow 10267 them to be 10268 independently moved 10269 according to the 10270 following rules. 10271 - s_waitcnt vmcnt(0) 10272 must happen after 10273 any preceding 10274 global/generic 10275 load/store/load 10276 atomic/store 10277 atomic/atomicrmw. 10278 - s_waitcnt lgkmcnt(0) 10279 must happen after 10280 any preceding 10281 local/generic 10282 load/store/load 10283 atomic/store 10284 atomic/atomicrmw. 10285 - Must happen before 10286 the following 10287 atomicrmw. 10288 - Ensures that all 10289 memory operations 10290 to global have 10291 completed before 10292 performing the 10293 atomicrmw that is 10294 being released. 10295 10296 3. buffer/global_atomic 10297 4. s_waitcnt vmcnt(0) 10298 10299 - Must happen before 10300 following 10301 buffer_inv. 10302 - Ensures the 10303 atomicrmw has 10304 completed before 10305 invalidating the 10306 cache. 10307 10308 5. buffer_inv sc1=1 10309 10310 - Must happen before 10311 any following 10312 global/generic 10313 load/load 10314 atomic/atomicrmw. 10315 - Ensures that 10316 following loads 10317 will not see stale 10318 global data. 10319 10320 atomicrmw acq_rel - system - global 1. buffer_wbl2 sc0=1 sc1=1 10321 10322 - Must happen before 10323 following s_waitcnt. 10324 - Performs L2 writeback to 10325 ensure previous 10326 global/generic 10327 store/atomicrmw are 10328 visible at system scope. 10329 10330 2. s_waitcnt lgkmcnt(0) & 10331 vmcnt(0) 10332 10333 - If TgSplit execution mode, 10334 omit lgkmcnt(0). 10335 - If OpenCL, omit 10336 lgkmcnt(0). 10337 - Could be split into 10338 separate s_waitcnt 10339 vmcnt(0) and 10340 s_waitcnt 10341 lgkmcnt(0) to allow 10342 them to be 10343 independently moved 10344 according to the 10345 following rules. 10346 - s_waitcnt vmcnt(0) 10347 must happen after 10348 any preceding 10349 global/generic 10350 load/store/load 10351 atomic/store 10352 atomic/atomicrmw. 10353 - s_waitcnt lgkmcnt(0) 10354 must happen after 10355 any preceding 10356 local/generic 10357 load/store/load 10358 atomic/store 10359 atomic/atomicrmw. 10360 - Must happen before 10361 the following 10362 atomicrmw. 10363 - Ensures that all 10364 memory operations 10365 to global and L2 writeback 10366 have completed before 10367 performing the 10368 atomicrmw that is 10369 being released. 10370 10371 3. buffer/global_atomic 10372 sc1=1 10373 4. s_waitcnt vmcnt(0) 10374 10375 - Must happen before 10376 following 10377 buffer_inv. 10378 - Ensures the 10379 atomicrmw has 10380 completed before 10381 invalidating the 10382 caches. 10383 10384 5. buffer_inv sc0=1 sc1=1 10385 10386 - Must happen before 10387 any following 10388 global/generic 10389 load/load 10390 atomic/atomicrmw. 10391 - Ensures that 10392 following loads 10393 will not see stale 10394 MTYPE NC global data. 10395 MTYPE RW and CC memory will 10396 never be stale due to the 10397 memory probes. 10398 10399 atomicrmw acq_rel - agent - generic 1. buffer_wbl2 sc1=1 10400 10401 - Must happen before 10402 following s_waitcnt. 10403 - Performs L2 writeback to 10404 ensure previous 10405 global/generic 10406 store/atomicrmw are 10407 visible at agent scope. 10408 10409 2. s_waitcnt lgkmcnt(0) & 10410 vmcnt(0) 10411 10412 - If TgSplit execution mode, 10413 omit lgkmcnt(0). 10414 - If OpenCL, omit 10415 lgkmcnt(0). 10416 - Could be split into 10417 separate s_waitcnt 10418 vmcnt(0) and 10419 s_waitcnt 10420 lgkmcnt(0) to allow 10421 them to be 10422 independently moved 10423 according to the 10424 following rules. 10425 - s_waitcnt vmcnt(0) 10426 must happen after 10427 any preceding 10428 global/generic 10429 load/store/load 10430 atomic/store 10431 atomic/atomicrmw. 10432 - s_waitcnt lgkmcnt(0) 10433 must happen after 10434 any preceding 10435 local/generic 10436 load/store/load 10437 atomic/store 10438 atomic/atomicrmw. 10439 - Must happen before 10440 the following 10441 atomicrmw. 10442 - Ensures that all 10443 memory operations 10444 to global have 10445 completed before 10446 performing the 10447 atomicrmw that is 10448 being released. 10449 10450 3. flat_atomic 10451 4. s_waitcnt vmcnt(0) & 10452 lgkmcnt(0) 10453 10454 - If TgSplit execution mode, 10455 omit lgkmcnt(0). 10456 - If OpenCL, omit 10457 lgkmcnt(0). 10458 - Must happen before 10459 following 10460 buffer_inv. 10461 - Ensures the 10462 atomicrmw has 10463 completed before 10464 invalidating the 10465 cache. 10466 10467 5. buffer_inv sc1=1 10468 10469 - Must happen before 10470 any following 10471 global/generic 10472 load/load 10473 atomic/atomicrmw. 10474 - Ensures that 10475 following loads 10476 will not see stale 10477 global data. 10478 10479 atomicrmw acq_rel - system - generic 1. buffer_wbl2 sc0=1 sc1=1 10480 10481 - Must happen before 10482 following s_waitcnt. 10483 - Performs L2 writeback to 10484 ensure previous 10485 global/generic 10486 store/atomicrmw are 10487 visible at system scope. 10488 10489 2. s_waitcnt lgkmcnt(0) & 10490 vmcnt(0) 10491 10492 - If TgSplit execution mode, 10493 omit lgkmcnt(0). 10494 - If OpenCL, omit 10495 lgkmcnt(0). 10496 - Could be split into 10497 separate s_waitcnt 10498 vmcnt(0) and 10499 s_waitcnt 10500 lgkmcnt(0) to allow 10501 them to be 10502 independently moved 10503 according to the 10504 following rules. 10505 - s_waitcnt vmcnt(0) 10506 must happen after 10507 any preceding 10508 global/generic 10509 load/store/load 10510 atomic/store 10511 atomic/atomicrmw. 10512 - s_waitcnt lgkmcnt(0) 10513 must happen after 10514 any preceding 10515 local/generic 10516 load/store/load 10517 atomic/store 10518 atomic/atomicrmw. 10519 - Must happen before 10520 the following 10521 atomicrmw. 10522 - Ensures that all 10523 memory operations 10524 to global and L2 writeback 10525 have completed before 10526 performing the 10527 atomicrmw that is 10528 being released. 10529 10530 3. flat_atomic sc1=1 10531 4. s_waitcnt vmcnt(0) & 10532 lgkmcnt(0) 10533 10534 - If TgSplit execution mode, 10535 omit lgkmcnt(0). 10536 - If OpenCL, omit 10537 lgkmcnt(0). 10538 - Must happen before 10539 following 10540 buffer_inv. 10541 - Ensures the 10542 atomicrmw has 10543 completed before 10544 invalidating the 10545 caches. 10546 10547 5. buffer_inv sc0=1 sc1=1 10548 10549 - Must happen before 10550 any following 10551 global/generic 10552 load/load 10553 atomic/atomicrmw. 10554 - Ensures that 10555 following loads 10556 will not see stale 10557 MTYPE NC global data. 10558 MTYPE RW and CC memory will 10559 never be stale due to the 10560 memory probes. 10561 10562 fence acq_rel - singlethread *none* *none* 10563 - wavefront 10564 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 10565 10566 - Use lgkmcnt(0) if not 10567 TgSplit execution mode 10568 and vmcnt(0) if TgSplit 10569 execution mode. 10570 - If OpenCL and 10571 address space is 10572 not generic, omit 10573 lgkmcnt(0). 10574 - If OpenCL and 10575 address space is 10576 local, omit 10577 vmcnt(0). 10578 - However, 10579 since LLVM 10580 currently has no 10581 address space on 10582 the fence need to 10583 conservatively 10584 always generate 10585 (see comment for 10586 previous fence). 10587 - s_waitcnt vmcnt(0) 10588 must happen after 10589 any preceding 10590 global/generic 10591 load/store/ 10592 load atomic/store atomic/ 10593 atomicrmw. 10594 - s_waitcnt lgkmcnt(0) 10595 must happen after 10596 any preceding 10597 local/generic 10598 load/load 10599 atomic/store/store 10600 atomic/atomicrmw. 10601 - Must happen before 10602 any following 10603 global/generic 10604 load/load 10605 atomic/store/store 10606 atomic/atomicrmw. 10607 - Ensures that all 10608 memory operations 10609 have 10610 completed before 10611 performing any 10612 following global 10613 memory operations. 10614 - Ensures that the 10615 preceding 10616 local/generic load 10617 atomic/atomicrmw 10618 with an equal or 10619 wider sync scope 10620 and memory ordering 10621 stronger than 10622 unordered (this is 10623 termed the 10624 acquire-fence-paired-atomic) 10625 has completed 10626 before following 10627 global memory 10628 operations. This 10629 satisfies the 10630 requirements of 10631 acquire. 10632 - Ensures that all 10633 previous memory 10634 operations have 10635 completed before a 10636 following 10637 local/generic store 10638 atomic/atomicrmw 10639 with an equal or 10640 wider sync scope 10641 and memory ordering 10642 stronger than 10643 unordered (this is 10644 termed the 10645 release-fence-paired-atomic). 10646 This satisfies the 10647 requirements of 10648 release. 10649 - Must happen before 10650 the following 10651 buffer_inv. 10652 - Ensures that the 10653 acquire-fence-paired 10654 atomic has completed 10655 before invalidating 10656 the 10657 cache. Therefore 10658 any following 10659 locations read must 10660 be no older than 10661 the value read by 10662 the 10663 acquire-fence-paired-atomic. 10664 10665 3. buffer_inv sc0=1 10666 10667 - If not TgSplit execution 10668 mode, omit. 10669 - Ensures that 10670 following 10671 loads will not see 10672 stale data. 10673 10674 fence acq_rel - agent *none* 1. buffer_wbl2 sc1=1 10675 10676 - If OpenCL and 10677 address space is 10678 local, omit. 10679 - Must happen before 10680 following s_waitcnt. 10681 - Performs L2 writeback to 10682 ensure previous 10683 global/generic 10684 store/atomicrmw are 10685 visible at agent scope. 10686 10687 2. s_waitcnt lgkmcnt(0) & 10688 vmcnt(0) 10689 10690 - If TgSplit execution mode, 10691 omit lgkmcnt(0). 10692 - If OpenCL and 10693 address space is 10694 not generic, omit 10695 lgkmcnt(0). 10696 - However, since LLVM 10697 currently has no 10698 address space on 10699 the fence need to 10700 conservatively 10701 always generate 10702 (see comment for 10703 previous fence). 10704 - Could be split into 10705 separate s_waitcnt 10706 vmcnt(0) and 10707 s_waitcnt 10708 lgkmcnt(0) to allow 10709 them to be 10710 independently moved 10711 according to the 10712 following rules. 10713 - s_waitcnt vmcnt(0) 10714 must happen after 10715 any preceding 10716 global/generic 10717 load/store/load 10718 atomic/store 10719 atomic/atomicrmw. 10720 - s_waitcnt lgkmcnt(0) 10721 must happen after 10722 any preceding 10723 local/generic 10724 load/store/load 10725 atomic/store 10726 atomic/atomicrmw. 10727 - Must happen before 10728 the following 10729 buffer_inv. 10730 - Ensures that the 10731 preceding 10732 global/local/generic 10733 load 10734 atomic/atomicrmw 10735 with an equal or 10736 wider sync scope 10737 and memory ordering 10738 stronger than 10739 unordered (this is 10740 termed the 10741 acquire-fence-paired-atomic) 10742 has completed 10743 before invalidating 10744 the cache. This 10745 satisfies the 10746 requirements of 10747 acquire. 10748 - Ensures that all 10749 previous memory 10750 operations have 10751 completed before a 10752 following 10753 global/local/generic 10754 store 10755 atomic/atomicrmw 10756 with an equal or 10757 wider sync scope 10758 and memory ordering 10759 stronger than 10760 unordered (this is 10761 termed the 10762 release-fence-paired-atomic). 10763 This satisfies the 10764 requirements of 10765 release. 10766 10767 3. buffer_inv sc1=1 10768 10769 - Must happen before 10770 any following 10771 global/generic 10772 load/load 10773 atomic/store/store 10774 atomic/atomicrmw. 10775 - Ensures that 10776 following loads 10777 will not see stale 10778 global data. This 10779 satisfies the 10780 requirements of 10781 acquire. 10782 10783 fence acq_rel - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10784 10785 - If OpenCL and 10786 address space is 10787 local, omit. 10788 - Must happen before 10789 following s_waitcnt. 10790 - Performs L2 writeback to 10791 ensure previous 10792 global/generic 10793 store/atomicrmw are 10794 visible at system scope. 10795 10796 1. s_waitcnt lgkmcnt(0) & 10797 vmcnt(0) 10798 10799 - If TgSplit execution mode, 10800 omit lgkmcnt(0). 10801 - If OpenCL and 10802 address space is 10803 not generic, omit 10804 lgkmcnt(0). 10805 - However, since LLVM 10806 currently has no 10807 address space on 10808 the fence need to 10809 conservatively 10810 always generate 10811 (see comment for 10812 previous fence). 10813 - Could be split into 10814 separate s_waitcnt 10815 vmcnt(0) and 10816 s_waitcnt 10817 lgkmcnt(0) to allow 10818 them to be 10819 independently moved 10820 according to the 10821 following rules. 10822 - s_waitcnt vmcnt(0) 10823 must happen after 10824 any preceding 10825 global/generic 10826 load/store/load 10827 atomic/store 10828 atomic/atomicrmw. 10829 - s_waitcnt lgkmcnt(0) 10830 must happen after 10831 any preceding 10832 local/generic 10833 load/store/load 10834 atomic/store 10835 atomic/atomicrmw. 10836 - Must happen before 10837 the following 10838 buffer_inv. 10839 - Ensures that the 10840 preceding 10841 global/local/generic 10842 load 10843 atomic/atomicrmw 10844 with an equal or 10845 wider sync scope 10846 and memory ordering 10847 stronger than 10848 unordered (this is 10849 termed the 10850 acquire-fence-paired-atomic) 10851 has completed 10852 before invalidating 10853 the cache. This 10854 satisfies the 10855 requirements of 10856 acquire. 10857 - Ensures that all 10858 previous memory 10859 operations have 10860 completed before a 10861 following 10862 global/local/generic 10863 store 10864 atomic/atomicrmw 10865 with an equal or 10866 wider sync scope 10867 and memory ordering 10868 stronger than 10869 unordered (this is 10870 termed the 10871 release-fence-paired-atomic). 10872 This satisfies the 10873 requirements of 10874 release. 10875 10876 2. buffer_inv sc0=1 sc1=1 10877 10878 - Must happen before 10879 any following 10880 global/generic 10881 load/load 10882 atomic/store/store 10883 atomic/atomicrmw. 10884 - Ensures that 10885 following loads 10886 will not see stale 10887 MTYPE NC global data. 10888 MTYPE RW and CC memory will 10889 never be stale due to the 10890 memory probes. 10891 10892 **Sequential Consistent Atomic** 10893 ------------------------------------------------------------------------------------ 10894 load atomic seq_cst - singlethread - global *Same as corresponding 10895 - wavefront - local load atomic acquire, 10896 - generic except must generate 10897 all instructions even 10898 for OpenCL.* 10899 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10900 - generic 10901 - Use lgkmcnt(0) if not 10902 TgSplit execution mode 10903 and vmcnt(0) if TgSplit 10904 execution mode. 10905 - s_waitcnt lgkmcnt(0) must 10906 happen after 10907 preceding 10908 local/generic load 10909 atomic/store 10910 atomic/atomicrmw 10911 with memory 10912 ordering of seq_cst 10913 and with equal or 10914 wider sync scope. 10915 (Note that seq_cst 10916 fences have their 10917 own s_waitcnt 10918 lgkmcnt(0) and so do 10919 not need to be 10920 considered.) 10921 - s_waitcnt vmcnt(0) 10922 must happen after 10923 preceding 10924 global/generic load 10925 atomic/store 10926 atomic/atomicrmw 10927 with memory 10928 ordering of seq_cst 10929 and with equal or 10930 wider sync scope. 10931 (Note that seq_cst 10932 fences have their 10933 own s_waitcnt 10934 vmcnt(0) and so do 10935 not need to be 10936 considered.) 10937 - Ensures any 10938 preceding 10939 sequential 10940 consistent global/local 10941 memory instructions 10942 have completed 10943 before executing 10944 this sequentially 10945 consistent 10946 instruction. This 10947 prevents reordering 10948 a seq_cst store 10949 followed by a 10950 seq_cst load. (Note 10951 that seq_cst is 10952 stronger than 10953 acquire/release as 10954 the reordering of 10955 load acquire 10956 followed by a store 10957 release is 10958 prevented by the 10959 s_waitcnt of 10960 the release, but 10961 there is nothing 10962 preventing a store 10963 release followed by 10964 load acquire from 10965 completing out of 10966 order. The s_waitcnt 10967 could be placed after 10968 seq_store or before 10969 the seq_load. We 10970 choose the load to 10971 make the s_waitcnt be 10972 as late as possible 10973 so that the store 10974 may have already 10975 completed.) 10976 10977 2. *Following 10978 instructions same as 10979 corresponding load 10980 atomic acquire, 10981 except must generate 10982 all instructions even 10983 for OpenCL.* 10984 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 10985 local address space cannot 10986 be used.* 10987 10988 *Same as corresponding 10989 load atomic acquire, 10990 except must generate 10991 all instructions even 10992 for OpenCL.* 10993 10994 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10995 - system - generic vmcnt(0) 10996 10997 - If TgSplit execution mode, 10998 omit lgkmcnt(0). 10999 - Could be split into 11000 separate s_waitcnt 11001 vmcnt(0) 11002 and s_waitcnt 11003 lgkmcnt(0) to allow 11004 them to be 11005 independently moved 11006 according to the 11007 following rules. 11008 - s_waitcnt lgkmcnt(0) 11009 must happen after 11010 preceding 11011 global/generic load 11012 atomic/store 11013 atomic/atomicrmw 11014 with memory 11015 ordering of seq_cst 11016 and with equal or 11017 wider sync scope. 11018 (Note that seq_cst 11019 fences have their 11020 own s_waitcnt 11021 lgkmcnt(0) and so do 11022 not need to be 11023 considered.) 11024 - s_waitcnt vmcnt(0) 11025 must happen after 11026 preceding 11027 global/generic load 11028 atomic/store 11029 atomic/atomicrmw 11030 with memory 11031 ordering of seq_cst 11032 and with equal or 11033 wider sync scope. 11034 (Note that seq_cst 11035 fences have their 11036 own s_waitcnt 11037 vmcnt(0) and so do 11038 not need to be 11039 considered.) 11040 - Ensures any 11041 preceding 11042 sequential 11043 consistent global 11044 memory instructions 11045 have completed 11046 before executing 11047 this sequentially 11048 consistent 11049 instruction. This 11050 prevents reordering 11051 a seq_cst store 11052 followed by a 11053 seq_cst load. (Note 11054 that seq_cst is 11055 stronger than 11056 acquire/release as 11057 the reordering of 11058 load acquire 11059 followed by a store 11060 release is 11061 prevented by the 11062 s_waitcnt of 11063 the release, but 11064 there is nothing 11065 preventing a store 11066 release followed by 11067 load acquire from 11068 completing out of 11069 order. The s_waitcnt 11070 could be placed after 11071 seq_store or before 11072 the seq_load. We 11073 choose the load to 11074 make the s_waitcnt be 11075 as late as possible 11076 so that the store 11077 may have already 11078 completed.) 11079 11080 2. *Following 11081 instructions same as 11082 corresponding load 11083 atomic acquire, 11084 except must generate 11085 all instructions even 11086 for OpenCL.* 11087 store atomic seq_cst - singlethread - global *Same as corresponding 11088 - wavefront - local store atomic release, 11089 - workgroup - generic except must generate 11090 - agent all instructions even 11091 - system for OpenCL.* 11092 atomicrmw seq_cst - singlethread - global *Same as corresponding 11093 - wavefront - local atomicrmw acq_rel, 11094 - workgroup - generic except must generate 11095 - agent all instructions even 11096 - system for OpenCL.* 11097 fence seq_cst - singlethread *none* *Same as corresponding 11098 - wavefront fence acq_rel, 11099 - workgroup except must generate 11100 - agent all instructions even 11101 - system for OpenCL.* 11102 ============ ============ ============== ========== ================================ 11103 11104.. _amdgpu-amdhsa-memory-model-gfx10: 11105 11106Memory Model GFX10 11107++++++++++++++++++ 11108 11109For GFX10: 11110 11111* Each agent has multiple shader arrays (SA). 11112* Each SA has multiple work-group processors (WGP). 11113* Each WGP has multiple compute units (CU). 11114* Each CU has multiple SIMDs that execute wavefronts. 11115* The wavefronts for a single work-group are executed in the same 11116 WGP. In CU wavefront execution mode the wavefronts may be executed by 11117 different SIMDs in the same CU. In WGP wavefront execution mode the 11118 wavefronts may be executed by different SIMDs in different CUs in the same 11119 WGP. 11120* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 11121 executing on it. 11122* All LDS operations of a WGP are performed as wavefront wide operations in a 11123 global order and involve no caching. Completion is reported to a wavefront in 11124 execution order. 11125* The LDS memory has multiple request queues shared by the SIMDs of a 11126 WGP. Therefore, the LDS operations performed by different wavefronts of a 11127 work-group can be reordered relative to each other, which can result in 11128 reordering the visibility of vector memory operations with respect to LDS 11129 operations of other wavefronts in the same work-group. A ``s_waitcnt 11130 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 11131 vector memory operations between wavefronts of a work-group, but not between 11132 operations performed by the same wavefront. 11133* The vector memory operations are performed as wavefront wide operations. 11134 Completion of load/store/sample operations are reported to a wavefront in 11135 execution order of other load/store/sample operations performed by that 11136 wavefront. 11137* The vector memory operations access a vector L0 cache. There is a single L0 11138 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 11139 special action is required for coherence between the lanes of a single 11140 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 11141 wavefronts executing in the same work-group as they may be executing on SIMDs 11142 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 11143 required for coherence between wavefronts executing in different work-groups 11144 as they may be executing on different WGPs. 11145* The scalar memory operations access a scalar L0 cache shared by all wavefronts 11146 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 11147 operations are used in a restricted way so do not impact the memory model. See 11148 :ref:`amdgpu-amdhsa-memory-spaces`. 11149* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 11150 the same SA. Therefore, no special action is required for coherence between 11151 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 11152 required for coherence between wavefronts executing in different work-groups 11153 as they may be executing on different SAs that access different L1s. 11154* The L1 caches have independent quadrants to service disjoint ranges of virtual 11155 addresses. 11156* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 11157 vector and scalar memory operations performed by different wavefronts, whether 11158 executing in the same or different work-groups (which may be executing on 11159 different CUs accessing different L0s), can be reordered relative to each 11160 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 11161 synchronization between vector memory operations of different wavefronts. It 11162 ensures a previous vector memory operation has completed before executing a 11163 subsequent vector memory or LDS operation and so can be used to meet the 11164 requirements of acquire, release and sequential consistency. 11165* The L1 caches use an L2 cache shared by all SAs on the same agent. 11166* The L2 cache has independent channels to service disjoint ranges of virtual 11167 addresses. 11168* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 11169 quadrant has a separate request queue per L2 channel. Therefore, the vector 11170 and scalar memory operations performed by wavefronts executing in different 11171 work-groups (which may be executing on different SAs) of an agent can be 11172 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 11173 required to ensure synchronization between vector memory operations of 11174 different SAs. It ensures a previous vector memory operation has completed 11175 before executing a subsequent vector memory and so can be used to meet the 11176 requirements of acquire, release and sequential consistency. 11177* The L2 cache can be kept coherent with other agents on some targets, or ranges 11178 of virtual addresses can be set up to bypass it to ensure system coherence. 11179* On GFX10.3 a memory attached last level (MALL) cache exists for GPU memory. 11180 The MALL cache is fully coherent with GPU memory and has no impact on system 11181 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache. 11182 11183Scalar memory operations are only used to access memory that is proven to not 11184change during the execution of the kernel dispatch. This includes constant 11185address space and global address space for program scope ``const`` variables. 11186Therefore, the kernel machine code does not have to maintain the scalar cache to 11187ensure it is coherent with the vector caches. The scalar and vector caches are 11188invalidated between kernel dispatches by CP since constant address space data 11189may change between kernel dispatch executions. See 11190:ref:`amdgpu-amdhsa-memory-spaces`. 11191 11192The one exception is if scalar writes are used to spill SGPR registers. In this 11193case the AMDGPU backend ensures the memory location used to spill is never 11194accessed by vector memory operations at the same time. If scalar writes are used 11195then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 11196return since the locations may be used for vector memory instructions by a 11197future wavefront that uses the same scratch area, or a function call that 11198creates a frame at the same address, respectively. There is no need for a 11199``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 11200 11201For kernarg backing memory: 11202 11203* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 11204* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 11205 needing to invalidate the L2 cache. 11206* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 11207 so the L2 cache will be coherent with the CPU and other agents. 11208 11209Scratch backing memory (which is used for the private address space) is accessed 11210with MTYPE NC (non-coherent). Since the private address space is only accessed 11211by a single thread, and is always write-before-read, there is never a need to 11212invalidate these entries from the L0 or L1 caches. 11213 11214Wavefronts are executed in native mode with in-order reporting of loads and 11215sample instructions. In this mode vmcnt reports completion of load, atomic with 11216return and sample instructions in order, and the vscnt reports the completion of 11217store and atomic without return in order. See ``MEM_ORDERED`` field in 11218:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11219 11220Wavefronts can be executed in WGP or CU wavefront execution mode: 11221 11222* In WGP wavefront execution mode the wavefronts of a work-group are executed 11223 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 11224 CU L0 caches is required for work-group synchronization. Also accesses to L1 11225 at work-group scope need to be explicitly ordered as the accesses from 11226 different CUs are not ordered. 11227* In CU wavefront execution mode the wavefronts of a work-group are executed on 11228 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 11229 the work-group access the same L0 which in turn ensures L1 accesses are 11230 ordered and so do not require explicit management of the caches for 11231 work-group synchronization. 11232 11233See ``WGP_MODE`` field in 11234:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 11235:ref:`amdgpu-target-features`. 11236 11237The code sequences used to implement the memory model for GFX10 are defined in 11238table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 11239 11240 .. table:: AMDHSA Memory Model Code Sequences GFX10 11241 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 11242 11243 ============ ============ ============== ========== ================================ 11244 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 11245 Ordering Sync Scope Address GFX10 11246 Space 11247 ============ ============ ============== ========== ================================ 11248 **Non-Atomic** 11249 ------------------------------------------------------------------------------------ 11250 load *none* *none* - global - !volatile & !nontemporal 11251 - generic 11252 - private 1. buffer/global/flat_load 11253 - constant 11254 - !volatile & nontemporal 11255 11256 1. buffer/global/flat_load 11257 slc=1 11258 11259 - volatile 11260 11261 1. buffer/global/flat_load 11262 glc=1 dlc=1 11263 2. s_waitcnt vmcnt(0) 11264 11265 - Must happen before 11266 any following volatile 11267 global/generic 11268 load/store. 11269 - Ensures that 11270 volatile 11271 operations to 11272 different 11273 addresses will not 11274 be reordered by 11275 hardware. 11276 11277 load *none* *none* - local 1. ds_load 11278 store *none* *none* - global - !volatile & !nontemporal 11279 - generic 11280 - private 1. buffer/global/flat_store 11281 - constant 11282 - !volatile & nontemporal 11283 11284 1. buffer/global/flat_store 11285 glc=1 slc=1 11286 11287 - volatile 11288 11289 1. buffer/global/flat_store 11290 2. s_waitcnt vscnt(0) 11291 11292 - Must happen before 11293 any following volatile 11294 global/generic 11295 load/store. 11296 - Ensures that 11297 volatile 11298 operations to 11299 different 11300 addresses will not 11301 be reordered by 11302 hardware. 11303 11304 store *none* *none* - local 1. ds_store 11305 **Unordered Atomic** 11306 ------------------------------------------------------------------------------------ 11307 load atomic unordered *any* *any* *Same as non-atomic*. 11308 store atomic unordered *any* *any* *Same as non-atomic*. 11309 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 11310 **Monotonic Atomic** 11311 ------------------------------------------------------------------------------------ 11312 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 11313 - wavefront - generic 11314 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 11315 - generic glc=1 11316 11317 - If CU wavefront execution 11318 mode, omit glc=1. 11319 11320 load atomic monotonic - singlethread - local 1. ds_load 11321 - wavefront 11322 - workgroup 11323 load atomic monotonic - agent - global 1. buffer/global/flat_load 11324 - system - generic glc=1 dlc=1 11325 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 11326 - wavefront - generic 11327 - workgroup 11328 - agent 11329 - system 11330 store atomic monotonic - singlethread - local 1. ds_store 11331 - wavefront 11332 - workgroup 11333 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 11334 - wavefront - generic 11335 - workgroup 11336 - agent 11337 - system 11338 atomicrmw monotonic - singlethread - local 1. ds_atomic 11339 - wavefront 11340 - workgroup 11341 **Acquire Atomic** 11342 ------------------------------------------------------------------------------------ 11343 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 11344 - wavefront - local 11345 - generic 11346 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 11347 11348 - If CU wavefront execution 11349 mode, omit glc=1. 11350 11351 2. s_waitcnt vmcnt(0) 11352 11353 - If CU wavefront execution 11354 mode, omit. 11355 - Must happen before 11356 the following buffer_gl0_inv 11357 and before any following 11358 global/generic 11359 load/load 11360 atomic/store/store 11361 atomic/atomicrmw. 11362 11363 3. buffer_gl0_inv 11364 11365 - If CU wavefront execution 11366 mode, omit. 11367 - Ensures that 11368 following 11369 loads will not see 11370 stale data. 11371 11372 load atomic acquire - workgroup - local 1. ds_load 11373 2. s_waitcnt lgkmcnt(0) 11374 11375 - If OpenCL, omit. 11376 - Must happen before 11377 the following buffer_gl0_inv 11378 and before any following 11379 global/generic load/load 11380 atomic/store/store 11381 atomic/atomicrmw. 11382 - Ensures any 11383 following global 11384 data read is no 11385 older than the local load 11386 atomic value being 11387 acquired. 11388 11389 3. buffer_gl0_inv 11390 11391 - If CU wavefront execution 11392 mode, omit. 11393 - If OpenCL, omit. 11394 - Ensures that 11395 following 11396 loads will not see 11397 stale data. 11398 11399 load atomic acquire - workgroup - generic 1. flat_load glc=1 11400 11401 - If CU wavefront execution 11402 mode, omit glc=1. 11403 11404 2. s_waitcnt lgkmcnt(0) & 11405 vmcnt(0) 11406 11407 - If CU wavefront execution 11408 mode, omit vmcnt(0). 11409 - If OpenCL, omit 11410 lgkmcnt(0). 11411 - Must happen before 11412 the following 11413 buffer_gl0_inv and any 11414 following global/generic 11415 load/load 11416 atomic/store/store 11417 atomic/atomicrmw. 11418 - Ensures any 11419 following global 11420 data read is no 11421 older than a local load 11422 atomic value being 11423 acquired. 11424 11425 3. buffer_gl0_inv 11426 11427 - If CU wavefront execution 11428 mode, omit. 11429 - Ensures that 11430 following 11431 loads will not see 11432 stale data. 11433 11434 load atomic acquire - agent - global 1. buffer/global_load 11435 - system glc=1 dlc=1 11436 2. s_waitcnt vmcnt(0) 11437 11438 - Must happen before 11439 following 11440 buffer_gl*_inv. 11441 - Ensures the load 11442 has completed 11443 before invalidating 11444 the caches. 11445 11446 3. buffer_gl0_inv; 11447 buffer_gl1_inv 11448 11449 - Must happen before 11450 any following 11451 global/generic 11452 load/load 11453 atomic/atomicrmw. 11454 - Ensures that 11455 following 11456 loads will not see 11457 stale global data. 11458 11459 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 11460 - system 2. s_waitcnt vmcnt(0) & 11461 lgkmcnt(0) 11462 11463 - If OpenCL omit 11464 lgkmcnt(0). 11465 - Must happen before 11466 following 11467 buffer_gl*_invl. 11468 - Ensures the flat_load 11469 has completed 11470 before invalidating 11471 the caches. 11472 11473 3. buffer_gl0_inv; 11474 buffer_gl1_inv 11475 11476 - Must happen before 11477 any following 11478 global/generic 11479 load/load 11480 atomic/atomicrmw. 11481 - Ensures that 11482 following loads 11483 will not see stale 11484 global data. 11485 11486 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 11487 - wavefront - local 11488 - generic 11489 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 11490 2. s_waitcnt vm/vscnt(0) 11491 11492 - If CU wavefront execution 11493 mode, omit. 11494 - Use vmcnt(0) if atomic with 11495 return and vscnt(0) if 11496 atomic with no-return. 11497 - Must happen before 11498 the following buffer_gl0_inv 11499 and before any following 11500 global/generic 11501 load/load 11502 atomic/store/store 11503 atomic/atomicrmw. 11504 11505 3. buffer_gl0_inv 11506 11507 - If CU wavefront execution 11508 mode, omit. 11509 - Ensures that 11510 following 11511 loads will not see 11512 stale data. 11513 11514 atomicrmw acquire - workgroup - local 1. ds_atomic 11515 2. s_waitcnt lgkmcnt(0) 11516 11517 - If OpenCL, omit. 11518 - Must happen before 11519 the following 11520 buffer_gl0_inv. 11521 - Ensures any 11522 following global 11523 data read is no 11524 older than the local 11525 atomicrmw value 11526 being acquired. 11527 11528 3. buffer_gl0_inv 11529 11530 - If OpenCL omit. 11531 - Ensures that 11532 following 11533 loads will not see 11534 stale data. 11535 11536 atomicrmw acquire - workgroup - generic 1. flat_atomic 11537 2. s_waitcnt lgkmcnt(0) & 11538 vm/vscnt(0) 11539 11540 - If CU wavefront execution 11541 mode, omit vm/vscnt(0). 11542 - If OpenCL, omit lgkmcnt(0). 11543 - Use vmcnt(0) if atomic with 11544 return and vscnt(0) if 11545 atomic with no-return. 11546 - Must happen before 11547 the following 11548 buffer_gl0_inv. 11549 - Ensures any 11550 following global 11551 data read is no 11552 older than a local 11553 atomicrmw value 11554 being acquired. 11555 11556 3. buffer_gl0_inv 11557 11558 - If CU wavefront execution 11559 mode, omit. 11560 - Ensures that 11561 following 11562 loads will not see 11563 stale data. 11564 11565 atomicrmw acquire - agent - global 1. buffer/global_atomic 11566 - system 2. s_waitcnt vm/vscnt(0) 11567 11568 - Use vmcnt(0) if atomic with 11569 return and vscnt(0) if 11570 atomic with no-return. 11571 - Must happen before 11572 following 11573 buffer_gl*_inv. 11574 - Ensures the 11575 atomicrmw has 11576 completed before 11577 invalidating the 11578 caches. 11579 11580 3. buffer_gl0_inv; 11581 buffer_gl1_inv 11582 11583 - Must happen before 11584 any following 11585 global/generic 11586 load/load 11587 atomic/atomicrmw. 11588 - Ensures that 11589 following loads 11590 will not see stale 11591 global data. 11592 11593 atomicrmw acquire - agent - generic 1. flat_atomic 11594 - system 2. s_waitcnt vm/vscnt(0) & 11595 lgkmcnt(0) 11596 11597 - If OpenCL, omit 11598 lgkmcnt(0). 11599 - Use vmcnt(0) if atomic with 11600 return and vscnt(0) if 11601 atomic with no-return. 11602 - Must happen before 11603 following 11604 buffer_gl*_inv. 11605 - Ensures the 11606 atomicrmw has 11607 completed before 11608 invalidating the 11609 caches. 11610 11611 3. buffer_gl0_inv; 11612 buffer_gl1_inv 11613 11614 - Must happen before 11615 any following 11616 global/generic 11617 load/load 11618 atomic/atomicrmw. 11619 - Ensures that 11620 following loads 11621 will not see stale 11622 global data. 11623 11624 fence acquire - singlethread *none* *none* 11625 - wavefront 11626 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 11627 vmcnt(0) & vscnt(0) 11628 11629 - If CU wavefront execution 11630 mode, omit vmcnt(0) and 11631 vscnt(0). 11632 - If OpenCL and 11633 address space is 11634 not generic, omit 11635 lgkmcnt(0). 11636 - If OpenCL and 11637 address space is 11638 local, omit 11639 vmcnt(0) and vscnt(0). 11640 - However, since LLVM 11641 currently has no 11642 address space on 11643 the fence need to 11644 conservatively 11645 always generate. If 11646 fence had an 11647 address space then 11648 set to address 11649 space of OpenCL 11650 fence flag, or to 11651 generic if both 11652 local and global 11653 flags are 11654 specified. 11655 - Could be split into 11656 separate s_waitcnt 11657 vmcnt(0), s_waitcnt 11658 vscnt(0) and s_waitcnt 11659 lgkmcnt(0) to allow 11660 them to be 11661 independently moved 11662 according to the 11663 following rules. 11664 - s_waitcnt vmcnt(0) 11665 must happen after 11666 any preceding 11667 global/generic load 11668 atomic/ 11669 atomicrmw-with-return-value 11670 with an equal or 11671 wider sync scope 11672 and memory ordering 11673 stronger than 11674 unordered (this is 11675 termed the 11676 fence-paired-atomic). 11677 - s_waitcnt vscnt(0) 11678 must happen after 11679 any preceding 11680 global/generic 11681 atomicrmw-no-return-value 11682 with an equal or 11683 wider sync scope 11684 and memory ordering 11685 stronger than 11686 unordered (this is 11687 termed the 11688 fence-paired-atomic). 11689 - s_waitcnt lgkmcnt(0) 11690 must happen after 11691 any preceding 11692 local/generic load 11693 atomic/atomicrmw 11694 with an equal or 11695 wider sync scope 11696 and memory ordering 11697 stronger than 11698 unordered (this is 11699 termed the 11700 fence-paired-atomic). 11701 - Must happen before 11702 the following 11703 buffer_gl0_inv. 11704 - Ensures that the 11705 fence-paired atomic 11706 has completed 11707 before invalidating 11708 the 11709 cache. Therefore 11710 any following 11711 locations read must 11712 be no older than 11713 the value read by 11714 the 11715 fence-paired-atomic. 11716 11717 3. buffer_gl0_inv 11718 11719 - If CU wavefront execution 11720 mode, omit. 11721 - Ensures that 11722 following 11723 loads will not see 11724 stale data. 11725 11726 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 11727 - system vmcnt(0) & vscnt(0) 11728 11729 - If OpenCL and 11730 address space is 11731 not generic, omit 11732 lgkmcnt(0). 11733 - If OpenCL and 11734 address space is 11735 local, omit 11736 vmcnt(0) and vscnt(0). 11737 - However, since LLVM 11738 currently has no 11739 address space on 11740 the fence need to 11741 conservatively 11742 always generate 11743 (see comment for 11744 previous fence). 11745 - Could be split into 11746 separate s_waitcnt 11747 vmcnt(0), s_waitcnt 11748 vscnt(0) and s_waitcnt 11749 lgkmcnt(0) to allow 11750 them to be 11751 independently moved 11752 according to the 11753 following rules. 11754 - s_waitcnt vmcnt(0) 11755 must happen after 11756 any preceding 11757 global/generic load 11758 atomic/ 11759 atomicrmw-with-return-value 11760 with an equal or 11761 wider sync scope 11762 and memory ordering 11763 stronger than 11764 unordered (this is 11765 termed the 11766 fence-paired-atomic). 11767 - s_waitcnt vscnt(0) 11768 must happen after 11769 any preceding 11770 global/generic 11771 atomicrmw-no-return-value 11772 with an equal or 11773 wider sync scope 11774 and memory ordering 11775 stronger than 11776 unordered (this is 11777 termed the 11778 fence-paired-atomic). 11779 - s_waitcnt lgkmcnt(0) 11780 must happen after 11781 any preceding 11782 local/generic load 11783 atomic/atomicrmw 11784 with an equal or 11785 wider sync scope 11786 and memory ordering 11787 stronger than 11788 unordered (this is 11789 termed the 11790 fence-paired-atomic). 11791 - Must happen before 11792 the following 11793 buffer_gl*_inv. 11794 - Ensures that the 11795 fence-paired atomic 11796 has completed 11797 before invalidating 11798 the 11799 caches. Therefore 11800 any following 11801 locations read must 11802 be no older than 11803 the value read by 11804 the 11805 fence-paired-atomic. 11806 11807 2. buffer_gl0_inv; 11808 buffer_gl1_inv 11809 11810 - Must happen before any 11811 following global/generic 11812 load/load 11813 atomic/store/store 11814 atomic/atomicrmw. 11815 - Ensures that 11816 following loads 11817 will not see stale 11818 global data. 11819 11820 **Release Atomic** 11821 ------------------------------------------------------------------------------------ 11822 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 11823 - wavefront - local 11824 - generic 11825 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11826 - generic vmcnt(0) & vscnt(0) 11827 11828 - If CU wavefront execution 11829 mode, omit vmcnt(0) and 11830 vscnt(0). 11831 - If OpenCL, omit 11832 lgkmcnt(0). 11833 - Could be split into 11834 separate s_waitcnt 11835 vmcnt(0), s_waitcnt 11836 vscnt(0) and s_waitcnt 11837 lgkmcnt(0) to allow 11838 them to be 11839 independently moved 11840 according to the 11841 following rules. 11842 - s_waitcnt vmcnt(0) 11843 must happen after 11844 any preceding 11845 global/generic load/load 11846 atomic/ 11847 atomicrmw-with-return-value. 11848 - s_waitcnt vscnt(0) 11849 must happen after 11850 any preceding 11851 global/generic 11852 store/store 11853 atomic/ 11854 atomicrmw-no-return-value. 11855 - s_waitcnt lgkmcnt(0) 11856 must happen after 11857 any preceding 11858 local/generic 11859 load/store/load 11860 atomic/store 11861 atomic/atomicrmw. 11862 - Must happen before 11863 the following 11864 store. 11865 - Ensures that all 11866 memory operations 11867 have 11868 completed before 11869 performing the 11870 store that is being 11871 released. 11872 11873 2. buffer/global/flat_store 11874 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 11875 11876 - If CU wavefront execution 11877 mode, omit. 11878 - If OpenCL, omit. 11879 - Could be split into 11880 separate s_waitcnt 11881 vmcnt(0) and s_waitcnt 11882 vscnt(0) to allow 11883 them to be 11884 independently moved 11885 according to the 11886 following rules. 11887 - s_waitcnt vmcnt(0) 11888 must happen after 11889 any preceding 11890 global/generic load/load 11891 atomic/ 11892 atomicrmw-with-return-value. 11893 - s_waitcnt vscnt(0) 11894 must happen after 11895 any preceding 11896 global/generic 11897 store/store atomic/ 11898 atomicrmw-no-return-value. 11899 - Must happen before 11900 the following 11901 store. 11902 - Ensures that all 11903 global memory 11904 operations have 11905 completed before 11906 performing the 11907 store that is being 11908 released. 11909 11910 2. ds_store 11911 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 11912 - system - generic vmcnt(0) & vscnt(0) 11913 11914 - If OpenCL and 11915 address space is 11916 not generic, omit 11917 lgkmcnt(0). 11918 - Could be split into 11919 separate s_waitcnt 11920 vmcnt(0), s_waitcnt vscnt(0) 11921 and s_waitcnt 11922 lgkmcnt(0) to allow 11923 them to be 11924 independently moved 11925 according to the 11926 following rules. 11927 - s_waitcnt vmcnt(0) 11928 must happen after 11929 any preceding 11930 global/generic 11931 load/load 11932 atomic/ 11933 atomicrmw-with-return-value. 11934 - s_waitcnt vscnt(0) 11935 must happen after 11936 any preceding 11937 global/generic 11938 store/store atomic/ 11939 atomicrmw-no-return-value. 11940 - s_waitcnt lgkmcnt(0) 11941 must happen after 11942 any preceding 11943 local/generic 11944 load/store/load 11945 atomic/store 11946 atomic/atomicrmw. 11947 - Must happen before 11948 the following 11949 store. 11950 - Ensures that all 11951 memory operations 11952 have 11953 completed before 11954 performing the 11955 store that is being 11956 released. 11957 11958 2. buffer/global/flat_store 11959 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 11960 - wavefront - local 11961 - generic 11962 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11963 - generic vmcnt(0) & vscnt(0) 11964 11965 - If CU wavefront execution 11966 mode, omit vmcnt(0) and 11967 vscnt(0). 11968 - If OpenCL, omit lgkmcnt(0). 11969 - Could be split into 11970 separate s_waitcnt 11971 vmcnt(0), s_waitcnt 11972 vscnt(0) and s_waitcnt 11973 lgkmcnt(0) to allow 11974 them to be 11975 independently moved 11976 according to the 11977 following rules. 11978 - s_waitcnt vmcnt(0) 11979 must happen after 11980 any preceding 11981 global/generic load/load 11982 atomic/ 11983 atomicrmw-with-return-value. 11984 - s_waitcnt vscnt(0) 11985 must happen after 11986 any preceding 11987 global/generic 11988 store/store 11989 atomic/ 11990 atomicrmw-no-return-value. 11991 - s_waitcnt lgkmcnt(0) 11992 must happen after 11993 any preceding 11994 local/generic 11995 load/store/load 11996 atomic/store 11997 atomic/atomicrmw. 11998 - Must happen before 11999 the following 12000 atomicrmw. 12001 - Ensures that all 12002 memory operations 12003 have 12004 completed before 12005 performing the 12006 atomicrmw that is 12007 being released. 12008 12009 2. buffer/global/flat_atomic 12010 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12011 12012 - If CU wavefront execution 12013 mode, omit. 12014 - If OpenCL, omit. 12015 - Could be split into 12016 separate s_waitcnt 12017 vmcnt(0) and s_waitcnt 12018 vscnt(0) to allow 12019 them to be 12020 independently moved 12021 according to the 12022 following rules. 12023 - s_waitcnt vmcnt(0) 12024 must happen after 12025 any preceding 12026 global/generic load/load 12027 atomic/ 12028 atomicrmw-with-return-value. 12029 - s_waitcnt vscnt(0) 12030 must happen after 12031 any preceding 12032 global/generic 12033 store/store atomic/ 12034 atomicrmw-no-return-value. 12035 - Must happen before 12036 the following 12037 store. 12038 - Ensures that all 12039 global memory 12040 operations have 12041 completed before 12042 performing the 12043 store that is being 12044 released. 12045 12046 2. ds_atomic 12047 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 12048 - system - generic vmcnt(0) & vscnt(0) 12049 12050 - If OpenCL, omit 12051 lgkmcnt(0). 12052 - Could be split into 12053 separate s_waitcnt 12054 vmcnt(0), s_waitcnt 12055 vscnt(0) and s_waitcnt 12056 lgkmcnt(0) to allow 12057 them to be 12058 independently moved 12059 according to the 12060 following rules. 12061 - s_waitcnt vmcnt(0) 12062 must happen after 12063 any preceding 12064 global/generic 12065 load/load atomic/ 12066 atomicrmw-with-return-value. 12067 - s_waitcnt vscnt(0) 12068 must happen after 12069 any preceding 12070 global/generic 12071 store/store atomic/ 12072 atomicrmw-no-return-value. 12073 - s_waitcnt lgkmcnt(0) 12074 must happen after 12075 any preceding 12076 local/generic 12077 load/store/load 12078 atomic/store 12079 atomic/atomicrmw. 12080 - Must happen before 12081 the following 12082 atomicrmw. 12083 - Ensures that all 12084 memory operations 12085 to global and local 12086 have completed 12087 before performing 12088 the atomicrmw that 12089 is being released. 12090 12091 2. buffer/global/flat_atomic 12092 fence release - singlethread *none* *none* 12093 - wavefront 12094 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12095 vmcnt(0) & vscnt(0) 12096 12097 - If CU wavefront execution 12098 mode, omit vmcnt(0) and 12099 vscnt(0). 12100 - If OpenCL and 12101 address space is 12102 not generic, omit 12103 lgkmcnt(0). 12104 - If OpenCL and 12105 address space is 12106 local, omit 12107 vmcnt(0) and vscnt(0). 12108 - However, since LLVM 12109 currently has no 12110 address space on 12111 the fence need to 12112 conservatively 12113 always generate. If 12114 fence had an 12115 address space then 12116 set to address 12117 space of OpenCL 12118 fence flag, or to 12119 generic if both 12120 local and global 12121 flags are 12122 specified. 12123 - Could be split into 12124 separate s_waitcnt 12125 vmcnt(0), s_waitcnt 12126 vscnt(0) and s_waitcnt 12127 lgkmcnt(0) to allow 12128 them to be 12129 independently moved 12130 according to the 12131 following rules. 12132 - s_waitcnt vmcnt(0) 12133 must happen after 12134 any preceding 12135 global/generic 12136 load/load 12137 atomic/ 12138 atomicrmw-with-return-value. 12139 - s_waitcnt vscnt(0) 12140 must happen after 12141 any preceding 12142 global/generic 12143 store/store atomic/ 12144 atomicrmw-no-return-value. 12145 - s_waitcnt lgkmcnt(0) 12146 must happen after 12147 any preceding 12148 local/generic 12149 load/store/load 12150 atomic/store atomic/ 12151 atomicrmw. 12152 - Must happen before 12153 any following store 12154 atomic/atomicrmw 12155 with an equal or 12156 wider sync scope 12157 and memory ordering 12158 stronger than 12159 unordered (this is 12160 termed the 12161 fence-paired-atomic). 12162 - Ensures that all 12163 memory operations 12164 have 12165 completed before 12166 performing the 12167 following 12168 fence-paired-atomic. 12169 12170 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 12171 - system vmcnt(0) & vscnt(0) 12172 12173 - If OpenCL and 12174 address space is 12175 not generic, omit 12176 lgkmcnt(0). 12177 - If OpenCL and 12178 address space is 12179 local, omit 12180 vmcnt(0) and vscnt(0). 12181 - However, since LLVM 12182 currently has no 12183 address space on 12184 the fence need to 12185 conservatively 12186 always generate. If 12187 fence had an 12188 address space then 12189 set to address 12190 space of OpenCL 12191 fence flag, or to 12192 generic if both 12193 local and global 12194 flags are 12195 specified. 12196 - Could be split into 12197 separate s_waitcnt 12198 vmcnt(0), s_waitcnt 12199 vscnt(0) and s_waitcnt 12200 lgkmcnt(0) to allow 12201 them to be 12202 independently moved 12203 according to the 12204 following rules. 12205 - s_waitcnt vmcnt(0) 12206 must happen after 12207 any preceding 12208 global/generic 12209 load/load atomic/ 12210 atomicrmw-with-return-value. 12211 - s_waitcnt vscnt(0) 12212 must happen after 12213 any preceding 12214 global/generic 12215 store/store atomic/ 12216 atomicrmw-no-return-value. 12217 - s_waitcnt lgkmcnt(0) 12218 must happen after 12219 any preceding 12220 local/generic 12221 load/store/load 12222 atomic/store 12223 atomic/atomicrmw. 12224 - Must happen before 12225 any following store 12226 atomic/atomicrmw 12227 with an equal or 12228 wider sync scope 12229 and memory ordering 12230 stronger than 12231 unordered (this is 12232 termed the 12233 fence-paired-atomic). 12234 - Ensures that all 12235 memory operations 12236 have 12237 completed before 12238 performing the 12239 following 12240 fence-paired-atomic. 12241 12242 **Acquire-Release Atomic** 12243 ------------------------------------------------------------------------------------ 12244 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 12245 - wavefront - local 12246 - generic 12247 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12248 vmcnt(0) & vscnt(0) 12249 12250 - If CU wavefront execution 12251 mode, omit vmcnt(0) and 12252 vscnt(0). 12253 - If OpenCL, omit 12254 lgkmcnt(0). 12255 - Must happen after 12256 any preceding 12257 local/generic 12258 load/store/load 12259 atomic/store 12260 atomic/atomicrmw. 12261 - Could be split into 12262 separate s_waitcnt 12263 vmcnt(0), s_waitcnt 12264 vscnt(0), and s_waitcnt 12265 lgkmcnt(0) to allow 12266 them to be 12267 independently moved 12268 according to the 12269 following rules. 12270 - s_waitcnt vmcnt(0) 12271 must happen after 12272 any preceding 12273 global/generic load/load 12274 atomic/ 12275 atomicrmw-with-return-value. 12276 - s_waitcnt vscnt(0) 12277 must happen after 12278 any preceding 12279 global/generic 12280 store/store 12281 atomic/ 12282 atomicrmw-no-return-value. 12283 - s_waitcnt lgkmcnt(0) 12284 must happen after 12285 any preceding 12286 local/generic 12287 load/store/load 12288 atomic/store 12289 atomic/atomicrmw. 12290 - Must happen before 12291 the following 12292 atomicrmw. 12293 - Ensures that all 12294 memory operations 12295 have 12296 completed before 12297 performing the 12298 atomicrmw that is 12299 being released. 12300 12301 2. buffer/global_atomic 12302 3. s_waitcnt vm/vscnt(0) 12303 12304 - If CU wavefront execution 12305 mode, omit. 12306 - Use vmcnt(0) if atomic with 12307 return and vscnt(0) if 12308 atomic with no-return. 12309 - Must happen before 12310 the following 12311 buffer_gl0_inv. 12312 - Ensures any 12313 following global 12314 data read is no 12315 older than the 12316 atomicrmw value 12317 being acquired. 12318 12319 4. buffer_gl0_inv 12320 12321 - If CU wavefront execution 12322 mode, omit. 12323 - Ensures that 12324 following 12325 loads will not see 12326 stale data. 12327 12328 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12329 12330 - If CU wavefront execution 12331 mode, omit. 12332 - If OpenCL, omit. 12333 - Could be split into 12334 separate s_waitcnt 12335 vmcnt(0) and s_waitcnt 12336 vscnt(0) to allow 12337 them to be 12338 independently moved 12339 according to the 12340 following rules. 12341 - s_waitcnt vmcnt(0) 12342 must happen after 12343 any preceding 12344 global/generic load/load 12345 atomic/ 12346 atomicrmw-with-return-value. 12347 - s_waitcnt vscnt(0) 12348 must happen after 12349 any preceding 12350 global/generic 12351 store/store atomic/ 12352 atomicrmw-no-return-value. 12353 - Must happen before 12354 the following 12355 store. 12356 - Ensures that all 12357 global memory 12358 operations have 12359 completed before 12360 performing the 12361 store that is being 12362 released. 12363 12364 2. ds_atomic 12365 3. s_waitcnt lgkmcnt(0) 12366 12367 - If OpenCL, omit. 12368 - Must happen before 12369 the following 12370 buffer_gl0_inv. 12371 - Ensures any 12372 following global 12373 data read is no 12374 older than the local load 12375 atomic value being 12376 acquired. 12377 12378 4. buffer_gl0_inv 12379 12380 - If CU wavefront execution 12381 mode, omit. 12382 - If OpenCL omit. 12383 - Ensures that 12384 following 12385 loads will not see 12386 stale data. 12387 12388 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 12389 vmcnt(0) & vscnt(0) 12390 12391 - If CU wavefront execution 12392 mode, omit vmcnt(0) and 12393 vscnt(0). 12394 - If OpenCL, omit lgkmcnt(0). 12395 - Could be split into 12396 separate s_waitcnt 12397 vmcnt(0), s_waitcnt 12398 vscnt(0) and s_waitcnt 12399 lgkmcnt(0) to allow 12400 them to be 12401 independently moved 12402 according to the 12403 following rules. 12404 - s_waitcnt vmcnt(0) 12405 must happen after 12406 any preceding 12407 global/generic load/load 12408 atomic/ 12409 atomicrmw-with-return-value. 12410 - s_waitcnt vscnt(0) 12411 must happen after 12412 any preceding 12413 global/generic 12414 store/store 12415 atomic/ 12416 atomicrmw-no-return-value. 12417 - s_waitcnt lgkmcnt(0) 12418 must happen after 12419 any preceding 12420 local/generic 12421 load/store/load 12422 atomic/store 12423 atomic/atomicrmw. 12424 - Must happen before 12425 the following 12426 atomicrmw. 12427 - Ensures that all 12428 memory operations 12429 have 12430 completed before 12431 performing the 12432 atomicrmw that is 12433 being released. 12434 12435 2. flat_atomic 12436 3. s_waitcnt lgkmcnt(0) & 12437 vmcnt(0) & vscnt(0) 12438 12439 - If CU wavefront execution 12440 mode, omit vmcnt(0) and 12441 vscnt(0). 12442 - If OpenCL, omit lgkmcnt(0). 12443 - Must happen before 12444 the following 12445 buffer_gl0_inv. 12446 - Ensures any 12447 following global 12448 data read is no 12449 older than the load 12450 atomic value being 12451 acquired. 12452 12453 3. buffer_gl0_inv 12454 12455 - If CU wavefront execution 12456 mode, omit. 12457 - Ensures that 12458 following 12459 loads will not see 12460 stale data. 12461 12462 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 12463 - system vmcnt(0) & vscnt(0) 12464 12465 - If OpenCL, omit 12466 lgkmcnt(0). 12467 - Could be split into 12468 separate s_waitcnt 12469 vmcnt(0), s_waitcnt 12470 vscnt(0) and s_waitcnt 12471 lgkmcnt(0) to allow 12472 them to be 12473 independently moved 12474 according to the 12475 following rules. 12476 - s_waitcnt vmcnt(0) 12477 must happen after 12478 any preceding 12479 global/generic 12480 load/load atomic/ 12481 atomicrmw-with-return-value. 12482 - s_waitcnt vscnt(0) 12483 must happen after 12484 any preceding 12485 global/generic 12486 store/store atomic/ 12487 atomicrmw-no-return-value. 12488 - s_waitcnt lgkmcnt(0) 12489 must happen after 12490 any preceding 12491 local/generic 12492 load/store/load 12493 atomic/store 12494 atomic/atomicrmw. 12495 - Must happen before 12496 the following 12497 atomicrmw. 12498 - Ensures that all 12499 memory operations 12500 to global have 12501 completed before 12502 performing the 12503 atomicrmw that is 12504 being released. 12505 12506 2. buffer/global_atomic 12507 3. s_waitcnt vm/vscnt(0) 12508 12509 - Use vmcnt(0) if atomic with 12510 return and vscnt(0) if 12511 atomic with no-return. 12512 - Must happen before 12513 following 12514 buffer_gl*_inv. 12515 - Ensures the 12516 atomicrmw has 12517 completed before 12518 invalidating the 12519 caches. 12520 12521 4. buffer_gl0_inv; 12522 buffer_gl1_inv 12523 12524 - Must happen before 12525 any following 12526 global/generic 12527 load/load 12528 atomic/atomicrmw. 12529 - Ensures that 12530 following loads 12531 will not see stale 12532 global data. 12533 12534 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 12535 - system vmcnt(0) & vscnt(0) 12536 12537 - If OpenCL, omit 12538 lgkmcnt(0). 12539 - Could be split into 12540 separate s_waitcnt 12541 vmcnt(0), s_waitcnt 12542 vscnt(0), and s_waitcnt 12543 lgkmcnt(0) to allow 12544 them to be 12545 independently moved 12546 according to the 12547 following rules. 12548 - s_waitcnt vmcnt(0) 12549 must happen after 12550 any preceding 12551 global/generic 12552 load/load atomic 12553 atomicrmw-with-return-value. 12554 - s_waitcnt vscnt(0) 12555 must happen after 12556 any preceding 12557 global/generic 12558 store/store atomic/ 12559 atomicrmw-no-return-value. 12560 - s_waitcnt lgkmcnt(0) 12561 must happen after 12562 any preceding 12563 local/generic 12564 load/store/load 12565 atomic/store 12566 atomic/atomicrmw. 12567 - Must happen before 12568 the following 12569 atomicrmw. 12570 - Ensures that all 12571 memory operations 12572 have 12573 completed before 12574 performing the 12575 atomicrmw that is 12576 being released. 12577 12578 2. flat_atomic 12579 3. s_waitcnt vm/vscnt(0) & 12580 lgkmcnt(0) 12581 12582 - If OpenCL, omit 12583 lgkmcnt(0). 12584 - Use vmcnt(0) if atomic with 12585 return and vscnt(0) if 12586 atomic with no-return. 12587 - Must happen before 12588 following 12589 buffer_gl*_inv. 12590 - Ensures the 12591 atomicrmw has 12592 completed before 12593 invalidating the 12594 caches. 12595 12596 4. buffer_gl0_inv; 12597 buffer_gl1_inv 12598 12599 - Must happen before 12600 any following 12601 global/generic 12602 load/load 12603 atomic/atomicrmw. 12604 - Ensures that 12605 following loads 12606 will not see stale 12607 global data. 12608 12609 fence acq_rel - singlethread *none* *none* 12610 - wavefront 12611 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12612 vmcnt(0) & vscnt(0) 12613 12614 - If CU wavefront execution 12615 mode, omit vmcnt(0) and 12616 vscnt(0). 12617 - If OpenCL and 12618 address space is 12619 not generic, omit 12620 lgkmcnt(0). 12621 - If OpenCL and 12622 address space is 12623 local, omit 12624 vmcnt(0) and vscnt(0). 12625 - However, 12626 since LLVM 12627 currently has no 12628 address space on 12629 the fence need to 12630 conservatively 12631 always generate 12632 (see comment for 12633 previous fence). 12634 - Could be split into 12635 separate s_waitcnt 12636 vmcnt(0), s_waitcnt 12637 vscnt(0) and s_waitcnt 12638 lgkmcnt(0) to allow 12639 them to be 12640 independently moved 12641 according to the 12642 following rules. 12643 - s_waitcnt vmcnt(0) 12644 must happen after 12645 any preceding 12646 global/generic 12647 load/load 12648 atomic/ 12649 atomicrmw-with-return-value. 12650 - s_waitcnt vscnt(0) 12651 must happen after 12652 any preceding 12653 global/generic 12654 store/store atomic/ 12655 atomicrmw-no-return-value. 12656 - s_waitcnt lgkmcnt(0) 12657 must happen after 12658 any preceding 12659 local/generic 12660 load/store/load 12661 atomic/store atomic/ 12662 atomicrmw. 12663 - Must happen before 12664 any following 12665 global/generic 12666 load/load 12667 atomic/store/store 12668 atomic/atomicrmw. 12669 - Ensures that all 12670 memory operations 12671 have 12672 completed before 12673 performing any 12674 following global 12675 memory operations. 12676 - Ensures that the 12677 preceding 12678 local/generic load 12679 atomic/atomicrmw 12680 with an equal or 12681 wider sync scope 12682 and memory ordering 12683 stronger than 12684 unordered (this is 12685 termed the 12686 acquire-fence-paired-atomic) 12687 has completed 12688 before following 12689 global memory 12690 operations. This 12691 satisfies the 12692 requirements of 12693 acquire. 12694 - Ensures that all 12695 previous memory 12696 operations have 12697 completed before a 12698 following 12699 local/generic store 12700 atomic/atomicrmw 12701 with an equal or 12702 wider sync scope 12703 and memory ordering 12704 stronger than 12705 unordered (this is 12706 termed the 12707 release-fence-paired-atomic). 12708 This satisfies the 12709 requirements of 12710 release. 12711 - Must happen before 12712 the following 12713 buffer_gl0_inv. 12714 - Ensures that the 12715 acquire-fence-paired 12716 atomic has completed 12717 before invalidating 12718 the 12719 cache. Therefore 12720 any following 12721 locations read must 12722 be no older than 12723 the value read by 12724 the 12725 acquire-fence-paired-atomic. 12726 12727 3. buffer_gl0_inv 12728 12729 - If CU wavefront execution 12730 mode, omit. 12731 - Ensures that 12732 following 12733 loads will not see 12734 stale data. 12735 12736 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 12737 - system vmcnt(0) & vscnt(0) 12738 12739 - If OpenCL and 12740 address space is 12741 not generic, omit 12742 lgkmcnt(0). 12743 - If OpenCL and 12744 address space is 12745 local, omit 12746 vmcnt(0) and vscnt(0). 12747 - However, since LLVM 12748 currently has no 12749 address space on 12750 the fence need to 12751 conservatively 12752 always generate 12753 (see comment for 12754 previous fence). 12755 - Could be split into 12756 separate s_waitcnt 12757 vmcnt(0), s_waitcnt 12758 vscnt(0) and s_waitcnt 12759 lgkmcnt(0) to allow 12760 them to be 12761 independently moved 12762 according to the 12763 following rules. 12764 - s_waitcnt vmcnt(0) 12765 must happen after 12766 any preceding 12767 global/generic 12768 load/load 12769 atomic/ 12770 atomicrmw-with-return-value. 12771 - s_waitcnt vscnt(0) 12772 must happen after 12773 any preceding 12774 global/generic 12775 store/store atomic/ 12776 atomicrmw-no-return-value. 12777 - s_waitcnt lgkmcnt(0) 12778 must happen after 12779 any preceding 12780 local/generic 12781 load/store/load 12782 atomic/store 12783 atomic/atomicrmw. 12784 - Must happen before 12785 the following 12786 buffer_gl*_inv. 12787 - Ensures that the 12788 preceding 12789 global/local/generic 12790 load 12791 atomic/atomicrmw 12792 with an equal or 12793 wider sync scope 12794 and memory ordering 12795 stronger than 12796 unordered (this is 12797 termed the 12798 acquire-fence-paired-atomic) 12799 has completed 12800 before invalidating 12801 the caches. This 12802 satisfies the 12803 requirements of 12804 acquire. 12805 - Ensures that all 12806 previous memory 12807 operations have 12808 completed before a 12809 following 12810 global/local/generic 12811 store 12812 atomic/atomicrmw 12813 with an equal or 12814 wider sync scope 12815 and memory ordering 12816 stronger than 12817 unordered (this is 12818 termed the 12819 release-fence-paired-atomic). 12820 This satisfies the 12821 requirements of 12822 release. 12823 12824 2. buffer_gl0_inv; 12825 buffer_gl1_inv 12826 12827 - Must happen before 12828 any following 12829 global/generic 12830 load/load 12831 atomic/store/store 12832 atomic/atomicrmw. 12833 - Ensures that 12834 following loads 12835 will not see stale 12836 global data. This 12837 satisfies the 12838 requirements of 12839 acquire. 12840 12841 **Sequential Consistent Atomic** 12842 ------------------------------------------------------------------------------------ 12843 load atomic seq_cst - singlethread - global *Same as corresponding 12844 - wavefront - local load atomic acquire, 12845 - generic except must generate 12846 all instructions even 12847 for OpenCL.* 12848 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12849 - generic vmcnt(0) & vscnt(0) 12850 12851 - If CU wavefront execution 12852 mode, omit vmcnt(0) and 12853 vscnt(0). 12854 - Could be split into 12855 separate s_waitcnt 12856 vmcnt(0), s_waitcnt 12857 vscnt(0), and s_waitcnt 12858 lgkmcnt(0) to allow 12859 them to be 12860 independently moved 12861 according to the 12862 following rules. 12863 - s_waitcnt lgkmcnt(0) must 12864 happen after 12865 preceding 12866 local/generic load 12867 atomic/store 12868 atomic/atomicrmw 12869 with memory 12870 ordering of seq_cst 12871 and with equal or 12872 wider sync scope. 12873 (Note that seq_cst 12874 fences have their 12875 own s_waitcnt 12876 lgkmcnt(0) and so do 12877 not need to be 12878 considered.) 12879 - s_waitcnt vmcnt(0) 12880 must happen after 12881 preceding 12882 global/generic load 12883 atomic/ 12884 atomicrmw-with-return-value 12885 with memory 12886 ordering of seq_cst 12887 and with equal or 12888 wider sync scope. 12889 (Note that seq_cst 12890 fences have their 12891 own s_waitcnt 12892 vmcnt(0) and so do 12893 not need to be 12894 considered.) 12895 - s_waitcnt vscnt(0) 12896 Must happen after 12897 preceding 12898 global/generic store 12899 atomic/ 12900 atomicrmw-no-return-value 12901 with memory 12902 ordering of seq_cst 12903 and with equal or 12904 wider sync scope. 12905 (Note that seq_cst 12906 fences have their 12907 own s_waitcnt 12908 vscnt(0) and so do 12909 not need to be 12910 considered.) 12911 - Ensures any 12912 preceding 12913 sequential 12914 consistent global/local 12915 memory instructions 12916 have completed 12917 before executing 12918 this sequentially 12919 consistent 12920 instruction. This 12921 prevents reordering 12922 a seq_cst store 12923 followed by a 12924 seq_cst load. (Note 12925 that seq_cst is 12926 stronger than 12927 acquire/release as 12928 the reordering of 12929 load acquire 12930 followed by a store 12931 release is 12932 prevented by the 12933 s_waitcnt of 12934 the release, but 12935 there is nothing 12936 preventing a store 12937 release followed by 12938 load acquire from 12939 completing out of 12940 order. The s_waitcnt 12941 could be placed after 12942 seq_store or before 12943 the seq_load. We 12944 choose the load to 12945 make the s_waitcnt be 12946 as late as possible 12947 so that the store 12948 may have already 12949 completed.) 12950 12951 2. *Following 12952 instructions same as 12953 corresponding load 12954 atomic acquire, 12955 except must generate 12956 all instructions even 12957 for OpenCL.* 12958 load atomic seq_cst - workgroup - local 12959 12960 1. s_waitcnt vmcnt(0) & vscnt(0) 12961 12962 - If CU wavefront execution 12963 mode, omit. 12964 - Could be split into 12965 separate s_waitcnt 12966 vmcnt(0) and s_waitcnt 12967 vscnt(0) to allow 12968 them to be 12969 independently moved 12970 according to the 12971 following rules. 12972 - s_waitcnt vmcnt(0) 12973 Must happen after 12974 preceding 12975 global/generic load 12976 atomic/ 12977 atomicrmw-with-return-value 12978 with memory 12979 ordering of seq_cst 12980 and with equal or 12981 wider sync scope. 12982 (Note that seq_cst 12983 fences have their 12984 own s_waitcnt 12985 vmcnt(0) and so do 12986 not need to be 12987 considered.) 12988 - s_waitcnt vscnt(0) 12989 Must happen after 12990 preceding 12991 global/generic store 12992 atomic/ 12993 atomicrmw-no-return-value 12994 with memory 12995 ordering of seq_cst 12996 and with equal or 12997 wider sync scope. 12998 (Note that seq_cst 12999 fences have their 13000 own s_waitcnt 13001 vscnt(0) and so do 13002 not need to be 13003 considered.) 13004 - Ensures any 13005 preceding 13006 sequential 13007 consistent global 13008 memory instructions 13009 have completed 13010 before executing 13011 this sequentially 13012 consistent 13013 instruction. This 13014 prevents reordering 13015 a seq_cst store 13016 followed by a 13017 seq_cst load. (Note 13018 that seq_cst is 13019 stronger than 13020 acquire/release as 13021 the reordering of 13022 load acquire 13023 followed by a store 13024 release is 13025 prevented by the 13026 s_waitcnt of 13027 the release, but 13028 there is nothing 13029 preventing a store 13030 release followed by 13031 load acquire from 13032 completing out of 13033 order. The s_waitcnt 13034 could be placed after 13035 seq_store or before 13036 the seq_load. We 13037 choose the load to 13038 make the s_waitcnt be 13039 as late as possible 13040 so that the store 13041 may have already 13042 completed.) 13043 13044 2. *Following 13045 instructions same as 13046 corresponding load 13047 atomic acquire, 13048 except must generate 13049 all instructions even 13050 for OpenCL.* 13051 13052 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 13053 - system - generic vmcnt(0) & vscnt(0) 13054 13055 - Could be split into 13056 separate s_waitcnt 13057 vmcnt(0), s_waitcnt 13058 vscnt(0) and s_waitcnt 13059 lgkmcnt(0) to allow 13060 them to be 13061 independently moved 13062 according to the 13063 following rules. 13064 - s_waitcnt lgkmcnt(0) 13065 must happen after 13066 preceding 13067 local load 13068 atomic/store 13069 atomic/atomicrmw 13070 with memory 13071 ordering of seq_cst 13072 and with equal or 13073 wider sync scope. 13074 (Note that seq_cst 13075 fences have their 13076 own s_waitcnt 13077 lgkmcnt(0) and so do 13078 not need to be 13079 considered.) 13080 - s_waitcnt vmcnt(0) 13081 must happen after 13082 preceding 13083 global/generic load 13084 atomic/ 13085 atomicrmw-with-return-value 13086 with memory 13087 ordering of seq_cst 13088 and with equal or 13089 wider sync scope. 13090 (Note that seq_cst 13091 fences have their 13092 own s_waitcnt 13093 vmcnt(0) and so do 13094 not need to be 13095 considered.) 13096 - s_waitcnt vscnt(0) 13097 Must happen after 13098 preceding 13099 global/generic store 13100 atomic/ 13101 atomicrmw-no-return-value 13102 with memory 13103 ordering of seq_cst 13104 and with equal or 13105 wider sync scope. 13106 (Note that seq_cst 13107 fences have their 13108 own s_waitcnt 13109 vscnt(0) and so do 13110 not need to be 13111 considered.) 13112 - Ensures any 13113 preceding 13114 sequential 13115 consistent global 13116 memory instructions 13117 have completed 13118 before executing 13119 this sequentially 13120 consistent 13121 instruction. This 13122 prevents reordering 13123 a seq_cst store 13124 followed by a 13125 seq_cst load. (Note 13126 that seq_cst is 13127 stronger than 13128 acquire/release as 13129 the reordering of 13130 load acquire 13131 followed by a store 13132 release is 13133 prevented by the 13134 s_waitcnt of 13135 the release, but 13136 there is nothing 13137 preventing a store 13138 release followed by 13139 load acquire from 13140 completing out of 13141 order. The s_waitcnt 13142 could be placed after 13143 seq_store or before 13144 the seq_load. We 13145 choose the load to 13146 make the s_waitcnt be 13147 as late as possible 13148 so that the store 13149 may have already 13150 completed.) 13151 13152 2. *Following 13153 instructions same as 13154 corresponding load 13155 atomic acquire, 13156 except must generate 13157 all instructions even 13158 for OpenCL.* 13159 store atomic seq_cst - singlethread - global *Same as corresponding 13160 - wavefront - local store atomic release, 13161 - workgroup - generic except must generate 13162 - agent all instructions even 13163 - system for OpenCL.* 13164 atomicrmw seq_cst - singlethread - global *Same as corresponding 13165 - wavefront - local atomicrmw acq_rel, 13166 - workgroup - generic except must generate 13167 - agent all instructions even 13168 - system for OpenCL.* 13169 fence seq_cst - singlethread *none* *Same as corresponding 13170 - wavefront fence acq_rel, 13171 - workgroup except must generate 13172 - agent all instructions even 13173 - system for OpenCL.* 13174 ============ ============ ============== ========== ================================ 13175 13176.. _amdgpu-amdhsa-trap-handler-abi: 13177 13178Trap Handler ABI 13179~~~~~~~~~~~~~~~~ 13180 13181For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 13182runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 13183supports the ``s_trap`` instruction. For usage see: 13184 13185- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 13186- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 13187- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table` 13188 13189 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 13190 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 13191 13192 =================== =============== =============== ======================================= 13193 Usage Code Sequence Trap Handler Description 13194 Inputs 13195 =================== =============== =============== ======================================= 13196 reserved ``s_trap 0x00`` Reserved by hardware. 13197 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 13198 ``queue_ptr`` intrinsic (not implemented). 13199 ``VGPR0``: 13200 ``arg`` 13201 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13202 ``queue_ptr`` the trap instruction. The associated 13203 queue is signalled to put it into the 13204 error state. When the queue is put in 13205 the error state, the waves executing 13206 dispatches on the queue will be 13207 terminated. 13208 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13209 as a no-operation. The trap handler 13210 is entered and immediately returns to 13211 continue execution of the wavefront. 13212 - If the debugger is enabled, causes 13213 the debug trap to be reported by the 13214 debugger and the wavefront is put in 13215 the halt state with the PC at the 13216 instruction. The debugger must 13217 increment the PC and resume the wave. 13218 reserved ``s_trap 0x04`` Reserved. 13219 reserved ``s_trap 0x05`` Reserved. 13220 reserved ``s_trap 0x06`` Reserved. 13221 reserved ``s_trap 0x07`` Reserved. 13222 reserved ``s_trap 0x08`` Reserved. 13223 reserved ``s_trap 0xfe`` Reserved. 13224 reserved ``s_trap 0xff`` Reserved. 13225 =================== =============== =============== ======================================= 13226 13227.. 13228 13229 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 13230 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 13231 13232 =================== =============== =============== ======================================= 13233 Usage Code Sequence Trap Handler Description 13234 Inputs 13235 =================== =============== =============== ======================================= 13236 reserved ``s_trap 0x00`` Reserved by hardware. 13237 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 13238 breakpoints. Causes wave to be halted 13239 with the PC at the trap instruction. 13240 The debugger is responsible to resume 13241 the wave, including the instruction 13242 that the breakpoint overwrote. 13243 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13244 ``queue_ptr`` the trap instruction. The associated 13245 queue is signalled to put it into the 13246 error state. When the queue is put in 13247 the error state, the waves executing 13248 dispatches on the queue will be 13249 terminated. 13250 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13251 as a no-operation. The trap handler 13252 is entered and immediately returns to 13253 continue execution of the wavefront. 13254 - If the debugger is enabled, causes 13255 the debug trap to be reported by the 13256 debugger and the wavefront is put in 13257 the halt state with the PC at the 13258 instruction. The debugger must 13259 increment the PC and resume the wave. 13260 reserved ``s_trap 0x04`` Reserved. 13261 reserved ``s_trap 0x05`` Reserved. 13262 reserved ``s_trap 0x06`` Reserved. 13263 reserved ``s_trap 0x07`` Reserved. 13264 reserved ``s_trap 0x08`` Reserved. 13265 reserved ``s_trap 0xfe`` Reserved. 13266 reserved ``s_trap 0xff`` Reserved. 13267 =================== =============== =============== ======================================= 13268 13269.. 13270 13271 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 and Above 13272 :name: amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table 13273 13274 =================== =============== ================ ================= ======================================= 13275 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 13276 =================== =============== ================ ================= ======================================= 13277 reserved ``s_trap 0x00`` Reserved by hardware. 13278 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 13279 breakpoints. Causes wave to be halted 13280 with the PC at the trap instruction. 13281 The debugger is responsible to resume 13282 the wave, including the instruction 13283 that the breakpoint overwrote. 13284 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 13285 ``queue_ptr`` the trap instruction. The associated 13286 queue is signalled to put it into the 13287 error state. When the queue is put in 13288 the error state, the waves executing 13289 dispatches on the queue will be 13290 terminated. 13291 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 13292 as a no-operation. The trap handler 13293 is entered and immediately returns to 13294 continue execution of the wavefront. 13295 - If the debugger is enabled, causes 13296 the debug trap to be reported by the 13297 debugger and the wavefront is put in 13298 the halt state with the PC at the 13299 instruction. The debugger must 13300 increment the PC and resume the wave. 13301 reserved ``s_trap 0x04`` Reserved. 13302 reserved ``s_trap 0x05`` Reserved. 13303 reserved ``s_trap 0x06`` Reserved. 13304 reserved ``s_trap 0x07`` Reserved. 13305 reserved ``s_trap 0x08`` Reserved. 13306 reserved ``s_trap 0xfe`` Reserved. 13307 reserved ``s_trap 0xff`` Reserved. 13308 =================== =============== ================ ================= ======================================= 13309 13310.. _amdgpu-amdhsa-function-call-convention: 13311 13312Call Convention 13313~~~~~~~~~~~~~~~ 13314 13315.. note:: 13316 13317 This section is currently incomplete and has inaccuracies. It is WIP that will 13318 be updated as information is determined. 13319 13320See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 13321addresses. Unswizzled addresses are normal linear addresses. 13322 13323.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 13324 13325Kernel Functions 13326++++++++++++++++ 13327 13328This section describes the call convention ABI for the outer kernel function. 13329 13330See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 13331convention. 13332 13333The following is not part of the AMDGPU kernel calling convention but describes 13334how the AMDGPU implements function calls: 13335 133361. Clang decides the kernarg layout to match the *HSA Programmer's Language 13337 Reference* [HSA]_. 13338 13339 - All structs are passed directly. 13340 - Lambda values are passed *TBA*. 13341 13342 .. TODO:: 13343 13344 - Does this really follow HSA rules? Or are structs >16 bytes passed 13345 by-value struct? 13346 - What is ABI for lambda values? 13347 133484. The kernel performs certain setup in its prolog, as described in 13349 :ref:`amdgpu-amdhsa-kernel-prolog`. 13350 13351.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 13352 13353Non-Kernel Functions 13354++++++++++++++++++++ 13355 13356This section describes the call convention ABI for functions other than the 13357outer kernel function. 13358 13359If a kernel has function calls then scratch is always allocated and used for 13360the call stack which grows from low address to high address using the swizzled 13361scratch address space. 13362 13363On entry to a function: 13364 133651. SGPR0-3 contain a V# with the following properties (see 13366 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 13367 13368 * Base address pointing to the beginning of the wavefront scratch backing 13369 memory. 13370 * Swizzled with dword element size and stride of wavefront size elements. 13371 133722. The FLAT_SCRATCH register pair is setup. See 13373 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 133743. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 13375 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 133764. The EXEC register is set to the lanes active on entry to the function. 133775. MODE register: *TBD* 133786. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 13379 below. 133807. SGPR30-31 return address (RA). The code address that the function must 13381 return to when it completes. The value is undefined if the function is *no 13382 return*. 133838. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 13384 offset relative to the beginning of the wavefront scratch backing memory. 13385 13386 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 13387 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 13388 manner. 13389 13390 The unswizzled SP value can be converted into the swizzled SP value by: 13391 13392 | swizzled SP = unswizzled SP / wavefront size 13393 13394 This may be used to obtain the private address space address of stack 13395 objects and to convert this address to a flat address by adding the flat 13396 scratch aperture base address. 13397 13398 The swizzled SP value is always 4 bytes aligned for the ``r600`` 13399 architecture and 16 byte aligned for the ``amdgcn`` architecture. 13400 13401 .. note:: 13402 13403 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 13404 OpenCL language which has the largest base type defined as 16 bytes. 13405 13406 On entry, the swizzled SP value is the address of the first function 13407 argument passed on the stack. Other stack passed arguments are positive 13408 offsets from the entry swizzled SP value. 13409 13410 The function may use positive offsets beyond the last stack passed argument 13411 for stack allocated local variables and register spill slots. If necessary, 13412 the function may align these to greater alignment than 16 bytes. After these 13413 the function may dynamically allocate space for such things as runtime sized 13414 ``alloca`` local allocations. 13415 13416 If the function calls another function, it will place any stack allocated 13417 arguments after the last local allocation and adjust SGPR32 to the address 13418 after the last local allocation. 13419 134209. All other registers are unspecified. 1342110. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 13422 to the function. 13423 13424On exit from a function: 13425 134261. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 13427 described below. Any registers used are considered clobbered registers. 134282. The following registers are preserved and have the same value as on entry: 13429 13430 * FLAT_SCRATCH 13431 * EXEC 13432 * GFX6-GFX8: M0 13433 * All SGPR registers except the clobbered registers of SGPR4-31. 13434 * VGPR40-47 13435 * VGPR56-63 13436 * VGPR72-79 13437 * VGPR88-95 13438 * VGPR104-111 13439 * VGPR120-127 13440 * VGPR136-143 13441 * VGPR152-159 13442 * VGPR168-175 13443 * VGPR184-191 13444 * VGPR200-207 13445 * VGPR216-223 13446 * VGPR232-239 13447 * VGPR248-255 13448 13449 .. note:: 13450 13451 Except the argument registers, the VGPRs clobbered and the preserved 13452 registers are intermixed at regular intervals in order to keep a 13453 similar ratio independent of the number of allocated VGPRs. 13454 13455 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 13456 * Lanes of all VGPRs that are inactive at the call site. 13457 13458 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 13459 optimization may mark some of clobbered SGPR and VGPR registers as 13460 preserved if it can be determined that the called function does not change 13461 their value. 13462 134632. The PC is set to the RA provided on entry. 134643. MODE register: *TBD*. 134654. All other registers are clobbered. 134665. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 13467 function is available to the caller. 13468 13469.. TODO:: 13470 13471 - How are function results returned? The address of structured types is passed 13472 by reference, but what about other types? 13473 13474The function input arguments are made up of the formal arguments explicitly 13475declared by the source language function plus the implicit input arguments used 13476by the implementation. 13477 13478The source language input arguments are: 13479 134801. Any source language implicit ``this`` or ``self`` argument comes first as a 13481 pointer type. 134822. Followed by the function formal arguments in left to right source order. 13483 13484The source language result arguments are: 13485 134861. The function result argument. 13487 13488The source language input or result struct type arguments that are less than or 13489equal to 16 bytes, are decomposed recursively into their base type fields, and 13490each field is passed as if a separate argument. For input arguments, if the 13491called function requires the struct to be in memory, for example because its 13492address is taken, then the function body is responsible for allocating a stack 13493location and copying the field arguments into it. Clang terms this *direct 13494struct*. 13495 13496The source language input struct type arguments that are greater than 16 bytes, 13497are passed by reference. The caller is responsible for allocating a stack 13498location to make a copy of the struct value and pass the address as the input 13499argument. The called function is responsible to perform the dereference when 13500accessing the input argument. Clang terms this *by-value struct*. 13501 13502A source language result struct type argument that is greater than 16 bytes, is 13503returned by reference. The caller is responsible for allocating a stack location 13504to hold the result value and passes the address as the last input argument 13505(before the implicit input arguments). In this case there are no result 13506arguments. The called function is responsible to perform the dereference when 13507storing the result value. Clang terms this *structured return (sret)*. 13508 13509*TODO: correct the ``sret`` definition.* 13510 13511.. TODO:: 13512 13513 Is this definition correct? Or is ``sret`` only used if passing in registers, and 13514 pass as non-decomposed struct as stack argument? Or something else? Is the 13515 memory location in the caller stack frame, or a stack memory argument and so 13516 no address is passed as the caller can directly write to the argument stack 13517 location? But then the stack location is still live after return. If an 13518 argument stack location is it the first stack argument or the last one? 13519 13520Lambda argument types are treated as struct types with an implementation defined 13521set of fields. 13522 13523.. TODO:: 13524 13525 Need to specify the ABI for lambda types for AMDGPU. 13526 13527For AMDGPU backend all source language arguments (including the decomposed 13528struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 13529they are passed in SGPRs. 13530 13531The AMDGPU backend walks the function call graph from the leaves to determine 13532which implicit input arguments are used, propagating to each caller of the 13533function. The used implicit arguments are appended to the function arguments 13534after the source language arguments in the following order: 13535 13536.. TODO:: 13537 13538 Is recursion or external functions supported? 13539 135401. Work-Item ID (1 VGPR) 13541 13542 The X, Y and Z work-item ID are packed into a single VGRP with the following 13543 layout. Only fields actually used by the function are set. The other bits 13544 are undefined. 13545 13546 The values come from the initial kernel execution state. See 13547 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 13548 13549 .. table:: Work-item implicit argument layout 13550 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 13551 13552 ======= ======= ============== 13553 Bits Size Field Name 13554 ======= ======= ============== 13555 9:0 10 bits X Work-Item ID 13556 19:10 10 bits Y Work-Item ID 13557 29:20 10 bits Z Work-Item ID 13558 31:30 2 bits Unused 13559 ======= ======= ============== 13560 135612. Dispatch Ptr (2 SGPRs) 13562 13563 The value comes from the initial kernel execution state. See 13564 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13565 135663. Queue Ptr (2 SGPRs) 13567 13568 The value comes from the initial kernel execution state. See 13569 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13570 135714. Kernarg Segment Ptr (2 SGPRs) 13572 13573 The value comes from the initial kernel execution state. See 13574 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13575 135765. Dispatch id (2 SGPRs) 13577 13578 The value comes from the initial kernel execution state. See 13579 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13580 135816. Work-Group ID X (1 SGPR) 13582 13583 The value comes from the initial kernel execution state. See 13584 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13585 135867. Work-Group ID Y (1 SGPR) 13587 13588 The value comes from the initial kernel execution state. See 13589 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13590 135918. Work-Group ID Z (1 SGPR) 13592 13593 The value comes from the initial kernel execution state. See 13594 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13595 135969. Implicit Argument Ptr (2 SGPRs) 13597 13598 The value is computed by adding an offset to Kernarg Segment Ptr to get the 13599 global address space pointer to the first kernarg implicit argument. 13600 13601The input and result arguments are assigned in order in the following manner: 13602 13603.. note:: 13604 13605 There are likely some errors and omissions in the following description that 13606 need correction. 13607 13608 .. TODO:: 13609 13610 Check the Clang source code to decipher how function arguments and return 13611 results are handled. Also see the AMDGPU specific values used. 13612 13613* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 13614 VGPR31. 13615 13616 If there are more arguments than will fit in these registers, the remaining 13617 arguments are allocated on the stack in order on naturally aligned 13618 addresses. 13619 13620 .. TODO:: 13621 13622 How are overly aligned structures allocated on the stack? 13623 13624* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 13625 SGPR29. 13626 13627 If there are more arguments than will fit in these registers, the remaining 13628 arguments are allocated on the stack in order on naturally aligned 13629 addresses. 13630 13631Note that decomposed struct type arguments may have some fields passed in 13632registers and some in memory. 13633 13634.. TODO:: 13635 13636 So, a struct which can pass some fields as decomposed register arguments, will 13637 pass the rest as decomposed stack elements? But an argument that will not start 13638 in registers will not be decomposed and will be passed as a non-decomposed 13639 stack value? 13640 13641The following is not part of the AMDGPU function calling convention but 13642describes how the AMDGPU implements function calls: 13643 136441. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 13645 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 13646 are used, or for the reasons defined in ``SIFrameLowering``. 136472. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 13648 to access the incoming stack arguments in the function. The BP is needed 13649 only when the function requires the runtime stack alignment. 13650 136513. Allocating SGPR arguments on the stack are not supported. 13652 136534. No CFI is currently generated. See 13654 :ref:`amdgpu-dwarf-call-frame-information`. 13655 13656 .. note:: 13657 13658 CFI will be generated that defines the CFA as the unswizzled address 13659 relative to the wave scratch base in the unswizzled private address space 13660 of the lowest address stack allocated local variable. 13661 13662 ``DW_AT_frame_base`` will be defined as the swizzled address in the 13663 swizzled private address space by dividing the CFA by the wavefront size 13664 (since CFA is always at least dword aligned which matches the scratch 13665 swizzle element size). 13666 13667 If no dynamic stack alignment was performed, the stack allocated arguments 13668 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 13669 local variables and register spill slots are accessed as positive offsets 13670 relative to ``DW_AT_frame_base``. 13671 136725. Function argument passing is implemented by copying the input physical 13673 registers to virtual registers on entry. The register allocator can spill if 13674 necessary. These are copied back to physical registers at call sites. The 13675 net effect is that each function call can have these values in entirely 13676 distinct locations. The IPRA can help avoid shuffling argument registers. 136776. Call sites are implemented by setting up the arguments at positive offsets 13678 from SP. Then SP is incremented to account for the known frame size before 13679 the call and decremented after the call. 13680 13681 .. note:: 13682 13683 The CFI will reflect the changed calculation needed to compute the CFA 13684 from SP. 13685 136867. 4 byte spill slots are used in the stack frame. One slot is allocated for an 13687 emergency spill slot. Buffer instructions are used for stack accesses and 13688 not the ``flat_scratch`` instruction. 13689 13690 .. TODO:: 13691 13692 Explain when the emergency spill slot is used. 13693 13694.. TODO:: 13695 13696 Possible broken issues: 13697 13698 - Stack arguments must be aligned to required alignment. 13699 - Stack is aligned to max(16, max formal argument alignment) 13700 - Direct argument < 64 bits should check register budget. 13701 - Register budget calculation should respect ``inreg`` for SGPR. 13702 - SGPR overflow is not handled. 13703 - struct with 1 member unpeeling is not checking size of member. 13704 - ``sret`` is after ``this`` pointer. 13705 - Caller is not implementing stack realignment: need an extra pointer. 13706 - Should say AMDGPU passes FP rather than SP. 13707 - Should CFI define CFA as address of locals or arguments. Difference is 13708 apparent when have implemented dynamic alignment. 13709 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 13710 highest address of stack frame and use negative offset for locals. Would 13711 allow SP to be the same as FP and could support signal-handler-like as now 13712 have a real SP for the top of the stack. 13713 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 13714 arguments? 13715 13716AMDPAL 13717------ 13718 13719This section provides code conventions used when the target triple OS is 13720``amdpal`` (see :ref:`amdgpu-target-triples`). 13721 13722.. _amdgpu-amdpal-code-object-metadata-section: 13723 13724Code Object Metadata 13725~~~~~~~~~~~~~~~~~~~~ 13726 13727.. note:: 13728 13729 The metadata is currently in development and is subject to major 13730 changes. Only the current version is supported. *When this document 13731 was generated the version was 2.6.* 13732 13733Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 13734record (see :ref:`amdgpu-note-records-v3-onwards`). 13735 13736The metadata is represented as Message Pack formatted binary data (see 13737[MsgPack]_). The top level is a Message Pack map that includes the keys 13738defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 13739and referenced tables. 13740 13741Additional information can be added to the maps. To avoid conflicts, any 13742key names should be prefixed by "*vendor-name*." where ``vendor-name`` 13743can be the name of the vendor and specific vendor tool that generates the 13744information. The prefix is abbreviated to simply "." when it appears 13745within a map that has been added by the same *vendor-name*. 13746 13747 .. table:: AMDPAL Code Object Metadata Map 13748 :name: amdgpu-amdpal-code-object-metadata-map-table 13749 13750 =================== ============== ========= ====================================================================== 13751 String Key Value Type Required? Description 13752 =================== ============== ========= ====================================================================== 13753 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 13754 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 13755 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 13756 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 13757 definition of the keys included in that map. 13758 =================== ============== ========= ====================================================================== 13759 13760.. 13761 13762 .. table:: AMDPAL Code Object Pipeline Metadata Map 13763 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 13764 13765 ====================================== ============== ========= =================================================== 13766 String Key Value Type Required? Description 13767 ====================================== ============== ========= =================================================== 13768 ".name" string Source name of the pipeline. 13769 ".type" string Pipeline type, e.g. VsPs. Values include: 13770 13771 - "VsPs" 13772 - "Gs" 13773 - "Cs" 13774 - "Ngg" 13775 - "Tess" 13776 - "GsTess" 13777 - "NggTess" 13778 13779 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 13780 2 integers 64 bits is the "stable" portion of the hash, used 13781 for e.g. shader replacement lookup. Upper 64 bits 13782 is the "unique" portion of the hash, used for 13783 e.g. pipeline cache lookup. The value is 13784 implementation defined, and can not be relied on 13785 between different builds of the compiler. 13786 ".shaders" map Per-API shader metadata. See 13787 :ref:`amdgpu-amdpal-code-object-shader-map-table` 13788 for the definition of the keys included in that 13789 map. 13790 ".hardware_stages" map Per-hardware stage metadata. See 13791 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 13792 for the definition of the keys included in that 13793 map. 13794 ".shader_functions" map Per-shader function metadata. See 13795 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 13796 for the definition of the keys included in that 13797 map. 13798 ".registers" map Required Hardware register configuration. See 13799 :ref:`amdgpu-amdpal-code-object-register-map-table` 13800 for the definition of the keys included in that 13801 map. 13802 ".user_data_limit" integer Number of user data entries accessed by this 13803 pipeline. 13804 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 13805 NoUserDataSpilling. 13806 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 13807 viewport array index feature. Pipelines which use 13808 this feature can render into all 16 viewports, 13809 whereas pipelines which do not use it are 13810 restricted to viewport #0. 13811 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 13812 handling data-passing between the ES and GS 13813 shader stages. This can be zero if the data is 13814 passed using off-chip buffers. This value should 13815 be used to program all user-SGPRs which have been 13816 marked with "UserDataMapping::EsGsLdsSize" 13817 (typically only the GS and VS HW stages will ever 13818 have a user-SGPR so marked). 13819 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 13820 (maximum number of threads in a subgroup). 13821 ".num_interpolants" integer Graphics only. Number of PS interpolants. 13822 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 13823 ".api" string Name of the client graphics API. 13824 ".api_create_info" binary Graphics API shader create info binary blob. Can 13825 be defined by the driver using the compiler if 13826 they want to be able to correlate API-specific 13827 information used during creation at a later time. 13828 ====================================== ============== ========= =================================================== 13829 13830.. 13831 13832 .. table:: AMDPAL Code Object Shader Map 13833 :name: amdgpu-amdpal-code-object-shader-map-table 13834 13835 13836 +-------------+--------------+-------------------------------------------------------------------+ 13837 |String Key |Value Type |Description | 13838 +=============+==============+===================================================================+ 13839 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 13840 |- ".vertex" | |for the definition of the keys included in that map. | 13841 |- ".hull" | | | 13842 |- ".domain" | | | 13843 |- ".geometry"| | | 13844 |- ".pixel" | | | 13845 +-------------+--------------+-------------------------------------------------------------------+ 13846 13847.. 13848 13849 .. table:: AMDPAL Code Object API Shader Metadata Map 13850 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 13851 13852 ==================== ============== ========= ===================================================================== 13853 String Key Value Type Required? Description 13854 ==================== ============== ========= ===================================================================== 13855 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 13856 2 integers is implementation defined, and can not be relied on between 13857 different builds of the compiler. 13858 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 13859 string include: 13860 13861 - ".ls" 13862 - ".hs" 13863 - ".es" 13864 - ".gs" 13865 - ".vs" 13866 - ".ps" 13867 - ".cs" 13868 13869 ==================== ============== ========= ===================================================================== 13870 13871.. 13872 13873 .. table:: AMDPAL Code Object Hardware Stage Map 13874 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 13875 13876 +-------------+--------------+-----------------------------------------------------------------------+ 13877 |String Key |Value Type |Description | 13878 +=============+==============+=======================================================================+ 13879 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 13880 |- ".hs" | |for the definition of the keys included in that map. | 13881 |- ".es" | | | 13882 |- ".gs" | | | 13883 |- ".vs" | | | 13884 |- ".ps" | | | 13885 |- ".cs" | | | 13886 +-------------+--------------+-----------------------------------------------------------------------+ 13887 13888.. 13889 13890 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 13891 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 13892 13893 ========================== ============== ========= =============================================================== 13894 String Key Value Type Required? Description 13895 ========================== ============== ========= =============================================================== 13896 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 13897 ".scratch_memory_size" integer Scratch memory size in bytes. 13898 ".lds_size" integer Local Data Share size in bytes. 13899 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 13900 ".vgpr_count" integer Number of VGPRs used. 13901 ".agpr_count" integer Number of AGPRs used. 13902 ".sgpr_count" integer Number of SGPRs used. 13903 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 13904 directive to instruct the compiler to limit the VGPR usage to 13905 be less than or equal to the specified value (only set if 13906 different from HW default). 13907 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 13908 default). 13909 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 13910 3 integers 13911 ".wavefront_size" integer Wavefront size (only set if different from HW default). 13912 ".uses_uavs" boolean The shader reads or writes UAVs. 13913 ".uses_rovs" boolean The shader reads or writes ROVs. 13914 ".writes_uavs" boolean The shader writes to one or more UAVs. 13915 ".writes_depth" boolean The shader writes out a depth value. 13916 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 13917 memory or GDS. 13918 ".uses_prim_id" boolean The shader uses PrimID. 13919 ========================== ============== ========= =============================================================== 13920 13921.. 13922 13923 .. table:: AMDPAL Code Object Shader Function Map 13924 :name: amdgpu-amdpal-code-object-shader-function-map-table 13925 13926 =============== ============== ==================================================================== 13927 String Key Value Type Description 13928 =============== ============== ==================================================================== 13929 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 13930 entry address. The value is the function's metadata. See 13931 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 13932 =============== ============== ==================================================================== 13933 13934.. 13935 13936 .. table:: AMDPAL Code Object Shader Function Metadata Map 13937 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 13938 13939 ============================= ============== ================================================================= 13940 String Key Value Type Description 13941 ============================= ============== ================================================================= 13942 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 13943 2 integers is implementation defined, and can not be relied on between 13944 different builds of the compiler. 13945 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 13946 ".lds_size" integer Size in bytes of LDS memory. 13947 ".vgpr_count" integer Number of VGPRs used by the shader. 13948 ".sgpr_count" integer Number of SGPRs used by the shader. 13949 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 13950 ".shader_subtype" string Shader subtype/kind. Values include: 13951 13952 - "Unknown" 13953 13954 ============================= ============== ================================================================= 13955 13956.. 13957 13958 .. table:: AMDPAL Code Object Register Map 13959 :name: amdgpu-amdpal-code-object-register-map-table 13960 13961 ========================== ============== ==================================================================== 13962 32-bit Integer Key Value Type Description 13963 ========================== ============== ==================================================================== 13964 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 13965 a GRBM register (i.e., driver accessible GPU register number, not 13966 shader GPR register number). The driver is required to program each 13967 specified register to the corresponding specified value when 13968 executing this pipeline. Typically, the ``reg offsets`` are the 13969 ``uint16_t`` offsets to each register as defined by the hardware 13970 chip headers. The register is set to the provided value. However, a 13971 ``reg offset`` that specifies a user data register (e.g., 13972 COMPUTE_USER_DATA_0) needs special treatment. See 13973 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 13974 information. 13975 ========================== ============== ==================================================================== 13976 13977.. _amdgpu-amdpal-code-object-user-data-section: 13978 13979User Data 13980+++++++++ 13981 13982Each hardware stage has a set of 32-bit physical SPI *user data registers* 13983(either 16 or 32 based on graphics IP and the stage) which can be 13984written from a command buffer and then loaded into SGPRs when waves are 13985launched via a subsequent dispatch or draw operation. This is the way 13986most arguments are passed from the application/runtime to a hardware 13987shader. 13988 13989PAL abstracts this functionality by exposing a set of 128 *user data 13990entries* per pipeline a client can use to pass arguments from a command 13991buffer to one or more shaders in that pipeline. The ELF code object must 13992specify a mapping from virtualized *user data entries* to physical *user 13993data registers*, and PAL is responsible for implementing that mapping, 13994including spilling overflow *user data entries* to memory if needed. 13995 13996Since the *user data registers* are GRBM-accessible SPI registers, this 13997mapping is actually embedded in the ``.registers`` metadata entry. For 13998most registers, the value in that map is a literal 32-bit value that 13999should be written to the register by the driver. However, when the 14000register is a *user data register* (any USER_DATA register e.g., 14001SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 14002the driver to write either a *user data entry* value or one of several 14003driver-internal values to the register. This encoding is described in 14004the following table: 14005 14006.. note:: 14007 14008 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 14009 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 14010 always be programmed to the address of the GlobalTable, and *user data 14011 register* 1 must always be programmed to the address of the PerShaderTable. 14012 14013.. 14014 14015 .. table:: AMDPAL User Data Mapping 14016 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 14017 14018 ========== ================= =============================================================================== 14019 Value Name Description 14020 ========== ================= =============================================================================== 14021 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 14022 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 14023 always point to *user data register* 0). 14024 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 14025 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 14026 for more detail (should always point to *user data register* 1). 14027 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 14028 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 14029 more detail. 14030 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 14031 reference the draw index in the vertex shader. Only supported by the first 14032 stage in a graphics pipeline. 14033 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 14034 a graphics pipeline. 14035 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 14036 graphics pipeline. 14037 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 14038 a buffer containing the grid dimensions for a Compute dispatch operation. The 14039 high half of the address is stored in the next sequential user-SGPR. Only 14040 supported by compute pipelines. 14041 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 14042 space used for the ES/GS pseudo-ring-buffer for passing data between shader 14043 stages. 14044 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 14045 pipeline instancing. 14046 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 14047 can only appear for one shader stage per pipeline. 14048 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 14049 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 14050 only appear for one shader stage per pipeline. 14051 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 14052 only appear for one shader stage per pipeline (PS). These replace color targets 14053 and are completely separate from any UAVs used by the shader. This is optional, 14054 and only used by the PS when UAV exports are used to replace color-target 14055 exports to optimize specific shaders. 14056 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 14057 some NGG pipelines to perform culling. This value contains the address of the 14058 first of two consecutive registers which provide the full GPU address. 14059 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 14060 ========== ================= =============================================================================== 14061 14062.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 14063 14064Per-Shader Table 14065################ 14066 14067Low 32 bits of the GPU address for an optional buffer in the ``.data`` 14068section of the ELF. The high 32 bits of the address match the high 32 bits 14069of the shader's program counter. 14070 14071The buffer can be anything the shader compiler needs it for, and 14072allows each shader to have its own region of the ``.data`` section. 14073Typically, this could be a table of buffer SRD's and the data pointed to 14074by the buffer SRD's, but it could be a flat-address region of memory as 14075well. Its layout and usage are defined by the shader compiler. 14076 14077Each shader's table in the ``.data`` section is referenced by the symbol 14078``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 14079hardware shader stage the data is for. E.g., 14080``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 14081 14082.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 14083 14084Spill Table 14085########### 14086 14087It is possible for a hardware shader to need access to more *user data 14088entries* than there are slots available in user data registers for one 14089or more hardware shader stages. In that case, the PAL runtime expects 14090the necessary *user data entries* to be spilled to GPU memory and use 14091one user data register to point to the spilled user data memory. The 14092value of the *user data entry* must then represent the location where 14093a shader expects to read the low 32-bits of the table's GPU virtual 14094address. The *spill table* itself represents a set of 32-bit values 14095managed by the PAL runtime in GPU-accessible memory that can be made 14096indirectly accessible to a hardware shader. 14097 14098Unspecified OS 14099-------------- 14100 14101This section provides code conventions used when the target triple OS is 14102empty (see :ref:`amdgpu-target-triples`). 14103 14104Trap Handler ABI 14105~~~~~~~~~~~~~~~~ 14106 14107For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 14108not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 14109instructions are handled as follows: 14110 14111 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 14112 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 14113 14114 =============== =============== =========================================== 14115 Usage Code Sequence Description 14116 =============== =============== =========================================== 14117 llvm.trap s_endpgm Causes wavefront to be terminated. 14118 llvm.debugtrap *none* Compiler warning given that there is no 14119 trap handler installed. 14120 =============== =============== =========================================== 14121 14122Source Languages 14123================ 14124 14125.. _amdgpu-opencl: 14126 14127OpenCL 14128------ 14129 14130When the language is OpenCL the following differences occur: 14131 141321. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 141332. The AMDGPU backend appends additional arguments to the kernel's explicit 14134 arguments for the AMDHSA OS (see 14135 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 141363. Additional metadata is generated 14137 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 14138 14139 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 14140 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 14141 14142 ======== ==== ========= =========================================== 14143 Position Byte Byte Description 14144 Size Alignment 14145 ======== ==== ========= =========================================== 14146 1 8 8 OpenCL Global Offset X 14147 2 8 8 OpenCL Global Offset Y 14148 3 8 8 OpenCL Global Offset Z 14149 4 8 8 OpenCL address of printf buffer 14150 5 8 8 OpenCL address of virtual queue used by 14151 enqueue_kernel. 14152 6 8 8 OpenCL address of AqlWrap struct used by 14153 enqueue_kernel. 14154 7 8 8 Pointer argument used for Multi-gird 14155 synchronization. 14156 ======== ==== ========= =========================================== 14157 14158.. _amdgpu-hcc: 14159 14160HCC 14161--- 14162 14163When the language is HCC the following differences occur: 14164 141651. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 14166 14167.. _amdgpu-assembler: 14168 14169Assembler 14170--------- 14171 14172AMDGPU backend has LLVM-MC based assembler which is currently in development. 14173It supports AMDGCN GFX6-GFX10. 14174 14175This section describes general syntax for instructions and operands. 14176 14177Instructions 14178~~~~~~~~~~~~ 14179 14180An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 14181 14182 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 14183 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 14184 14185:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 14186:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 14187 14188The order of operands and modifiers is fixed. 14189Most modifiers are optional and may be omitted. 14190 14191Links to detailed instruction syntax description may be found in the following 14192table. Note that features under development are not included 14193in this description. 14194 14195 ============= ============================================= ======================================= 14196 Architecture Core ISA ISA Variants and Extensions 14197 ============= ============================================= ======================================= 14198 GCN 2 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 14199 GCN 3, GCN 4 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 14200 GCN 5 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 14201 14202 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 14203 14204 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 14205 14206 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 14207 14208 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 14209 14210 CDNA 1 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 14211 CDNA 2 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 14212 RDNA 1 :doc:`GFX10 RDNA1<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1010<AMDGPU/AMDGPUAsmGFX10>` 14213 14214 :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 14215 14216 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 14217 14218 :doc:`gfx1013<AMDGPU/AMDGPUAsmGFX1013>` 14219 14220 RDNA 2 :doc:`GFX10 RDNA2<AMDGPU/AMDGPUAsmGFX1030>` :doc:`gfx1030<AMDGPU/AMDGPUAsmGFX1030>` 14221 14222 :doc:`gfx1031<AMDGPU/AMDGPUAsmGFX1030>` 14223 14224 :doc:`gfx1032<AMDGPU/AMDGPUAsmGFX1030>` 14225 14226 :doc:`gfx1033<AMDGPU/AMDGPUAsmGFX1030>` 14227 14228 :doc:`gfx1034<AMDGPU/AMDGPUAsmGFX1030>` 14229 14230 :doc:`gfx1035<AMDGPU/AMDGPUAsmGFX1030>` 14231 14232 :doc:`gfx1036<AMDGPU/AMDGPUAsmGFX1030>` 14233 ============= ============================================= ======================================= 14234 14235For more information about instructions, their semantics and supported 14236combinations of operands, refer to one of instruction set architecture manuals 14237[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 14238[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_, 14239[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX90A-CDNA2]_, [AMD-GCN-GFX10-RDNA1]_ and 14240[AMD-GCN-GFX10-RDNA2]_. 14241 14242Operands 14243~~~~~~~~ 14244 14245Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 14246 14247Modifiers 14248~~~~~~~~~ 14249 14250Detailed description of modifiers may be found 14251:doc:`here<AMDGPUModifierSyntax>`. 14252 14253Instruction Examples 14254~~~~~~~~~~~~~~~~~~~~ 14255 14256DS 14257++ 14258 14259.. code-block:: nasm 14260 14261 ds_add_u32 v2, v4 offset:16 14262 ds_write_src2_b64 v2 offset0:4 offset1:8 14263 ds_cmpst_f32 v2, v4, v6 14264 ds_min_rtn_f64 v[8:9], v2, v[4:5] 14265 14266For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 14267Manual. 14268 14269FLAT 14270++++ 14271 14272.. code-block:: nasm 14273 14274 flat_load_dword v1, v[3:4] 14275 flat_store_dwordx3 v[3:4], v[5:7] 14276 flat_atomic_swap v1, v[3:4], v5 glc 14277 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 14278 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 14279 14280For full list of supported instructions, refer to "FLAT instructions" in ISA 14281Manual. 14282 14283MUBUF 14284+++++ 14285 14286.. code-block:: nasm 14287 14288 buffer_load_dword v1, off, s[4:7], s1 14289 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 14290 buffer_store_format_xy v[1:2], off, s[4:7], s1 14291 buffer_wbinvl1 14292 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 14293 14294For full list of supported instructions, refer to "MUBUF Instructions" in ISA 14295Manual. 14296 14297SMRD/SMEM 14298+++++++++ 14299 14300.. code-block:: nasm 14301 14302 s_load_dword s1, s[2:3], 0xfc 14303 s_load_dwordx8 s[8:15], s[2:3], s4 14304 s_load_dwordx16 s[88:103], s[2:3], s4 14305 s_dcache_inv_vol 14306 s_memtime s[4:5] 14307 14308For full list of supported instructions, refer to "Scalar Memory Operations" in 14309ISA Manual. 14310 14311SOP1 14312++++ 14313 14314.. code-block:: nasm 14315 14316 s_mov_b32 s1, s2 14317 s_mov_b64 s[0:1], 0x80000000 14318 s_cmov_b32 s1, 200 14319 s_wqm_b64 s[2:3], s[4:5] 14320 s_bcnt0_i32_b64 s1, s[2:3] 14321 s_swappc_b64 s[2:3], s[4:5] 14322 s_cbranch_join s[4:5] 14323 14324For full list of supported instructions, refer to "SOP1 Instructions" in ISA 14325Manual. 14326 14327SOP2 14328++++ 14329 14330.. code-block:: nasm 14331 14332 s_add_u32 s1, s2, s3 14333 s_and_b64 s[2:3], s[4:5], s[6:7] 14334 s_cselect_b32 s1, s2, s3 14335 s_andn2_b32 s2, s4, s6 14336 s_lshr_b64 s[2:3], s[4:5], s6 14337 s_ashr_i32 s2, s4, s6 14338 s_bfm_b64 s[2:3], s4, s6 14339 s_bfe_i64 s[2:3], s[4:5], s6 14340 s_cbranch_g_fork s[4:5], s[6:7] 14341 14342For full list of supported instructions, refer to "SOP2 Instructions" in ISA 14343Manual. 14344 14345SOPC 14346++++ 14347 14348.. code-block:: nasm 14349 14350 s_cmp_eq_i32 s1, s2 14351 s_bitcmp1_b32 s1, s2 14352 s_bitcmp0_b64 s[2:3], s4 14353 s_setvskip s3, s5 14354 14355For full list of supported instructions, refer to "SOPC Instructions" in ISA 14356Manual. 14357 14358SOPP 14359++++ 14360 14361.. code-block:: nasm 14362 14363 s_barrier 14364 s_nop 2 14365 s_endpgm 14366 s_waitcnt 0 ; Wait for all counters to be 0 14367 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 14368 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 14369 s_sethalt 9 14370 s_sleep 10 14371 s_sendmsg 0x1 14372 s_sendmsg sendmsg(MSG_INTERRUPT) 14373 s_trap 1 14374 14375For full list of supported instructions, refer to "SOPP Instructions" in ISA 14376Manual. 14377 14378Unless otherwise mentioned, little verification is performed on the operands 14379of SOPP Instructions, so it is up to the programmer to be familiar with the 14380range or acceptable values. 14381 14382VALU 14383++++ 14384 14385For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 14386the assembler will automatically use optimal encoding based on its operands. To 14387force specific encoding, one can add a suffix to the opcode of the instruction: 14388 14389* _e32 for 32-bit VOP1/VOP2/VOPC 14390* _e64 for 64-bit VOP3 14391* _dpp for VOP_DPP 14392* _sdwa for VOP_SDWA 14393 14394VOP1/VOP2/VOP3/VOPC examples: 14395 14396.. code-block:: nasm 14397 14398 v_mov_b32 v1, v2 14399 v_mov_b32_e32 v1, v2 14400 v_nop 14401 v_cvt_f64_i32_e32 v[1:2], v2 14402 v_floor_f32_e32 v1, v2 14403 v_bfrev_b32_e32 v1, v2 14404 v_add_f32_e32 v1, v2, v3 14405 v_mul_i32_i24_e64 v1, v2, 3 14406 v_mul_i32_i24_e32 v1, -3, v3 14407 v_mul_i32_i24_e32 v1, -100, v3 14408 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 14409 v_max_f16_e32 v1, v2, v3 14410 14411VOP_DPP examples: 14412 14413.. code-block:: nasm 14414 14415 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 14416 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14417 v_mov_b32 v0, v0 wave_shl:1 14418 v_mov_b32 v0, v0 row_mirror 14419 v_mov_b32 v0, v0 row_bcast:31 14420 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 14421 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14422 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14423 14424VOP_SDWA examples: 14425 14426.. code-block:: nasm 14427 14428 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 14429 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 14430 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 14431 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 14432 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 14433 14434For full list of supported instructions, refer to "Vector ALU instructions". 14435 14436.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 14437 14438Code Object V2 Predefined Symbols 14439~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14440 14441.. warning:: 14442 Code object V2 is not the default code object version emitted by 14443 this version of LLVM. 14444 14445The AMDGPU assembler defines and updates some symbols automatically. These 14446symbols do not affect code generation. 14447 14448.option.machine_version_major 14449+++++++++++++++++++++++++++++ 14450 14451Set to the GFX major generation number of the target being assembled for. For 14452example, when assembling for a "GFX9" target this will be set to the integer 14453value "9". The possible GFX major generation numbers are presented in 14454:ref:`amdgpu-processors`. 14455 14456.option.machine_version_minor 14457+++++++++++++++++++++++++++++ 14458 14459Set to the GFX minor generation number of the target being assembled for. For 14460example, when assembling for a "GFX810" target this will be set to the integer 14461value "1". The possible GFX minor generation numbers are presented in 14462:ref:`amdgpu-processors`. 14463 14464.option.machine_version_stepping 14465++++++++++++++++++++++++++++++++ 14466 14467Set to the GFX stepping generation number of the target being assembled for. 14468For example, when assembling for a "GFX704" target this will be set to the 14469integer value "4". The possible GFX stepping generation numbers are presented 14470in :ref:`amdgpu-processors`. 14471 14472.kernel.vgpr_count 14473++++++++++++++++++ 14474 14475Set to zero each time a 14476:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14477encountered. At each instruction, if the current value of this symbol is less 14478than or equal to the maximum VGPR number explicitly referenced within that 14479instruction then the symbol value is updated to equal that VGPR number plus 14480one. 14481 14482.kernel.sgpr_count 14483++++++++++++++++++ 14484 14485Set to zero each time a 14486:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14487encountered. At each instruction, if the current value of this symbol is less 14488than or equal to the maximum VGPR number explicitly referenced within that 14489instruction then the symbol value is updated to equal that SGPR number plus 14490one. 14491 14492.. _amdgpu-amdhsa-assembler-directives-v2: 14493 14494Code Object V2 Directives 14495~~~~~~~~~~~~~~~~~~~~~~~~~ 14496 14497.. warning:: 14498 Code object V2 is not the default code object version emitted by 14499 this version of LLVM. 14500 14501AMDGPU ABI defines auxiliary data in output code object. In assembly source, 14502one can specify them with assembler directives. 14503 14504.hsa_code_object_version major, minor 14505+++++++++++++++++++++++++++++++++++++ 14506 14507*major* and *minor* are integers that specify the version of the HSA code 14508object that will be generated by the assembler. 14509 14510.hsa_code_object_isa [major, minor, stepping, vendor, arch] 14511+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 14512 14513 14514*major*, *minor*, and *stepping* are all integers that describe the instruction 14515set architecture (ISA) version of the assembly program. 14516 14517*vendor* and *arch* are quoted strings. *vendor* should always be equal to 14518"AMD" and *arch* should always be equal to "AMDGPU". 14519 14520By default, the assembler will derive the ISA version, *vendor*, and *arch* 14521from the value of the -mcpu option that is passed to the assembler. 14522 14523.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 14524 14525.amdgpu_hsa_kernel (name) 14526+++++++++++++++++++++++++ 14527 14528This directives specifies that the symbol with given name is a kernel entry 14529point (label) and the object should contain corresponding symbol of type 14530STT_AMDGPU_HSA_KERNEL. 14531 14532.amd_kernel_code_t 14533++++++++++++++++++ 14534 14535This directive marks the beginning of a list of key / value pairs that are used 14536to specify the amd_kernel_code_t object that will be emitted by the assembler. 14537The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 14538amd_kernel_code_t values that are unspecified a default value will be used. The 14539default value for all keys is 0, with the following exceptions: 14540 14541- *amd_code_version_major* defaults to 1. 14542- *amd_kernel_code_version_minor* defaults to 2. 14543- *amd_machine_kind* defaults to 1. 14544- *amd_machine_version_major*, *machine_version_minor*, and 14545 *amd_machine_version_stepping* are derived from the value of the -mcpu option 14546 that is passed to the assembler. 14547- *kernel_code_entry_byte_offset* defaults to 256. 14548- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 14549 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 14550 Note that wavefront size is specified as a power of two, so a value of **n** 14551 means a size of 2^ **n**. 14552- *call_convention* defaults to -1. 14553- *kernarg_segment_alignment*, *group_segment_alignment*, and 14554 *private_segment_alignment* default to 4. Note that alignments are specified 14555 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 14556- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 14557 GFX90A onwards. 14558- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 14559 GFX10 onwards. 14560- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 14561 14562The *.amd_kernel_code_t* directive must be placed immediately after the 14563function label and before any instructions. 14564 14565For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 14566comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 14567 14568.. _amdgpu-amdhsa-assembler-example-v2: 14569 14570Code Object V2 Example Source Code 14571~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14572 14573.. warning:: 14574 Code Object V2 is not the default code object version emitted by 14575 this version of LLVM. 14576 14577Here is an example of a minimal assembly source file, defining one HSA kernel: 14578 14579.. code:: 14580 :number-lines: 14581 14582 .hsa_code_object_version 1,0 14583 .hsa_code_object_isa 14584 14585 .hsatext 14586 .globl hello_world 14587 .p2align 8 14588 .amdgpu_hsa_kernel hello_world 14589 14590 hello_world: 14591 14592 .amd_kernel_code_t 14593 enable_sgpr_kernarg_segment_ptr = 1 14594 is_ptr64 = 1 14595 compute_pgm_rsrc1_vgprs = 0 14596 compute_pgm_rsrc1_sgprs = 0 14597 compute_pgm_rsrc2_user_sgpr = 2 14598 compute_pgm_rsrc1_wgp_mode = 0 14599 compute_pgm_rsrc1_mem_ordered = 0 14600 compute_pgm_rsrc1_fwd_progress = 1 14601 .end_amd_kernel_code_t 14602 14603 s_load_dwordx2 s[0:1], s[0:1] 0x0 14604 v_mov_b32 v0, 3.14159 14605 s_waitcnt lgkmcnt(0) 14606 v_mov_b32 v1, s0 14607 v_mov_b32 v2, s1 14608 flat_store_dword v[1:2], v0 14609 s_endpgm 14610 .Lfunc_end0: 14611 .size hello_world, .Lfunc_end0-hello_world 14612 14613.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-onwards: 14614 14615Code Object V3 and Above Predefined Symbols 14616~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14617 14618The AMDGPU assembler defines and updates some symbols automatically. These 14619symbols do not affect code generation. 14620 14621.amdgcn.gfx_generation_number 14622+++++++++++++++++++++++++++++ 14623 14624Set to the GFX major generation number of the target being assembled for. For 14625example, when assembling for a "GFX9" target this will be set to the integer 14626value "9". The possible GFX major generation numbers are presented in 14627:ref:`amdgpu-processors`. 14628 14629.amdgcn.gfx_generation_minor 14630++++++++++++++++++++++++++++ 14631 14632Set to the GFX minor generation number of the target being assembled for. For 14633example, when assembling for a "GFX810" target this will be set to the integer 14634value "1". The possible GFX minor generation numbers are presented in 14635:ref:`amdgpu-processors`. 14636 14637.amdgcn.gfx_generation_stepping 14638+++++++++++++++++++++++++++++++ 14639 14640Set to the GFX stepping generation number of the target being assembled for. 14641For example, when assembling for a "GFX704" target this will be set to the 14642integer value "4". The possible GFX stepping generation numbers are presented 14643in :ref:`amdgpu-processors`. 14644 14645.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 14646 14647.amdgcn.next_free_vgpr 14648++++++++++++++++++++++ 14649 14650Set to zero before assembly begins. At each instruction, if the current value 14651of this symbol is less than or equal to the maximum VGPR number explicitly 14652referenced within that instruction then the symbol value is updated to equal 14653that VGPR number plus one. 14654 14655May be used to set the `.amdhsa_next_free_vgpr` directive in 14656:ref:`amdhsa-kernel-directives-table`. 14657 14658May be set at any time, e.g. manually set to zero at the start of each kernel. 14659 14660.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 14661 14662.amdgcn.next_free_sgpr 14663++++++++++++++++++++++ 14664 14665Set to zero before assembly begins. At each instruction, if the current value 14666of this symbol is less than or equal the maximum SGPR number explicitly 14667referenced within that instruction then the symbol value is updated to equal 14668that SGPR number plus one. 14669 14670May be used to set the `.amdhsa_next_free_spgr` directive in 14671:ref:`amdhsa-kernel-directives-table`. 14672 14673May be set at any time, e.g. manually set to zero at the start of each kernel. 14674 14675.. _amdgpu-amdhsa-assembler-directives-v3-onwards: 14676 14677Code Object V3 and Above Directives 14678~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14679 14680Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 14681architecture processors, and are not OS-specific. Directives which begin with 14682``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 14683``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 14684:ref:`amdgpu-processors`. 14685 14686.. _amdgpu-assembler-directive-amdgcn-target: 14687 14688.amdgcn_target <target-triple> "-" <target-id> 14689++++++++++++++++++++++++++++++++++++++++++++++ 14690 14691Optional directive which declares the ``<target-triple>-<target-id>`` supported 14692by the containing assembler source file. Used by the assembler to validate 14693command-line options such as ``-triple``, ``-mcpu``, and 14694``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 14695:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 14696 14697.. note:: 14698 14699 The target ID syntax used for code object V2 to V3 for this directive differs 14700 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 14701 14702.amdhsa_kernel <name> 14703+++++++++++++++++++++ 14704 14705Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 14706``<name>.kd``, in the current location of the current section. Only valid when 14707the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 14708instruction to execute, and does not need to be previously defined. 14709 14710Marks the beginning of a list of directives used to generate the bytes of a 14711kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 14712Directives which may appear in this list are described in 14713:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 14714be valid for the target being assembled for, and cannot be repeated. Directives 14715support the range of values specified by the field they reference in 14716:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 14717assumed to have its default value, unless it is marked as "Required", in which 14718case it is an error to omit the directive. This list of directives is 14719terminated by an ``.end_amdhsa_kernel`` directive. 14720 14721 .. table:: AMDHSA Kernel Assembler Directives 14722 :name: amdhsa-kernel-directives-table 14723 14724 ======================================================== =================== ============ =================== 14725 Directive Default Supported On Description 14726 ======================================================== =================== ============ =================== 14727 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 14728 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14729 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 14730 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14731 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 14732 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14733 ``.amdhsa_user_sgpr_count`` 0 GFX6-GFX10 Controls USER_SGPR_COUNT in COMPUTE_PGM_RSRC2 14734 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` 14735 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 14736 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14737 GFX940) 14738 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 14739 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14740 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 14741 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14742 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 14743 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14744 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 14745 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14746 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 14747 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14748 GFX940) 14749 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 14750 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14751 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 14752 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14753 Specific 14754 (wavefrontsize64) 14755 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 14756 (except :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14757 GFX940) 14758 ``.amdhsa_enable_private_segment`` 0 GFX940 Controls ENABLE_PRIVATE_SEGMENT in 14759 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14760 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 14761 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14762 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 14763 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14764 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 14765 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14766 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 14767 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14768 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 14769 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14770 Possible values are defined in 14771 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 14772 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 14773 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 14774 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14775 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 14776 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14777 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14778 ``.amdhsa_accum_offset`` Required GFX90A, Offset of a first AccVGPR in the unified register file. 14779 GFX940 Used to calculate ACCUM_OFFSET in 14780 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14781 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 14782 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14783 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14784 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 14785 (except scratch memory. Used to calculate 14786 GFX940) GRANULATED_WAVEFRONT_SGPR_COUNT in 14787 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14788 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 14789 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14790 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14791 (xnack) 14792 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 14793 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14794 Possible values are defined in 14795 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14796 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 14797 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14798 Possible values are defined in 14799 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14800 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 14801 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14802 Possible values are defined in 14803 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14804 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 14805 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14806 Possible values are defined in 14807 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14808 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 14809 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14810 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 14811 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14812 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 14813 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14814 ``.amdhsa_tg_split`` Target GFX90A, Controls TG_SPLIT in 14815 Feature GFX940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14816 Specific 14817 (tgsplit) 14818 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 14819 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14820 Specific 14821 (cumode) 14822 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 14823 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14824 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 14825 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14826 ``.amdhsa_shared_vgpr_count`` 0 GFX10 Controls SHARED_VGPR_COUNT in 14827 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 14828 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 14829 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14830 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 14831 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14832 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 14833 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14834 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 14835 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14836 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 14837 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14838 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 14839 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14840 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 14841 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14842 ======================================================== =================== ============ =================== 14843 14844.amdgpu_metadata 14845++++++++++++++++ 14846 14847Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 14848note record (see :ref:`amdgpu-elf-note-records-table-v3-onwards`). 14849 14850The contents must be in the [YAML]_ markup format, with the same structure and 14851semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 14852:ref:`amdgpu-amdhsa-code-object-metadata-v4` or 14853:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 14854 14855This directive is terminated by an ``.end_amdgpu_metadata`` directive. 14856 14857.. _amdgpu-amdhsa-assembler-example-v3-onwards: 14858 14859Code Object V3 and Above Example Source Code 14860~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14861 14862Here is an example of a minimal assembly source file, defining one HSA kernel: 14863 14864.. code:: 14865 :number-lines: 14866 14867 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14868 14869 .text 14870 .globl hello_world 14871 .p2align 8 14872 .type hello_world,@function 14873 hello_world: 14874 s_load_dwordx2 s[0:1], s[0:1] 0x0 14875 v_mov_b32 v0, 3.14159 14876 s_waitcnt lgkmcnt(0) 14877 v_mov_b32 v1, s0 14878 v_mov_b32 v2, s1 14879 flat_store_dword v[1:2], v0 14880 s_endpgm 14881 .Lfunc_end0: 14882 .size hello_world, .Lfunc_end0-hello_world 14883 14884 .rodata 14885 .p2align 6 14886 .amdhsa_kernel hello_world 14887 .amdhsa_user_sgpr_kernarg_segment_ptr 1 14888 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14889 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14890 .end_amdhsa_kernel 14891 14892 .amdgpu_metadata 14893 --- 14894 amdhsa.version: 14895 - 1 14896 - 0 14897 amdhsa.kernels: 14898 - .name: hello_world 14899 .symbol: hello_world.kd 14900 .kernarg_segment_size: 48 14901 .group_segment_fixed_size: 0 14902 .private_segment_fixed_size: 0 14903 .kernarg_segment_align: 4 14904 .wavefront_size: 64 14905 .sgpr_count: 2 14906 .vgpr_count: 3 14907 .max_flat_workgroup_size: 256 14908 .args: 14909 - .size: 8 14910 .offset: 0 14911 .value_kind: global_buffer 14912 .address_space: global 14913 .actual_access: write_only 14914 //... 14915 .end_amdgpu_metadata 14916 14917This kernel is equivalent to the following HIP program: 14918 14919.. code:: 14920 :number-lines: 14921 14922 __global__ void hello_world(float *p) { 14923 *p = 3.14159f; 14924 } 14925 14926If an assembly source file contains multiple kernels and/or functions, the 14927:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 14928:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 14929the ``.set <symbol>, <expression>`` directive. For example, in the case of two 14930kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 14931to group the function with the kernel that calls it and reset the symbols 14932between the two connected components: 14933 14934.. code:: 14935 :number-lines: 14936 14937 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14938 14939 // gpr tracking symbols are implicitly set to zero 14940 14941 .text 14942 .globl kern0 14943 .p2align 8 14944 .type kern0,@function 14945 kern0: 14946 // ... 14947 s_endpgm 14948 .Lkern0_end: 14949 .size kern0, .Lkern0_end-kern0 14950 14951 .rodata 14952 .p2align 6 14953 .amdhsa_kernel kern0 14954 // ... 14955 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14956 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14957 .end_amdhsa_kernel 14958 14959 // reset symbols to begin tracking usage in func1 and kern1 14960 .set .amdgcn.next_free_vgpr, 0 14961 .set .amdgcn.next_free_sgpr, 0 14962 14963 .text 14964 .hidden func1 14965 .global func1 14966 .p2align 2 14967 .type func1,@function 14968 func1: 14969 // ... 14970 s_setpc_b64 s[30:31] 14971 .Lfunc1_end: 14972 .size func1, .Lfunc1_end-func1 14973 14974 .globl kern1 14975 .p2align 8 14976 .type kern1,@function 14977 kern1: 14978 // ... 14979 s_getpc_b64 s[4:5] 14980 s_add_u32 s4, s4, func1@rel32@lo+4 14981 s_addc_u32 s5, s5, func1@rel32@lo+4 14982 s_swappc_b64 s[30:31], s[4:5] 14983 // ... 14984 s_endpgm 14985 .Lkern1_end: 14986 .size kern1, .Lkern1_end-kern1 14987 14988 .rodata 14989 .p2align 6 14990 .amdhsa_kernel kern1 14991 // ... 14992 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14993 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14994 .end_amdhsa_kernel 14995 14996These symbols cannot identify connected components in order to automatically 14997track the usage for each kernel. However, in some cases careful organization of 14998the kernels and functions in the source file means there is minimal additional 14999effort required to accurately calculate GPR usage. 15000 15001Additional Documentation 15002======================== 15003 15004.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 15005.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 15006.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 15007.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 15008.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 15009.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 15010.. [AMD-GCN-GFX90A-CDNA2] `AMD Instinct MI200 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA2_Shader_ISA_4February2022.pdf>`__ 15011.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 15012.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 15013.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 15014.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 15015.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 15016.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 15017.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 15018.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 15019.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 15020.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 15021.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 15022.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 15023.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 15024.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 15025.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 15026.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 15027.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 15028.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 15029